cloud-init switching to udhcpc

[Major Hayden]

Hey there,

For those of you who use Fedora often on public cloud providers where 
cloud-init handles the initial configuration, there's a change coming 
for the very early DHCP request during the boot. ☁️

As reported in BZ 2247055[0], the existing dhclient binary comes from 
ISC's dhcp and it stopped being maintained upstream in 2022. Cloud-init 
uses dhclient during the very early setup process to get an IP address 
for retrieving instance metadata.

Busybox provides udhcpc[1] and it works well for cloud-init's early boot 
needs. Support for udhcpc was also added to cloud-init.

I've tested this on AWS and it's working well. You shouldn't notice a 
difference when launching a cloud instance after this change, but I 
wanted to ensure everyone was aware it was coming. 😉

Have a great rest of the week!

[0] https://bugzilla.redhat.com/show_bug.cgi?id=2247055
[1] https://en.wikipedia.org/wiki/Udhcpc

-- 
Major Hayden

[Petr Menšík]

Hello Major,

I think this is stupid anyway. Why does cloud-init need 2 separate DHCP 
clients anyway? Can we gather what exactly they use it for and why 
alternative dhcp client is required?

It seems to me NetworkManager should be able to use just different 
configuration for "pre-connection" phase. But anyway, it is connected by 
that time. Pretending alternative dhcp client makes it "off-line" is 
ridiculous. It might not be reaching network.target with it, but still 
is connected.

It would be great, if Network Manager could be used in both phases. 
Could for example --configure-and-quit with custom --config and 
--state-file provide basic configuration needed before correct network 
start? Were that even considered?

Regards,
Petr

On 30. 01. 24 17:58, Major Hayden wrote:
>  Hey there,
>
>  For those of you who use Fedora often on public cloud providers where 
>  cloud-init handles the initial configuration, there's a change coming 
>  for the very early DHCP request during the boot. ☁️
>
>  As reported in BZ 2247055[0], the existing dhclient binary comes from 
>  ISC's dhcp and it stopped being maintained upstream in 2022. 
>  Cloud-init uses dhclient during the very early setup process to get an 
>  IP address for retrieving instance metadata.
>
>  Busybox provides udhcpc[1] and it works well for cloud-init's early 
>  boot needs. Support for udhcpc was also added to cloud-init.
>
>  I've tested this on AWS and it's working well. You shouldn't notice a 
>  difference when launching a cloud instance after this change, but I 
>  wanted to ensure everyone was aware it was coming. 😉
>
>  Have a great rest of the week!
>
>  [0] https://bugzilla.redhat.com/show_bug.cgi?id=2247055
>  [1] https://en.wikipedia.org/wiki/Udhcpc
>  
-- 
Petr Menšík
Software Engineer, RHEL
Red Hat, http://www.redhat.com/
PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB

[Major Hayden]

On Tue, Feb 6, 2024, at 13:56, Petr Menšík wrote:
>  I think this is stupid anyway. Why does cloud-init need 2 separate
> DHCP 
>  clients anyway? Can we gather what exactly they use it for and why 
>  alternative dhcp client is required? 
It looks like a small, basic, DHCP client is needed during the early boot just to get on
the network to reach the metadata service. That would be before NetworkManager gets
involved.

Stephen Gallagher pointed out that ELN doesn't have busybox, but it does have dhcpcd,
and that should work fine. I've reverted the switch to udhcpcd and I'm waiting on
upstream cloud-init to have a new release with the recently added dhcpcd support.

-- 
Major Hayden

[Chris Adams]

Once upon a time, Major Hayden <major(a)mhtx.net&gt; said:
>  Stephen Gallagher pointed out that ELN doesn't have busybox, but
> it does have dhcpcd, and that should work fine. I've reverted the switch to udhcpcd
> and I'm waiting on upstream cloud-init to have a new release with the recently added
> dhcpcd support. 
ISC dhcpd is also EOL upstream from October 5, 2022, so making a new
dependency on it is probably not a good idea.
-- 
Chris Adams <linux(a)cmadams.net&gt;

[Stephen Gallagher]

On Tue, Feb 6, 2024 at 3:30 PM Chris Adams <linux(a)cmadams.net&gt; wrote:
>  Once upon a time, Major Hayden <major(a)mhtx.net&gt; said:
>  > Stephen Gallagher pointed out that ELN doesn't have busybox, but it does have
> dhcpcd, and that should work fine. I've reverted the switch to udhcpcd and I'm
> waiting on upstream cloud-init to have a new release with the recently added dhcpcd
> support.
>
>  ISC dhcpd is also EOL upstream from October 5, 2022, so making a new
>  dependency on it is probably not a good idea. 
ISC dhcpd[1] and dhcpcd[2] are different projects.

[1] https://www.isc.org/dhcp/
[2] https://github.com/NetworkConfiguration/dhcpcd

[Chris Adams]

Once upon a time, Stephen Gallagher <sgallagh(a)redhat.com&gt; said:
>  On Tue, Feb 6, 2024 at 3:30 PM Chris Adams <linux(a)cmadams.net&gt;
> wrote:
>  > Once upon a time, Major Hayden <major(a)mhtx.net&gt; said:
>  > > Stephen Gallagher pointed out that ELN doesn't have busybox, but it does
> have dhcpcd, and that should work fine. I've reverted the switch to udhcpcd and
> I'm waiting on upstream cloud-init to have a new release with the recently added
> dhcpcd support.
>  >
>  > ISC dhcpd is also EOL upstream from October 5, 2022, so making a new
>  > dependency on it is probably not a good idea.
>  
>  ISC dhcpd[1] and dhcpcd[2] are different projects.
>  
>  [1] https://www.isc.org/dhcp/
>  [2] https://github.com/NetworkConfiguration/dhcpcd 
Sorry, I missed the extra "c" in there!
-- 
Chris Adams <linux(a)cmadams.net&gt;

[Petr Menšík]

I am dhcpcd maintainer and okay, dhcpcd is not bad choice. But its 
maintenance might fall into our team in RHEL.

I am interested why two DHCP clients are required inside a single tool. 
Why it could not use NetworkManager in both times, but with some 
different settings? Maybe alternative service? If there is missing 
ability to change behaviour, it may make sense to fix just that. Instead 
of keeping another client maintained. Do we support any kind of images, 
which would not have Network Manager present?

Was it already evaluated by someone from Network Manager maintainers, 
whether it would be possible reuse NM both times? If changes would be 
required, how significant? Maybe it were already ruled out for a good 
reason. I haven't found any discussion about that topic on cloud-init 
upstream.

On 06. 02. 24 21:04, Major Hayden wrote:
>  On Tue, Feb 6, 2024, at 13:56, Petr Menšík wrote:
> > I think this is stupid anyway. Why does cloud-init need 2 separate DHCP
> > clients anyway? Can we gather what exactly they use it for and why
> > alternative dhcp client is required?
>  It looks like a small, basic, DHCP client is needed during the early boot just to get on
> the network to reach the metadata service. That would be before NetworkManager gets
> involved. 
Yes, I understand that. But I don't understand why. I don't work with 
cloud instances, so forgive my naivety. NetworkManager can obtain 
address and configure interface to a working state. It seems we want to:

- prevent dispatches from running during metadata configuration phase
- not reaching network.target
- not using NetworkManager.service, but stripped down alternative

Is it different, because DBus is not yet running at that time? Could be 
just alternative configuration used to not trigger unwanted parts?

If we had just NetworkManager-metadata.service, using alternate state 
directory as well, couldn't it provide similar functionality as 
additional dhcpcd/dhclient? Just with few configuration files in 
addition, instead of additional compiled package(s)?

>  Stephen Gallagher pointed out that ELN doesn't have busybox, but
> it does have dhcpcd, and that should work fine. I've reverted the switch to udhcpcd
> and I'm waiting on upstream cloud-init to have a new release with the recently added
> dhcpcd support. 
Are there simple steps, where I can as a dhcpcd maintainer try, how 
exactly is dhcpcd used during instance configuration process? Can I 
prepare VM image using mentioned metadata? Do we have any documentation 
on this topic for Fedora or RHEL at least? Can you point me to some 
documentation, how to test it?

Best Regards, Petr

-- 
Petr Menšík
Software Engineer, RHEL
Red Hat,http://www.redhat.com/
PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB

[Major Hayden]

On 2/7/24 07:36, Petr Menšík wrote:
>  I am interested why two DHCP clients are required inside a single
> tool. 
>  Why it could not use NetworkManager in both times, but with some 
>  different settings? Maybe alternative service? If there is missing 
>  ability to change behaviour, it may make sense to fix just that. Instead 
>  of keeping another client maintained. Do we support any kind of images, 
>  which would not have Network Manager present? 
One of the design goals of cloud-init is to use the dhcp client to get 
just enough networking done so the cloud instance can get to the cloud's 
metadata service. That metadata service provides information about 
network configuration, users, ssh keys, initial scripts, and more.

If NetworkManager is used for this step, then the network would need to 
be fully reconfigured when cloud-init finishes. It's my understanding 
that they block the boot until the network configuration is fully 
completed (using data from the metadata service) and then NetworkManager 
is allowed to continue.

>  If we had just NetworkManager-metadata.service, using alternate state
>
>  directory as well, couldn't it provide similar functionality as 
>  additional dhcpcd/dhclient? Just with few configuration files in 
>  addition, instead of additional compiled package(s)? 
That's certainly possible, but the cloud-init design goal seems to be to 
hold off on NetworkManager until the network is fully configured.

>  Are there simple steps, where I can as a dhcpcd maintainer try, how 
>  exactly is dhcpcd used during instance configuration process? Can I 
>  prepare VM image using mentioned metadata? Do we have any documentation 
>  on this topic for Fedora or RHEL at least? Can you point me to some 
>  documentation, how to test it? 
Sure! You can build a Fedora cloud instance on a cloud of your choice, 
and rawhide should be using dhclient by default. However, cloud-init has 
not done a release yet with full dhcpcd support.

Once cloud-init makes a new release with dhcpcd support, you can remove 
dhclient, add dhcpcd, and re-run cloud-init by doing the following:

   dnf remove dhcp-client
   dnf install dhcpcd
   cloud-init clean --logs --reboot

I hope that helps!

-- 
Major Hayden

[Stephen Smoogen]

On Wed, 7 Feb 2024 at 09:25, Major Hayden <major(a)mhtx.net&gt; wrote:

>  On 2/7/24 07:36, Petr Menšík wrote:
>  > I am interested why two DHCP clients are required inside a single tool.
>  > Why it could not use NetworkManager in both times, but with some
>  > different settings? Maybe alternative service? If there is missing
>  > ability to change behaviour, it may make sense to fix just that. Instead
>  > of keeping another client maintained. Do we support any kind of images,
>  > which would not have Network Manager present?
>
>  One of the design goals of cloud-init is to use the dhcp client to get
>  just enough networking done so the cloud instance can get to the cloud's
>  metadata service. That metadata service provides information about
>  network configuration, users, ssh keys, initial scripts, and more.
>
>  
cloud-init isn't used just for cloud systems. I have seen it used for
similar reasons in enterprise locations where the configs are centrally
located and Network-Manager's late start either sets up things which need
to be replaced again post-install or might be 'blocked' without the info.



>  If NetworkManager is used for this step, then the network would need
> to
>  be fully reconfigured when cloud-init finishes. It's my understanding
>  that they block the boot until the network configuration is fully
>  completed (using data from the metadata service) and then NetworkManager
>  is allowed to continue.
>  
-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[Tomasz Torcz]

On Tue, Jan 30, 2024 at 10:58:19AM -0600, Major Hayden wrote:
>  Hey there,
>  
>  For those of you who use Fedora often on public cloud providers where
>  cloud-init handles the initial configuration, there's a change coming for
>  the very early DHCP request during the boot. ☁️
>  
>  [0] https://bugzilla.redhat.com/show_bug.cgi?id=2247055 
  I'm curious, why not systemd-networkd? It's mentioned once in
the bug, but there's not explanation about the choice later.

-- 
Tomasz Torcz                “Funeral in the morning, IDE hacking
tomek(a)pipebreaker.pl         in the afternoon and evening.” - Alan Cox

[Major Hayden]

On Tue, Jan 30, 2024, at 11:11, Tomasz Torcz wrote:
>  On Tue, Jan 30, 2024 at 10:58:19AM -0600, Major Hayden wrote:
>    I'm curious, why not systemd-networkd? It's mentioned once in
>  the bug, but there's not explanation about the choice later. 
That might be possible, but upstream cloud-init only support dhclient, udhcpc, and dhcpcd
upstream currently. That support would need to be added in cloud-init.

-- 
Major Hayden