Is NetworkManager-wait-online.service necessary by default?

[Chris Murphy]

Do any of Fedora desktop spins and Workstation edition need
NetworkManager-wait-online.service enabled by default?

Fedora 35 Workstation (updated, default "preset-all" service units)
$ systemd-analyze
Startup finished in 1.330s (kernel) + 1.284s (initrd) + 12.256s
(userspace) = 14.871s
graphical.target reached after 12.232s in userspace

Fedora 35 Workstation, same as above except
NetworkManager-wait-online.service is disabled
$ systemd-analyze
Startup finished in 1.294s (kernel) + 1.243s (initrd) + 5.704s
(userspace) = 8.242s
graphical.target reached after 5.670s in userspace

6.6s longer to wait for what? Is this service enabled just in case
someone adds an NFS or Samba mount to fstab? I'm not sure why this
service unit is enabled by default; and if we can either go without it
on the desktop, or if there's some other way to make it better,
because nearly doubling the boot time doesn't seem reasonable.


-- 
Chris Murphy

[Lennart Poettering]

On Di, 22.02.22 09:38, Chris Murphy (lists(a)colorremedies.com) wrote:

>  Do any of Fedora desktop spins and Workstation edition need
>  NetworkManager-wait-online.service enabled by default?
>
>  Fedora 35 Workstation (updated, default "preset-all" service units)
>  $ systemd-analyze
>  Startup finished in 1.330s (kernel) + 1.284s (initrd) + 12.256s
>  (userspace) = 14.871s
>  graphical.target reached after 12.232s in userspace
>
>  Fedora 35 Workstation, same as above except
>  NetworkManager-wait-online.service is disabled
>  $ systemd-analyze
>  Startup finished in 1.294s (kernel) + 1.243s (initrd) + 5.704s
>  (userspace) = 8.242s
>  graphical.target reached after 5.670s in userspace
>
>  6.6s longer to wait for what? Is this service enabled just in case
>  someone adds an NFS or Samba mount to fstab? I'm not sure why this
>  service unit is enabled by default; and if we can either go without it
>  on the desktop, or if there's some other way to make it better,
>  because nearly doubling the boot time doesn't seem reasonable. 
The idea is that it is only pulled into the bootup transaction if
something needs it. Typically that's stuff like NFS/SMB mounts listed
in /etc/fstab or so.

Note that the fact that the unit is enabled, doesn't mean it's
automatically in the boot transaction! That's because it (correctly)
hooks itself into network-online.target (rather than multi-user.target
or so) — and that target is only pulled in when something actually
needs it.

LTDR: figure our which service pulls in network-online.target for
you. A command like this should work:

    systemctl show -p WantedBy,RequiredBy network-online.target

Summary: NM-w-o.s being enabled is not a bug, if you use NM for
         network management.

         And: NM-w-o.s actually being pulled in *is* a bug, unless you
         have something like configured NFS/SMB mounts in /etc/fstab,
         that really cannot work without the network actually being
         up. There might be other valid candidates for this,
         i.e. other network client stuff that is really important for
         the boot to be up.

Lennart

--
Lennart Poettering, Berlin

[Lennart Poettering]

On Di, 22.02.22 18:08, Lennart Poettering (mzerqung(a)0pointer.de) wrote:

>           And: NM-w-o.s actually being pulled in *is* a bug, unless
> you
>           have something like configured NFS/SMB mounts in /etc/fstab,
>           that really cannot work without the network actually being
>           up. There might be other valid candidates for this,
>           i.e. other network client stuff that is really important for
>           the boot to be up. 
On one of my machines here I see that the following services pull it in:

    dnf-makecache.timer nfs-server.service rpc-statd-notify.service
    rpc-statd.service nfs-mountd.service

The DNF thing looks like an obvious bug to me. DNF's cache logic
really shouldn't add an expensive sync point to the boot like
this. Someone should file a bug to ask them to remove that.

The NFS stuff look all wrong to me too, i.e. NFS server stuff
afaics. That should not require the network to be fully up during
boot. But then again, NFS is weird, so what do I know?

Normally, unlikely client software, server software should really
watch rtnl or so and follow local network configuration to make its
services available, and thus it doesn't have to wait for the online
sync point...

Lennart

--
Lennart Poettering, Berlin

[Chris Murphy]

On Tue, Feb 22, 2022 at 10:14 AM Lennart Poettering
<mzerqung(a)0pointer.de&gt; wrote:
>  On Di, 22.02.22 18:08, Lennart Poettering (mzerqung(a)0pointer.de) wrote:
>
>  >          And: NM-w-o.s actually being pulled in *is* a bug, unless you
>  >          have something like configured NFS/SMB mounts in /etc/fstab,
>  >          that really cannot work without the network actually being
>  >          up. There might be other valid candidates for this,
>  >          i.e. other network client stuff that is really important for
>  >          the boot to be up.
>
>  On one of my machines here I see that the following services pull it in:
>
>      dnf-makecache.timer nfs-server.service rpc-statd-notify.service
>      rpc-statd.service nfs-mountd.service
>
>  The DNF thing looks like an obvious bug to me. DNF's cache logic
>  really shouldn't add an expensive sync point to the boot like
>  this. Someone should file a bug to ask them to remove that.
>
>  The NFS stuff look all wrong to me too, i.e. NFS server stuff
>  afaics. That should not require the network to be fully up during
>  boot. But then again, NFS is weird, so what do I know?
>
>  Normally, unlikely client software, server software should really
>  watch rtnl or so and follow local network configuration to make its
>  services available, and thus it doesn't have to wait for the online
>  sync point... 

OK so PackageKit, dnf, and nfs-utils.

In a default clean install, nfs-convert.service is enabled, but vendor
preset is disabled. I do not know why. When I use "systemctl
preset-all" it's not enabled. And I'm not finding any candidates in
/etc for a local change.


-- 
Chris Murphy

[Chris Murphy]

$ cat /usr/lib/systemd/system/dnf-makecache.timer
[Unit]
Description=dnf makecache --timer
ConditionKernelCommandLine=!rd.live.image
# See comment in dnf-makecache.service
ConditionPathExists=!/run/ostree-booted
Wants=network-online.target

[Timer]
OnBootSec=10min
OnUnitInactiveSec=1h
RandomizedDelaySec=60m
Unit=dnf-makecache.service

[Install]
WantedBy=timers.target
[chris@fovo ~]$


So what should the Wants be here, instead of network-online.target?
multiuser.target? I suppose they need to be someone tolerant of no
network being available during startup, it does seem to take a while
for wireless connections to succeed, but at least in my case it's up
by the time I've logged in.

Also the same thing for packagekit.service.

$ cat /usr/lib/systemd/system/packagekit.service
[Unit]
Description=PackageKit Daemon
# PK doesn't know how to do anything on ostree-managed systems;
# currently the design is to have dedicated daemons like
# eos-updater and rpm-ostree, and gnome-software talks to those.
ConditionPathExists=!/run/ostree-booted
Wants=network-online.target

[Service]
Type=dbus
BusName=org.freedesktop.PackageKit
User=root
ExecStart=/usr/libexec/packagekitd

======

$ grep -r "network-online" /usr/lib/systemd/system/
/usr/lib/systemd/system/NetworkManager-wait-online.service:Before=network-online.target
/usr/lib/systemd/system/NetworkManager-wait-online.service:WantedBy=network-online.target
/usr/lib/systemd/system/NetworkManager.service:#
WantedBy=network-online.target, so enabling it only has an effect if
/usr/lib/systemd/system/NetworkManager.service:# network-online.target
itself is enabled or pulled in by some other unit.
/usr/lib/systemd/system/auditd.service:## uncomment the second so that
network-online.target is part of After.
/usr/lib/systemd/system/auditd.service:##After=network-online.target
local-fs.target systemd-tmpfiles-setup.service
/usr/lib/systemd/system/cups-browsed.service:After=cups.service
avahi-daemon.service network-online.target
/usr/lib/systemd/system/cups-browsed.service:Wants=avahi-daemon.service
network-online.target
/usr/lib/systemd/system/dnf-makecache.service:After=network-online.target
/usr/lib/systemd/system/dnf-makecache.timer:Wants=network-online.target
/usr/lib/systemd/system/dnsmasq.service:;After=network-online.target
/usr/lib/systemd/system/iscsi.service:After=network.target
network-online.target iscsid.service iscsiuio.service
systemd-remount-fs.service
/usr/lib/systemd/system/iscsid.service:After=network-online.target
iscsiuio.service iscsi-init.service
/usr/lib/systemd/system/kdump.service:After=network.target
network-online.target remote-fs.target basic.target
/usr/lib/systemd/system/nfs-mountd.service:Wants=network-online.target
/usr/lib/systemd/system/nfs-mountd.service:After=network-online.target
local-fs.target
/usr/lib/systemd/system/nfs-server.service:Wants=rpcbind.socket
network-online.target
/usr/lib/systemd/system/nfs-server.service:After=network-online.target
local-fs.target
/usr/lib/systemd/system/openvpn-client@.service:After=syslog.target
network-online.target
/usr/lib/systemd/system/openvpn-client@.service:Wants=network-online.target
/usr/lib/systemd/system/openvpn-server@.service:After=syslog.target
network-online.target
/usr/lib/systemd/system/openvpn-server@.service:Wants=network-online.target
/usr/lib/systemd/system/packagekit.service:Wants=network-online.target
/usr/lib/systemd/system/podman-auto-update.service:Wants=network-online.target
/usr/lib/systemd/system/podman-auto-update.service:After=network-online.target
/usr/lib/systemd/system/rpc-statd-notify.service:Wants=network-online.target
/usr/lib/systemd/system/rpc-statd-notify.service:After=local-fs.target
network-online.target nss-lookup.target
/usr/lib/systemd/system/rpc-statd.service:Wants=network-online.target
/usr/lib/systemd/system/rpc-statd.service:After=network-online.target
nss-lookup.target rpcbind.socket
/usr/lib/systemd/system/systemd-networkd-wait-online.service:Before=network-online.target
shutdown.target
/usr/lib/systemd/system/systemd-networkd-wait-online.service:WantedBy=network-online.target
/usr/lib/systemd/system/systemd-networkd.service:#
WantedBy=network-online.target, so enabling it only has an effect if
/usr/lib/systemd/system/systemd-networkd.service:#
network-online.target itself is enabled or pulled in by some other
unit.


Is Wants=network-online.target the problem? Or also After=network-online.target?

It looks like there's potential for more service units causing this
problem than just the two I'm seeing.


--
Chris Murphy

[Zbigniew Jędrzejewski-Szmek]

On Tue, Feb 22, 2022 at 10:41:50AM -0700, Chris Murphy wrote:
>  $ cat /usr/lib/systemd/system/dnf-makecache.timer
>  [Unit]
>  Description=dnf makecache --timer
>  ConditionKernelCommandLine=!rd.live.image
>  # See comment in dnf-makecache.service
>  ConditionPathExists=!/run/ostree-booted
>  Wants=network-online.target
>  
>  [Timer]
>  OnBootSec=10min
>  OnUnitInactiveSec=1h
>  RandomizedDelaySec=60m
>  Unit=dnf-makecache.service
>  
>  [Install]
>  WantedBy=timers.target
>  [chris@fovo ~]$
>  
>  
>  So what should the Wants be here, instead of network-online.target?
>  multiuser.target? I suppose they need to be someone tolerant of no
>  network being available during startup, it does seem to take a while
>  for wireless connections to succeed, but at least in my case it's up
>  by the time I've logged in.
>  
>  Also the same thing for packagekit.service.
>  
>  $ cat /usr/lib/systemd/system/packagekit.service
>  [Unit]
>  Description=PackageKit Daemon
>  # PK doesn't know how to do anything on ostree-managed systems;
>  # currently the design is to have dedicated daemons like
>  # eos-updater and rpm-ostree, and gnome-software talks to those.
>  ConditionPathExists=!/run/ostree-booted
>  Wants=network-online.target
>  
>  [Service]
>  Type=dbus
>  BusName=org.freedesktop.PackageKit
>  User=root
>  ExecStart=/usr/libexec/packagekitd 
See my other mail: those two are not a problem.

In particular dnf-makecache.timer is started early (as part of
timers.target in early boot), but that's OK. At some point is fires
and dnf-makecache.service will be started, but that is usually a long
time after multi-user.target has been reached and the machine has
"finished booting"(*).

(*) Quotes are used because this shows that with multiple overlapping
targets the definition of "finished" is not clear.

>  ======
>  
>  $ grep -r "network-online" /usr/lib/systemd/system/
>  /usr/lib/systemd/system/NetworkManager-wait-online.service:Before=network-online.target
>
> /usr/lib/systemd/system/NetworkManager-wait-online.service:WantedBy=network-online.target
>  /usr/lib/systemd/system/NetworkManager.service:#
>  WantedBy=network-online.target, so enabling it only has an effect if
>  /usr/lib/systemd/system/NetworkManager.service:# network-online.target
>  itself is enabled or pulled in by some other unit.
>  /usr/lib/systemd/system/auditd.service:## uncomment the second so that
>  network-online.target is part of After.
>  /usr/lib/systemd/system/auditd.service:##After=network-online.target
>  local-fs.target systemd-tmpfiles-setup.service
>  /usr/lib/systemd/system/cups-browsed.service:After=cups.service
>  avahi-daemon.service network-online.target
>  /usr/lib/systemd/system/cups-browsed.service:Wants=avahi-daemon.service
>  network-online.target
>  /usr/lib/systemd/system/dnf-makecache.service:After=network-online.target
>  /usr/lib/systemd/system/dnf-makecache.timer:Wants=network-online.target
>  /usr/lib/systemd/system/dnsmasq.service:;After=network-online.target
>  /usr/lib/systemd/system/iscsi.service:After=network.target
>  network-online.target iscsid.service iscsiuio.service
>  systemd-remount-fs.service
>  /usr/lib/systemd/system/iscsid.service:After=network-online.target
>  iscsiuio.service iscsi-init.service
>  /usr/lib/systemd/system/kdump.service:After=network.target
>  network-online.target remote-fs.target basic.target
>  /usr/lib/systemd/system/nfs-mountd.service:Wants=network-online.target
>  /usr/lib/systemd/system/nfs-mountd.service:After=network-online.target
>  local-fs.target
>  /usr/lib/systemd/system/nfs-server.service:Wants=rpcbind.socket
>  network-online.target
>  /usr/lib/systemd/system/nfs-server.service:After=network-online.target
>  local-fs.target
>  /usr/lib/systemd/system/openvpn-client@.service:After=syslog.target
>  network-online.target
>  /usr/lib/systemd/system/openvpn-client@.service:Wants=network-online.target
>  /usr/lib/systemd/system/openvpn-server@.service:After=syslog.target
>  network-online.target
>  /usr/lib/systemd/system/openvpn-server@.service:Wants=network-online.target
>  /usr/lib/systemd/system/packagekit.service:Wants=network-online.target
>  /usr/lib/systemd/system/podman-auto-update.service:Wants=network-online.target
>  /usr/lib/systemd/system/podman-auto-update.service:After=network-online.target
>  /usr/lib/systemd/system/rpc-statd-notify.service:Wants=network-online.target
>  /usr/lib/systemd/system/rpc-statd-notify.service:After=local-fs.target
>  network-online.target nss-lookup.target
>  /usr/lib/systemd/system/rpc-statd.service:Wants=network-online.target
>  /usr/lib/systemd/system/rpc-statd.service:After=network-online.target
>  nss-lookup.target rpcbind.socket
>
> /usr/lib/systemd/system/systemd-networkd-wait-online.service:Before=network-online.target
>  shutdown.target
>
> /usr/lib/systemd/system/systemd-networkd-wait-online.service:WantedBy=network-online.target
>  /usr/lib/systemd/system/systemd-networkd.service:#
>  WantedBy=network-online.target, so enabling it only has an effect if
>  /usr/lib/systemd/system/systemd-networkd.service:#
>  network-online.target itself is enabled or pulled in by some other
>  unit. 
>  It looks like there's potential for more service units causing
> this
>  problem than just the two I'm seeing. 
There's probably some on this list that could lose the dependency,
but it's hard to figure out unless you know the implementation intimately.
I would ignore them unless an actual issue pops up ;)

Zbyszek

[Chris Murphy]

On Tue, Feb 22, 2022 at 10:50 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Tue, Feb 22, 2022 at 10:41:50AM -0700, Chris Murphy wrote:
>  > $ cat /usr/lib/systemd/system/dnf-makecache.timer
>  > [Unit]
>  > Description=dnf makecache --timer
>  > ConditionKernelCommandLine=!rd.live.image
>  > # See comment in dnf-makecache.service
>  > ConditionPathExists=!/run/ostree-booted
>  > Wants=network-online.target
>  >
>  > [Timer]
>  > OnBootSec=10min
>  > OnUnitInactiveSec=1h
>  > RandomizedDelaySec=60m
>  > Unit=dnf-makecache.service
>  >
>  > [Install]
>  > WantedBy=timers.target
>  > [chris@fovo ~]$
>  >
>  >
>  > So what should the Wants be here, instead of network-online.target?
>  > multiuser.target? I suppose they need to be someone tolerant of no
>  > network being available during startup, it does seem to take a while
>  > for wireless connections to succeed, but at least in my case it's up
>  > by the time I've logged in.
>  >
>  > Also the same thing for packagekit.service.
>  >
>  > $ cat /usr/lib/systemd/system/packagekit.service
>  > [Unit]
>  > Description=PackageKit Daemon
>  > # PK doesn't know how to do anything on ostree-managed systems;
>  > # currently the design is to have dedicated daemons like
>  > # eos-updater and rpm-ostree, and gnome-software talks to those.
>  > ConditionPathExists=!/run/ostree-booted
>  > Wants=network-online.target
>  >
>  > [Service]
>  > Type=dbus
>  > BusName=org.freedesktop.PackageKit
>  > User=root
>  > ExecStart=/usr/libexec/packagekitd
>
>  See my other mail: those two are not a problem. 
I'm lost on what is causing the problem, because I have a reproducible
doubling of boot times with NetworkManager-wait-online.service
enabled, versus not enabled. And
$ systemctl show -p WantedBy,RequiredBy network-online.target
RequiredBy=
WantedBy=packagekit.service dnf-makecache.timer

So it seems one or both of those is the cause of the problem; or I
don't understand the problem well enough to be involved in the
discussion (almost certainly true anyway).


-- 
Chris Murphy

[Gary Buhrmaster]

On Tue, Feb 22, 2022 at 5:42 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:

>  $ cat /usr/lib/systemd/system/packagekit.service
>  [Unit]
>  Description=PackageKit Daemon
>  # PK doesn't know how to do anything on ostree-managed systems;
>  # currently the design is to have dedicated daemons like
>  # eos-updater and rpm-ostree, and gnome-software talks to those.
>  ConditionPathExists=!/run/ostree-booted
>  Wants=network-online.target
>
>  [Service]
>  Type=dbus
>  BusName=org.freedesktop.PackageKit
>  User=root
>  ExecStart=/usr/libexec/packagekitd
>  
Older versions of the service file did
not include the Wants= in the package.
PackageKit 1.2.5 included the Wants=
But in any event, there is no After=.

Perhaps there are additional hints in:

   systemctl show -p WantedBy,RequiredBy,After,Before network-online.target

I also suspect i don't understand the
problem well enough to have the correct
clue to help.

[Chris Murphy]

On Tue, Feb 22, 2022 at 12:34 PM Gary Buhrmaster
<gary.buhrmaster(a)gmail.com&gt; wrote:
>  Perhaps there are additional hints in:
>
>     systemctl show -p WantedBy,RequiredBy,After,Before network-online.target
>
>  I also suspect i don't understand the
>  problem well enough to have the correct
>  clue to help. 

#     systemctl show -p WantedBy,RequiredBy,After,Before network-online.target
RequiredBy=
WantedBy=packagekit.service dnf-makecache.timer
Before=shutdown.target iscsid.service kdump.service
dnf-makecache.service iscsi.service
After=network.target NetworkManager-wait-online.service


-- 
Chris Murphy

[Lennart Poettering]

On Di, 22.02.22 14:36, Chris Murphy (lists(a)colorremedies.com) wrote:

>  On Tue, Feb 22, 2022 at 12:34 PM Gary Buhrmaster
>  <gary.buhrmaster(a)gmail.com&gt; wrote:
>  >
>  > Perhaps there are additional hints in:
>  >
>  >    systemctl show -p WantedBy,RequiredBy,After,Before network-online.target
>  >
>  > I also suspect i don't understand the
>  > problem well enough to have the correct
>  > clue to help.
>
>
>  #     systemctl show -p WantedBy,RequiredBy,After,Before network-online.target
>  RequiredBy=
>  WantedBy=packagekit.service dnf-makecache.timer
>  Before=shutdown.target iscsid.service kdump.service
>  dnf-makecache.service iscsi.service 
Are any of iscsid.service, iscsi.service, kdump.service actually
running for you? or enabled?

Most importantly though, let's take one step back: the dnf thing will
pull the service in, as mentioned, but not delay reaching of
multi-user.target or graphical.target. So, what are you actually
seeing? Will these two targets only be reached after NM-w-o.s is
reached? Or are they reached before it?

If the latter is the case, there actually is no problem, because yes,
NM-w-o.s will still run for a while after the relevant parts of the
boot completed, but it will not cause delays for anything that
matters, it just helps making dnf cache sync output a bit less noisy,
that's all. Or in other words: we consider the boot complete once
multi-user.target/graphical.target have been reached. Stuff that runs
after that is stuff we don't consider boot-up stuff anymore, but just
background stuff that runs after the actual boot phase is complete.

Lennart

--
Lennart Poettering, Berlin

[Chris Murphy]

On Tue, Feb 22, 2022 at 3:19 PM Lennart Poettering <mzerqung(a)0pointer.de&gt; wrote:
>  On Di, 22.02.22 14:36, Chris Murphy (lists(a)colorremedies.com) wrote:
>
>  > On Tue, Feb 22, 2022 at 12:34 PM Gary Buhrmaster
>  > <gary.buhrmaster(a)gmail.com&gt; wrote:
>  > >
>  > > Perhaps there are additional hints in:
>  > >
>  > >    systemctl show -p WantedBy,RequiredBy,After,Before network-online.target
>  > >
>  > > I also suspect i don't understand the
>  > > problem well enough to have the correct
>  > > clue to help.
>  >
>  >
>  > #     systemctl show -p WantedBy,RequiredBy,After,Before network-online.target
>  > RequiredBy=
>  > WantedBy=packagekit.service dnf-makecache.timer
>  > Before=shutdown.target iscsid.service kdump.service
>  > dnf-makecache.service iscsi.service
>
>  Are any of iscsid.service, iscsi.service, kdump.service actually
>  running for you? or enabled? 
None of those are running, all of them are enabled and have vendor
preset enabled as well.

kdump is coming from kexec-tools, which is installed due to anaconda
and abrt (looks like).

And the iscsi stuff is coming from iscsi-initiator-utils, which is
installed due to anaconda and gnome-boxes (looks like).



>  Most importantly though, let's take one step back: the dnf thing
> will
>  pull the service in, as mentioned, but not delay reaching of
>  multi-user.target or graphical.target. So, what are you actually
>  seeing? 
Longer time to get to the login window. If 'rhgb quiet' are removed, I
see the red cylon eye at NetworkManager-wait-online.service for maybe
3-10 seconds depending on whether I'm booting Fedora 35 or 36 (36 has
a ton of debug stuff enabled on my setup, so everything is slower).

systemd-analyze states double the time to get to graphical.target when
NetworkManager-wait-online.service is enabled.


> Will these two targets only be reached after NM-w-o.s is
>  reached? Or are they reached before it? 
I see iscsi related stuff before and after NM-w-o.s;  and kdump things
running before NM-w-o.s

journalctl -b | grep 'iscsi\|NetworkManager-wait-online\|kdump'
https://drive.google.com/file/d/1QhSs3v3dkQRG6hzKKyI1KGlc_La3afr4/view?us...


-- 
Chris Murphy

[Zbigniew Jędrzejewski-Szmek]

On Tue, Feb 22, 2022 at 08:26:22PM -0700, Chris Murphy wrote:
>  On Tue, Feb 22, 2022 at 3:19 PM Lennart Poettering
> <mzerqung(a)0pointer.de&gt; wrote:
>  >
>  > On Di, 22.02.22 14:36, Chris Murphy (lists(a)colorremedies.com) wrote:
>  >
>  > > On Tue, Feb 22, 2022 at 12:34 PM Gary Buhrmaster
>  > > <gary.buhrmaster(a)gmail.com&gt; wrote:
>  > > >
>  > > > Perhaps there are additional hints in:
>  > > >
>  > > >    systemctl show -p WantedBy,RequiredBy,After,Before
> network-online.target
>  > > >
>  > > > I also suspect i don't understand the
>  > > > problem well enough to have the correct
>  > > > clue to help.
>  > >
>  > >
>  > > #     systemctl show -p WantedBy,RequiredBy,After,Before network-online.target
>  > > RequiredBy=
>  > > WantedBy=packagekit.service dnf-makecache.timer
>  > > Before=shutdown.target iscsid.service kdump.service
>  > > dnf-makecache.service iscsi.service
>  >
>  > Are any of iscsid.service, iscsi.service, kdump.service actually
>  > running for you? or enabled?
>  
>  None of those are running, all of them are enabled and have vendor
>  preset enabled as well.
>  
>  kdump is coming from kexec-tools, which is installed due to anaconda
>  and abrt (looks like).
>  
>  And the iscsi stuff is coming from iscsi-initiator-utils, which is
>  installed due to anaconda and gnome-boxes (looks like).
>  
>  
>  
>  > Most importantly though, let's take one step back: the dnf thing will
>  > pull the service in, as mentioned, but not delay reaching of
>  > multi-user.target or graphical.target. So, what are you actually
>  > seeing?
>  
>  Longer time to get to the login window. If 'rhgb quiet' are removed, I
>  see the red cylon eye at NetworkManager-wait-online.service for maybe
>  3-10 seconds depending on whether I'm booting Fedora 35 or 36 (36 has
>  a ton of debug stuff enabled on my setup, so everything is slower). 
So yeah, if you see the cylon eye, something is definitely wrong.

>  journalctl -b | grep
> 'iscsi\|NetworkManager-wait-online\|kdump'
>  https://drive.google.com/file/d/1QhSs3v3dkQRG6hzKKyI1KGlc_La3afr4/view?us...
systemd[1]: iscsi.service: starting held back, waiting for: network-online.target

So this is the culprit. iscsi.service has Before=remote-fs-pre.target,
After=network-online.target, which means that it'll delay the boot.
remote-fs-pre.target is Before remote-fs.target which is Before multi-user.target.
(Also see the chart in bootup(7).)

There's a chain of deps that goes all the way to gnome-boxes and tmt:
iscsi-initiator-utils ← libvirt-daemon-driver-storage-iscsi ←
libvirt-daemon-driver-storage ← libvirt-daemon-kvm ← gnome-boxes,

iscsi-initiator-utils ← libvirt-daemon-driver-storage-iscsi ←
libvirt-daemon-driver-storage ← libvirt ← python3-testcloud ← tmt-provision-virtual
← tmt-all.

At least the dependency in gnome-boxes means it'll be installed
e.g. on Workstation.

I see three+ solutions:
a) change libvirt-daemon-driver-storage Requires:libvirt-daemon-driver-storage-iscsi
   to Suggests:libvirt-daemon-driver-storage-iscsi,
b) change the preset for iscsi.service to disabled,
c) drop the ordering in iscsi.service,
d) some combination of the above

I don't think we should have a service that delays boot in this way
installed and enabled by default. I'd vote for a), because iscsi is rather
specialized thing, and if you need it, you know you need it, and you can
install a package to get support. And b) requires an explicit command to
enable, so we might as well not install the package by default.

Zbyszek

[Kevin P. Fleming]

On Wed, Feb 23, 2022 at 4:25 PM Zbigniew Jędrzejewski-Szmek <
zbyszek(a)in.waw.pl&gt; wrote:

>  So this is the culprit. iscsi.service has
> Before=remote-fs-pre.target,
>  After=network-online.target, which means that it'll delay the boot.
>  remote-fs-pre.target is Before remote-fs.target which is Before
>  multi-user.target.
>  (Also see the chart in bootup(7).)
>
>  There's a chain of deps that goes all the way to gnome-boxes and tmt:
>  iscsi-initiator-utils ← libvirt-daemon-driver-storage-iscsi ←
>  libvirt-daemon-driver-storage ← libvirt-daemon-kvm ← gnome-boxes,
>
>  iscsi-initiator-utils ← libvirt-daemon-driver-storage-iscsi ←
>  libvirt-daemon-driver-storage ← libvirt ← python3-testcloud ←
>  tmt-provision-virtual
>  ← tmt-all.
>
>  At least the dependency in gnome-boxes means it'll be installed
>  e.g. on Workstation.
>
>  I see three+ solutions:
>  a) change libvirt-daemon-driver-storage
>  Requires:libvirt-daemon-driver-storage-iscsi
>     to Suggests:libvirt-daemon-driver-storage-iscsi,
>  b) change the preset for iscsi.service to disabled,
>  c) drop the ordering in iscsi.service,
>  d) some combination of the above
>
>  
Can confirm; 'systemctl disable iscsi.service' and a reboot of my F35
laptop resulted in a GDM login prompt before the network connection had
even been completed.

[Michael Catanzaro]

On Wed, Feb 23 2022 at 10:23:14 PM +0100, Zbigniew Jędrzejewski-Szmek 
<zbyszek(a)in.waw.pl&gt; wrote:
>  I'd vote for a) 
Me too.

I have a feeling this is going to dramatically improve our boot times. 
Shame we didn't notice this long ago. Oh well....

[Tom Hughes]

On 23/02/2022 21:23, Zbigniew Jędrzejewski-Szmek wrote:

>  a) change libvirt-daemon-driver-storage
> Requires:libvirt-daemon-driver-storage-iscsi
>      to Suggests:libvirt-daemon-driver-storage-iscsi, 
More generally why does installing libvirt neeed to force
installation of about ten storage drivers and all their
dependencies - why can't the user choose to remove some of
the more obscure ones?

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Cole Robinson]

On 2/23/22 5:22 PM, Tom Hughes via devel wrote:
>  On 23/02/2022 21:23, Zbigniew Jędrzejewski-Szmek wrote:
>  
> > a) change libvirt-daemon-driver-storage
> > Requires:libvirt-daemon-driver-storage-iscsi
> >     to Suggests:libvirt-daemon-driver-storage-iscsi,
>  
>  More generally why does installing libvirt neeed to force
>  installation of about ten storage drivers and all their
>  dependencies - why can't the user choose to remove some of
>  the more obscure ones?
>   
libvirt has a modularized packaging split. libvirt-daemon-driver-storage
pulls in every possible storage driver + lib that libvirt can use. Or
you can install libvirt-daemon-driver-storage-XXX individual sub
packages to get only the bits your app will use.

gnome-boxes which is in the default workstation package set has a dep on
libvirt-daemon-kvm which is itself a metapackage which pulls in
libvirt-daemon-driver-storage. gnome-boxes dep could be made more
granular, it uses libvirt storage APIs but I think only
libvirt-daemon-driver-storage-file

- Cole

[Tom Hughes]

On 24/02/2022 16:25, Cole Robinson wrote:
>  On 2/23/22 5:22 PM, Tom Hughes via devel wrote:
> > On 23/02/2022 21:23, Zbigniew Jędrzejewski-Szmek wrote:
> >
> >> a) change libvirt-daemon-driver-storage
> >> Requires:libvirt-daemon-driver-storage-iscsi
> >>      to Suggests:libvirt-daemon-driver-storage-iscsi,
> >
> > More generally why does installing libvirt neeed to force
> > installation of about ten storage drivers and all their
> > dependencies - why can't the user choose to remove some of
> > the more obscure ones?
> >
>  
>  libvirt has a modularized packaging split. libvirt-daemon-driver-storage
>  pulls in every possible storage driver + lib that libvirt can use. Or
>  you can install libvirt-daemon-driver-storage-XXX individual sub
>  packages to get only the bits your app will use. 
I was basing what I said on the fact that trying to remove
libvirt-daemon-driver-storage-iscsi wanted to remove the whole
of libvirt on my machine:

% sudo dnf erase libvirt-daemon-driver-storage-iscsi*

Dependencies resolved.

================================================================================

  Package                                 Arch   Version          Repo 
     Size

================================================================================

Removing:

  libvirt-daemon-driver-storage-iscsi     x86_64 7.6.0-5.fc35 
@updates  24 k

  libvirt-daemon-driver-storage-iscsi-direct

                                          x86_64 7.6.0-5.fc35 
@updates  32 k

Removing dependent packages:

  libvirt-daemon-kvm                      x86_64 7.6.0-5.fc35 
@updates   0

  vagrant-libvirt                         noarch 0.4.1-3.fc35 
@fedora  229 k

Removing unused dependencies:

  glusterfs-cli                           x86_64 9.5-1.fc35 
@updates 493 k

  guestfs-tools                           x86_64 1.47.3-1.fc35 
@updates  25 M

  hexedit                                 x86_64 1.5-2.fc35 
@fedora   77 k

  libacl                                  i686   2.3.1-2.fc35 
@fedora   35 k

  libglusterd0                            x86_64 9.5-1.fc35 
@updates  15 k

  libguestfs                              x86_64 1:1.46.2-1.fc35 
@updates 3.8 M

  libguestfs-appliance                    x86_64 1:1.46.2-1.fc35 
@updates 2.3 M

  libguestfs-xfs                          x86_64 1:1.46.2-1.fc35 
@updates   9

  libtpms                                 x86_64 
0.9.2-0.20220106gite81d634c27.fc35.0

 
@updates 986 k

  libvirt-daemon-driver-interface         x86_64 7.6.0-5.fc35 
@updates 593 k

  libvirt-daemon-driver-nodedev           x86_64 7.6.0-5.fc35 
@updates 650 k

  libvirt-daemon-driver-nwfilter          x86_64 7.6.0-5.fc35 
@updates 683 k

  libvirt-daemon-driver-qemu              x86_64 7.6.0-5.fc35 
@updates 2.5 M

  libvirt-daemon-driver-secret            x86_64 7.6.0-5.fc35 
@updates 585 k

  libvirt-daemon-driver-storage           x86_64 7.6.0-5.fc35 
@updates   0

  libvirt-daemon-driver-storage-core      x86_64 7.6.0-5.fc35 
@updates 767 k

  libvirt-daemon-driver-storage-disk      x86_64 7.6.0-5.fc35 
@updates  32 k

  libvirt-daemon-driver-storage-gluster   x86_64 7.6.0-5.fc35 
@updates  40 k

  libvirt-daemon-driver-storage-logical   x86_64 7.6.0-5.fc35 
@updates  32 k

  libvirt-daemon-driver-storage-mpath     x86_64 7.6.0-5.fc35 
@updates  16 k

  libvirt-daemon-driver-storage-rbd       x86_64 7.6.0-5.fc35 
@updates  44 k

  libvirt-daemon-driver-storage-scsi      x86_64 7.6.0-5.fc35 
@updates  24 k

  libvirt-daemon-driver-storage-sheepdog  x86_64 7.6.0-5.fc35 
@updates  20 k

  libvirt-daemon-driver-storage-zfs       x86_64 7.6.0-5.fc35 
@updates  24 k

  qemu-kvm-core                           x86_64 2:6.1.0-14.fc35 
@updates   0

  rubygem-excon                           noarch 0.79.0-1.fc35 
@fedora   98 k

  rubygem-fog-core                        noarch 2.2.4-2.fc35 
@fedora  118 k

  rubygem-fog-json                        noarch 1.2.0-7.fc35 
@fedora  3.9 k

  rubygem-fog-libvirt                     noarch 0.8.0-2.fc35 
@fedora   73 k

  rubygem-fog-xml                         noarch 0.1.3-7.fc35 
@fedora  8.3 k

  rubygem-formatador                      noarch 0.2.5-12.fc35 
@fedora  9.9 k

  rubygem-multi_json                      noarch 1.15.0-3.fc35 
@fedora   33 k

  rubygem-rexml                           noarch 3.2.5-151.fc35 
@fedora  399 k

  rubygem-ruby-libvirt                    x86_64 0.7.1-13.fc35 
@fedora  288 k

  supermin                                x86_64 5.3.1-1.fc35 
@fedora  1.5 M

  swtpm                                   x86_64 
0.7.0-2.20211109gitb79fd91.fc35

 
@updates 218 k

  swtpm-libs                              x86_64 
0.7.0-2.20211109gitb79fd91.fc35

 
@updates  99 k

  swtpm-tools                             x86_64 
0.7.0-2.20211109gitb79fd91.fc35

 
@updates 272 k

  zerofree                                x86_64 1.1.1-8.fc35 
@fedora   54 k

  zfs-fuse                                x86_64 0.7.2.2-20.fc35 
@fedora  6.0 M



Transaction Summary

================================================================================

Remove  44 Packages


Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Tom Hughes]

On 24/02/2022 16:28, Tom Hughes wrote:
>  On 24/02/2022 16:25, Cole Robinson wrote:
> > On 2/23/22 5:22 PM, Tom Hughes via devel wrote:
> >> On 23/02/2022 21:23, Zbigniew Jędrzejewski-Szmek wrote:
> >>
> >>> a) change libvirt-daemon-driver-storage
> >>> Requires:libvirt-daemon-driver-storage-iscsi
> >>>      to Suggests:libvirt-daemon-driver-storage-iscsi,
> >>
> >> More generally why does installing libvirt neeed to force
> >> installation of about ten storage drivers and all their
> >> dependencies - why can't the user choose to remove some of
> >> the more obscure ones?
> >>
> >
> > libvirt has a modularized packaging split. libvirt-daemon-driver-storage
> > pulls in every possible storage driver + lib that libvirt can use. Or
> > you can install libvirt-daemon-driver-storage-XXX individual sub
> > packages to get only the bits your app will use.
>  
>  I was basing what I said on the fact that trying to remove
>  libvirt-daemon-driver-storage-iscsi wanted to remove the whole
>  of libvirt on my machine: 
Ah but that is mostly auto clean.

Though libvirt-daemon-kvm is a dependency as you say and
don't I need that to run any kvm based vm?

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Daniel P. Berrangé]

On Thu, Feb 24, 2022 at 04:30:53PM +0000, Tom Hughes via devel wrote:
>  On 24/02/2022 16:28, Tom Hughes wrote:
>  > On 24/02/2022 16:25, Cole Robinson wrote:
>  > > On 2/23/22 5:22 PM, Tom Hughes via devel wrote:
>  > > > On 23/02/2022 21:23, Zbigniew Jędrzejewski-Szmek wrote:
>  > > > 
>  > > > > a) change libvirt-daemon-driver-storage
>  > > > > Requires:libvirt-daemon-driver-storage-iscsi
>  > > > >      to Suggests:libvirt-daemon-driver-storage-iscsi,
>  > > > 
>  > > > More generally why does installing libvirt neeed to force
>  > > > installation of about ten storage drivers and all their
>  > > > dependencies - why can't the user choose to remove some of
>  > > > the more obscure ones?
>  > > > 
>  > > 
>  > > libvirt has a modularized packaging split. libvirt-daemon-driver-storage
>  > > pulls in every possible storage driver + lib that libvirt can use. Or
>  > > you can install libvirt-daemon-driver-storage-XXX individual sub
>  > > packages to get only the bits your app will use.
>  > 
>  > I was basing what I said on the fact that trying to remove
>  > libvirt-daemon-driver-storage-iscsi wanted to remove the whole
>  > of libvirt on my machine:
>  
>  Ah but that is mostly auto clean. 
Yes, that is dnf being too aggressive in removing stuff.

>  Though libvirt-daemon-kvm is a dependency as you say and
>  don't I need that to run any kvm based vm? 
Correct, it is merely a metapackage that is intended to pull in all
libvirt sub-RPMs that are compatible with KVM usage.  If you remove
that package, you can choose exactly which individual sub-RPM features
you desire.

The same applies for QEMU, libvirt-daemon-kvm depends on qemu-kvm
which pulls in all the QEMU sub-RPMs that are compatible with KVM.
If you remove libvirt-daemon-kvm you can also choose a qemu-kvm-core
and whateever other features are desired.

We could none the less still consider use of Suggests in
libvirt-dameon-kvm. Will have a think if that could have a negative
impact on any apps currently depending on libvirt. 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Chris Adams]

Once upon a time, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; said:
>  So this is the culprit. iscsi.service has
> Before=remote-fs-pre.target,
>  After=network-online.target, which means that it'll delay the boot. 
If that's the problem, there's some other issue.  On my up-to-date F35
system, iscsi.service also has:

ConditionDirectoryNotEmpty=/var/lib/iscsi/nodes

And on my systems, that directory is empty, so iscsi.service shouldn't
be holding up anything.
-- 
Chris Adams <linux(a)cmadams.net&gt;

[Gary Buhrmaster]

On Wed, Feb 23, 2022 at 11:55 PM Chris Adams <linux(a)cmadams.net&gt; wrote:
>  Once upon a time, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; said:
>  > So this is the culprit. iscsi.service has Before=remote-fs-pre.target,
>  > After=network-online.target, which means that it'll delay the boot.
>
>  If that's the problem, there's some other issue.  On my up-to-date F35
>  system, iscsi.service also has:
>
>  ConditionDirectoryNotEmpty=/var/lib/iscsi/nodes
>
>  And on my systems, that directory is empty, so iscsi.service shouldn't
>  be holding up anything. 
How can one be sure that (in the general case) that
one of the units that you are running After will not
create the files/directories that will impact the test
Condition(s)?  Those tests occur before the unit
is actually started, but not before the ordering is
performed.

[Zbigniew Jędrzejewski-Szmek]

On Thu, Feb 24, 2022 at 12:17:27AM +0000, Gary Buhrmaster wrote:
>  On Wed, Feb 23, 2022 at 11:55 PM Chris Adams
> <linux(a)cmadams.net&gt; wrote:
>  >
>  > Once upon a time, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; said:
>  > > So this is the culprit. iscsi.service has Before=remote-fs-pre.target,
>  > > After=network-online.target, which means that it'll delay the boot.
>  >
>  > If that's the problem, there's some other issue.  On my up-to-date F35
>  > system, iscsi.service also has:
>  >
>  > ConditionDirectoryNotEmpty=/var/lib/iscsi/nodes
>  >
>  > And on my systems, that directory is empty, so iscsi.service shouldn't
>  > be holding up anything. 
Conditions are evaluated when the service would be exectued, so a unit
which is (eventually) skipped because of Conditions still has effect on
the boot ordering and may add additional jobs to the transaction.

systemd.unit(5) says:

  The conditions and asserts are checked at the time the queued
  start job is to be executed. The ordering dependencies are
  still respected, so other units are still pulled in and ordered
  as if this unit was successfully activated, and the conditions
  and asserts are executed the precise moment the unit would
  normally start and thus can validate system state after the
  units ordered before completed initialization.

>  How can one be sure that (in the general case) that one of the units
>  that you are running After will not create the files/directories
>  that will impact the test Condition(s)? 
We aren't. In fact the conditions are checked later as described above.

>  Those tests occur before the unit is actually started, but not
>  before the ordering is performed. 
Yes.

Zbyszek

[Benjamin Berg]

Hi,

On Thu, 2022-02-24 at 07:47 +0100, Zbigniew Jędrzejewski-Szmek wrote:
>  On Thu, Feb 24, 2022 at 12:17:27AM +0000, Gary Buhrmaster wrote:
>  > On Wed, Feb 23, 2022 at 11:55 PM Chris Adams <linux(a)cmadams.net&gt; wrote:
>  > > 
>  > > Once upon a time, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; said:
>  > > > So this is the culprit. iscsi.service has Before=remote-fs-pre.target,
>  > > > After=network-online.target, which means that it'll delay the boot.
>  > > 
>  > > If that's the problem, there's some other issue.  On my up-to-date F35
>  > > system, iscsi.service also has:
>  > > 
>  > > ConditionDirectoryNotEmpty=/var/lib/iscsi/nodes
>  > > 
>  > > And on my systems, that directory is empty, so iscsi.service shouldn't
>  > > be holding up anything.
>  
>  Conditions are evaluated when the service would be exectued, so a unit
>  which is (eventually) skipped because of Conditions still has effect on
>  the boot ordering and may add additional jobs to the transaction. 
We want to wait for network-online.target only if the service is
actually started. One way to achieve this might be to move waiting into
ExecStartPre=. I imagine this could work by creating a new unit

network-online-waitonly.target with
  After=network-online.target
  StopWhenUnneeded=yes

which is then used inside iscsi.service
  ExecStartPre=/usr/bin/systemctl start network-online-waitonly.target

Not great, but should work. I suppose the ideal solution would be that
the iscsi service handles waiting internally and only becomes ready
when the configured nodes are available or a timeout happens.

Benjamin

>  systemd.unit(5) says:
>  
>    The conditions and asserts are checked at the time the queued
>    start job is to be executed. The ordering dependencies are
>    still respected, so other units are still pulled in and ordered
>    as if this unit was successfully activated, and the conditions
>    and asserts are executed the precise moment the unit would
>    normally start and thus can validate system state after the
>    units ordered before completed initialization.
>  
>  > How can one be sure that (in the general case) that one of the units
>  > that you are running After will not create the files/directories
>  > that will impact the test Condition(s)?
>  
>  We aren't. In fact the conditions are checked later as described above.
>  
>  > Those tests occur before the unit is actually started, but not
>  > before the ordering is performed.
>  
>  Yes.
>  
>  Zbyszek
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Colin Walters]

On Thu, Feb 24, 2022, at 6:17 AM, Benjamin Berg wrote:

>  network-online-waitonly.target with
>    After=network-online.target
>    StopWhenUnneeded=yes
>
>  which is then used inside iscsi.service
>    ExecStartPre=/usr/bin/systemctl start network-online-waitonly.target 
No, avoid such things unless absolutely necessary - it makes the dependency graph dynamic,
which systemd does support but doing so brings a vast amount of complexity.

I think instead you can use e.g. a systemd generator:
https://www.freedesktop.org/software/systemd/man/systemd.generator.html
The generator (which can be the same binary) can then enable iscsi.service only if the
directory is non-empty.

(Which is making things dynamic, but all dynamic computation happens at a well-defined
eraly fixed point and is acted on together thereafter)

Now I'd agree this behavior is not obvious, and perhaps systemd should gain something
like
EnableConditionDirectoryNotEmpty= that is defined to be evaluated at the same time as
generators or so, and if the conditions evaluate to false then none of the unit
dependencies will be pulled in either.

[Benjamin Berg]

On Thu, 2022-02-24 at 10:14 -0500, Colin Walters wrote:
>  
>  On Thu, Feb 24, 2022, at 6:17 AM, Benjamin Berg wrote:
>  
>  > network-online-waitonly.target with
>  >   After=network-online.target
>  >   StopWhenUnneeded=yes
>  > 
>  > which is then used inside iscsi.service
>  >   ExecStartPre=/usr/bin/systemctl start network-online-waitonly.target
>  
>  No, avoid such things unless absolutely necessary - it makes the
>  dependency graph dynamic, which systemd does support but doing so
>  brings a vast amount of complexity. 
Yeah, we need something dynamic depending on whether a directory is
non-empty.

I must have been blind before, because thinking about it now, this is
exactly what .path units are for. All we should need is iscsi.path
watching for the directory to be non-empty and moving the install
section into the path unit.

With that, we should get exactly what we want at boot time. The only
side effect I see is that it will also trigger the service when the
first node is created. But that may fine, as it just triggers an
automatic login.

>  I think instead you can use e.g. a systemd generator:
>  https://www.freedesktop.org/software/systemd/man/systemd.generator.html
>  The generator (which can be the same binary) can then enable
>  iscsi.service only if the directory is non-empty. 
True, that is also a possible solution here. Maybe a bit better, though
it does move the logic to be outside of the systemd units, making it a
bit harder to see.

>  (Which is making things dynamic, but all dynamic computation happens
>  at a well-defined eraly fixed point and is acted on together
>  thereafter)
>  
>  Now I'd agree this behavior is not obvious, and perhaps systemd
>  should gain something like
>  EnableConditionDirectoryNotEmpty= that is defined to be evaluated at
>  the same time as generators or so, and if the conditions evaluate to
>  false then none of the unit dependencies will be pulled in either. 
Yeah, that sounds like an elegant solution to have a generic generator
to do the above.

Benjamin

>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Lennart Poettering]

On Do, 24.02.22 10:14, Colin Walters (walters(a)verbum.org) wrote:

>  I think instead you can use e.g. a systemd generator:
>  https://www.freedesktop.org/software/systemd/man/systemd.generator.html
>  The generator (which can be the same binary) can then enable iscsi.service only if the
> directory is non-empty. 
I would agree with this: I think the iscsi packagae should carry small
generator that pulls in the iscsi stack into the initial systemd
transaction if there's at least one entry in /var/lib/iscsi/nodes that
shall be set up at boot.

i.e. it should just create one symlink pulling iscsi.service into
remote-fs-pre.target if there's an iscsi device to set up. This could
be a shell script even, if you must. That way if people configure an
iscsi node to be set up at boot via that nodes file, then this is
enough to ensure it is really set up, and a secondary, manual
enablement of iscsi.service is not necessary. And of course, we
wouldn't enable iscsi.service by default anymore. To me this appears
to be the perfect behaviour.

Lennart

--
Lennart Poettering, Berlin

[Chris Adams]

Once upon a time, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; said:
>  Conditions are evaluated when the service would be exectued, so a
> unit
>  which is (eventually) skipped because of Conditions still has effect on
>  the boot ordering and may add additional jobs to the transaction. 
Okay, that's a nuance I didn't realize.  My guess would be that the
iscsi maintainer(s) didn't realize it either, and thought it would be
okay to have the service always enabled, so it would "just work" when
needed and also stay out of the way when not.

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Zbigniew Jędrzejewski-Szmek]

On Tue, Feb 22, 2022 at 10:32:52AM -0700, Chris Murphy wrote:
>  On Tue, Feb 22, 2022 at 10:14 AM Lennart Poettering
>  <mzerqung(a)0pointer.de&gt; wrote:
>  >
>  > On Di, 22.02.22 18:08, Lennart Poettering (mzerqung(a)0pointer.de) wrote:
>  >
>  > >          And: NM-w-o.s actually being pulled in *is* a bug, unless you
>  > >          have something like configured NFS/SMB mounts in /etc/fstab,
>  > >          that really cannot work without the network actually being
>  > >          up. There might be other valid candidates for this,
>  > >          i.e. other network client stuff that is really important for
>  > >          the boot to be up. 
Clarification: it *is* a bug *if* the unit that is ordered after it is
ordered before the default boot target. So e.g. something that is started
at boot but does not delay the default boot target is not a problem.

>  > On one of my machines here I see that the following services
> pull it in:
>  >
>  >     dnf-makecache.timer nfs-server.service rpc-statd-notify.service
>  >     rpc-statd.service nfs-mountd.service
>  >
>  > The DNF thing looks like an obvious bug to me. DNF's cache logic
>  > really shouldn't add an expensive sync point to the boot like
>  > this. Someone should file a bug to ask them to remove that. 
No, no. dnf-makecache.timer is "asynchronous" — it is not part of
multi-user.target. It is reasonable to order it after network is up
because this way it can just do its thing without spurious noise about
failed connections.

>  > The NFS stuff look all wrong to me too, i.e. NFS server stuff
>  > afaics. That should not require the network to be fully up during
>  > boot. But then again, NFS is weird, so what do I know?
>  >
>  > Normally, unlikely client software, server software should really
>  > watch rtnl or so and follow local network configuration to make its
>  > services available, and thus it doesn't have to wait for the online
>  > sync point... 
nfs works best when the network is up ;)
We probably could rewrite it to be more event-driven, but I guess
it's unlikely at this point.

>  OK so PackageKit, dnf, and nfs-utils.
>  
>  In a default clean install, nfs-convert.service is enabled, but vendor
>  preset is disabled. I do not know why. When I use "systemctl
>  preset-all" it's not enabled. And I'm not finding any candidates in
>  /etc for a local change. 
Zbyszek

[Gary Buhrmaster]

On Tue, Feb 22, 2022 at 5:43 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:

>  No, no. dnf-makecache.timer is "asynchronous" — it is not
> part of
>  multi-user.target. It is reasonable to order it after network is up
>  because this way it can just do its thing without spurious noise about
>  failed connections. 
It would seem to me that the service file should
have the Wants, not the timer (after all, it is
possible to start the service without using
a timer, and your argument about spurious
noise would apply there too).

[Lennart Poettering]

On Di, 22.02.22 18:42, Zbigniew Jędrzejewski-Szmek (zbyszek(a)in.waw.pl) wrote:

>  Clarification: it *is* a bug *if* the unit that is ordered after it
> is
>  ordered before the default boot target. So e.g. something that is started
>  at boot but does not delay the default boot target is not a problem. 
Indeed.

>  > > The DNF thing looks like an obvious bug to me. DNF's
> cache logic
>  > > really shouldn't add an expensive sync point to the boot like
>  > > this. Someone should file a bug to ask them to remove that.
>
>  No, no. dnf-makecache.timer is "asynchronous" — it is not part of
>  multi-user.target. It is reasonable to order it after network is up
>  because this way it can just do its thing without spurious noise about
>  failed connections. 
OK, I think this is indeed OK, then.

Lennart

--
Lennart Poettering, Berlin

[Lennart Poettering]

On Di, 22.02.22 10:32, Chris Murphy (lists(a)colorremedies.com) wrote:

>  > Normally, unlikely client software, server software should
> really
>  > watch rtnl or so and follow local network configuration to make its
>  > services available, and thus it doesn't have to wait for the online
>  > sync point...
>
>  OK so PackageKit, dnf, and nfs-utils.
>
>  In a default clean install, nfs-convert.service is enabled, but vendor
>  preset is disabled. I do not know why. When I use "systemctl
>  preset-all" it's not enabled. And I'm not finding any candidates in
>  /etc for a local change. 
We probably should have some form of release criterium that mandates
that on a default install (i.e. one where /etc/fstab has not been
added to include SMB/NFS mounts of so) it is not acceptable that
NM-w-o.s/network-online.target is pulled into the initial transaction
while being transitively ordered before multi-user.target or
graphical.target.

Because that's the issue as Zbigniew mentioned: not so much that
network-online.target is pulled in, but that it is pulled in *and*
something that actually delays the boot — i.e. that delays
multiuser.target or graphical.target being reached — is ordered after
it. Because that means multi-user.target/graphical.target cannot be
reached without the network-online.target also being reached.

By that standard dnf's use of network-online.target is not an issue,
as while dnf-makecache.service is ordered after it transitively, that
service is not ordered before multi-user.target/graphical.target, and
hence does *not* transitively delay m-u.t/g.t to be reached.

(And while we are at it, anything that pulls in and puts
systemd-udev-settle.service should similarly be prohibited).

Lennart

--
Lennart Poettering, Berlin

[Michael Catanzaro]

On Tue, Feb 22 2022 at 06:14:19 PM +0100, Lennart Poettering 
<mzerqung(a)0pointer.de&gt; wrote:
>      dnf-makecache.timer nfs-server.service rpc-statd-notify.service
>      rpc-statd.service nfs-mountd.service 
Another one: docker.service

[Chris Murphy]

On Tue, Feb 22, 2022 at 10:09 AM Lennart Poettering
<mzerqung(a)0pointer.de&gt; wrote:
>  On Di, 22.02.22 09:38, Chris Murphy (lists(a)colorremedies.com) wrote:
>
>  > Do any of Fedora desktop spins and Workstation edition need
>  > NetworkManager-wait-online.service enabled by default?
>  >
>  > Fedora 35 Workstation (updated, default "preset-all" service units)
>  > $ systemd-analyze
>  > Startup finished in 1.330s (kernel) + 1.284s (initrd) + 12.256s
>  > (userspace) = 14.871s
>  > graphical.target reached after 12.232s in userspace
>  >
>  > Fedora 35 Workstation, same as above except
>  > NetworkManager-wait-online.service is disabled
>  > $ systemd-analyze
>  > Startup finished in 1.294s (kernel) + 1.243s (initrd) + 5.704s
>  > (userspace) = 8.242s
>  > graphical.target reached after 5.670s in userspace
>  >
>  > 6.6s longer to wait for what? Is this service enabled just in case
>  > someone adds an NFS or Samba mount to fstab? I'm not sure why this
>  > service unit is enabled by default; and if we can either go without it
>  > on the desktop, or if there's some other way to make it better,
>  > because nearly doubling the boot time doesn't seem reasonable.
>
>  The idea is that it is only pulled into the bootup transaction if
>  something needs it. Typically that's stuff like NFS/SMB mounts listed
>  in /etc/fstab or so. 
Nothing like that is in fstab. Just some extra btrfs subvolumes
compared to default fstab.



>  LTDR: figure our which service pulls in network-online.target for
>  you. A command like this should work:
>
>      systemctl show -p WantedBy,RequiredBy network-online.target 
$ systemctl show -p WantedBy,RequiredBy network-online.target
RequiredBy=
WantedBy=packagekit.service dnf-makecache.timer
$




-- 
Chris Murphy