Almalinux 9.3 NetworkManager getting disabled after reboot

Support
1

Hello,
I have installed Almalinux 9.3 from the minimal iso and noticed that after a reboot, NetworkManager is disabled (I have enabled and started it previously). I enable it again, and after a reboot it is disabled again. I have tried various “fixes” found online, but without any luck.

My server is a VM on a KVM hypervisor. Fresh, minimal OS installation. Guest tools are installed and working. Other than the described issue, everything else appears to be working.

Grateful for any ideas that help resolve this problem,

D.

2

Can you show the output of systemctl status NetworkManager before and after reboot?

3

Hello, Thanks for replying. Output is bellow. I didn’t notice errors, it feels like something is “undo’ing” systemctl enable NetworkManager during boot/reboot.

Before reboot:

# systemctl status NetworkManager
○ NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:NetworkManager(8)


# systemctl is-enabled NetworkManager
disabled

# systemctl is-active NetworkManager
inactive

# systemctl enable NetworkManager
Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service → /usr/lib/systemd/system/NetworkManager.service.
Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service → /usr/lib/systemd/system/NetworkManager-dispatcher.service.
Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service → /usr/lib/systemd/system/NetworkManager-wait-online.service.

# systemctl start NetworkManager

# systemctl status NetworkManager
● NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-04-16 16:41:24 EEST; 7s ago
       Docs: man:NetworkManager(8)
   Main PID: 1107 (NetworkManager)
      Tasks: 4 (limit: 11137)
     Memory: 16.8M
        CPU: 137ms
     CGroup: /system.slice/NetworkManager.service
             └─1107 /usr/sbin/NetworkManager --no-daemon

Apr 16 16:41:24 my.hostname.tld NetworkManager[1107]: <info>  [1713274884.0993] policy: set 'prl_nettool-nm-ens3' (ens3) as default for IPv6 routing and DNS
Apr 16 16:41:24 my.hostname.tld NetworkManager[1107]: <info>  [1713274884.1274] device (lo): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
Apr 16 16:41:24 my.hostname.tld NetworkManager[1107]: <info>  [1713274884.1300] device (lo): Activation: successful, device activated.
Apr 16 16:41:25 my.hostname.tld NetworkManager[1107]: <info>  [1713274885.9113] device (ens3): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Apr 16 16:41:25 my.hostname.tld NetworkManager[1107]: <info>  [1713274885.9185] device (ens3): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Apr 16 16:41:25 my.hostname.tld NetworkManager[1107]: <info>  [1713274885.9190] device (ens3): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Apr 16 16:41:25 my.hostname.tld NetworkManager[1107]: <info>  [1713274885.9198] manager: NetworkManager state is now CONNECTED_SITE
Apr 16 16:41:25 my.hostname.tld NetworkManager[1107]: <info>  [1713274885.9206] device (ens3): Activation: successful, device activated.
Apr 16 16:41:25 my.hostname.tld NetworkManager[1107]: <info>  [1713274885.9266] manager: NetworkManager state is now CONNECTED_GLOBAL
Apr 16 16:41:25 my.hostname.tld NetworkManager[1107]: <info>  [1713274885.9283] manager: startup complete

After reboot:

# systemctl status NetworkManager
○ NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:NetworkManager(8)

Apr 16 16:43:00 my.hostname.tld NetworkManager[516]: <info>  [1713274980.1821] manager: NetworkManager state is now CONNECTED_SITE
Apr 16 16:43:00 my.hostname.tld NetworkManager[516]: <info>  [1713274980.1838] device (ens3): Activation: successful, device activated.
Apr 16 16:43:00 my.hostname.tld NetworkManager[516]: <info>  [1713274980.1865] manager: NetworkManager state is now CONNECTED_GLOBAL
Apr 16 16:43:00 my.hostname.tld NetworkManager[516]: <info>  [1713274980.1907] manager: startup complete
Apr 16 16:43:05 my.hostname.tld NetworkManager[516]: <info>  [1713274985.3597] caught SIGTERM, shutting down normally.
Apr 16 16:43:05 my.hostname.tld NetworkManager[516]: <info>  [1713274985.3623] manager: NetworkManager state is now CONNECTED_SITE
Apr 16 16:43:05 my.hostname.tld systemd[1]: Stopping Network Manager...
Apr 16 16:43:05 my.hostname.tld NetworkManager[516]: <info>  [1713274985.3705] exiting (success)
Apr 16 16:43:05 my.hostname.tld systemd[1]: NetworkManager.service: Deactivated successfully.
Apr 16 16:43:05 my.hostname.tld systemd[1]: Stopped Network Manager.

# systemctl is-enabled NetworkManager
disabled
4

That shows: disabled, like you had not ran the systemctl enable NetworkManager

1 Like
5

Hi. Right, I messed up the output here. I have updated my comment to “quartsize”, should be good now (also added some more output for context).

6

Anything potentially related in dmesg, journalctl, and /var/log/messages ?
“Just a KVM”, so no cloud-init or such?

7

Right, no cloud-init. Other than a few warnings / errors, the logs are really uneventful.

Warning: the ifcfg-rh plugin is deprecated

Apr 16 17:24:18 my qemu-ga[545]: info: guest-exec-status failed, pid: 1060 not found


Apr 16 17:25:57 wazuh dbus-broker-launch[505]: Activation request for 'org.freedesktop.nm_dispatcher' failed: The systemd unit 'dbus-org.freedesktop.nm-dispatcher.service' could not be found.

Disabled the ifcfg-rh plugin and selinux, to see if it changes anything. Unfortunately not.
Will have to get back later to check the other two.

8

Yeah sorta along the lines of what jlehtone is saying, but more specifically

Apr 16 16:43:05 my.hostname.tld NetworkManager[516]: <info>  [1713274985.3597] caught SIGTERM, shutting down normally.

looks like the turning point, like my read is that NetworkManager is enabled and starts up when you reboot, but something’s killing it (and ostensibly also disabling it) five seconds later. I’d be wondering if you can find anything illuminating in the full system journal around that timestamp, like is there some other service that’s starting that might be responsible for killing and disabling NetworkManager, that sort of thing.

9

Hello,

I have been busy with other tasks lately, but yesterday I found some time and got back to this issue. And actually I found a solution.

To put it shortly - the problem was our outdated hypervisor / guest tools. We have some compatibility issues on a few of our servers and have to hold back updates for certain packages including guest tools).

I was able to resolve the NetworkManager issue by replacing the outdated guest tool scripts with the newer ones. Now NetworkManager behaves normally.

The “fix”.
Inside the VM (with outdated hypervisor):
Replace all scripts in “/usr/lib/parallels-tools/tools/scripts/”

with the newer scripts “/usr/lib/vz-tools/tools/scripts/”
(these scripts are inside the VM with updated hypervisor).

Hopefully, this info will help someone save some time (probably not :sweat_smile:).

Cheers!