Running AlmaLinux 8.6 on Linode and just did an update that resulted in a panic upon reboot.
To update, I did the following:
- Update:
$ yum update
Last metadata expiration check: 1:29:17 ago on Sat Aug 6 22:48:21 2022.
Dependencies resolved.
===========================================================================================================================
Package Architecture Version Repository Size
===========================================================================================================================
Installing:
kernel x86_64 4.18.0-372.19.1.el8_6 baseos 8.0 M
kernel-core x86_64 4.18.0-372.19.1.el8_6 baseos 39 M
kernel-debug-devel x86_64 4.18.0-372.19.1.el8_6 baseos 21 M
kernel-devel x86_64 4.18.0-372.19.1.el8_6 baseos 21 M
kernel-modules x86_64 4.18.0-372.19.1.el8_6 baseos 32 M
kernel-modules-extra x86_64 4.18.0-372.19.1.el8_6 baseos 8.7 M
Upgrading:
kernel-headers x86_64 4.18.0-372.19.1.el8_6 baseos 9.4 M
kernel-tools x86_64 4.18.0-372.19.1.el8_6 baseos 8.3 M
kernel-tools-libs x86_64 4.18.0-372.19.1.el8_6 baseos 8.1 M
Transaction Summary
===========================================================================================================================
Install 6 Packages
Upgrade 3 Packages
Total download size: 156 M
Is this ok [y/N]: y
Downloading Packages:
(1/9): kernel-4.18.0-372.19.1.el8_6.x86_64.rpm 27 MB/s | 8.0 MB 00:00
(2/9): kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64.rpm 38 MB/s | 21 MB 00:00
(3/9): kernel-devel-4.18.0-372.19.1.el8_6.x86_64.rpm 49 MB/s | 21 MB 00:00
(4/9): kernel-core-4.18.0-372.19.1.el8_6.x86_64.rpm 41 MB/s | 39 MB 00:00
(5/9): kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64.rpm 29 MB/s | 8.7 MB 00:00
(6/9): kernel-headers-4.18.0-372.19.1.el8_6.x86_64.rpm 21 MB/s | 9.4 MB 00:00
(7/9): kernel-tools-4.18.0-372.19.1.el8_6.x86_64.rpm 19 MB/s | 8.3 MB 00:00
(8/9): kernel-modules-4.18.0-372.19.1.el8_6.x86_64.rpm 32 MB/s | 32 MB 00:01
(9/9): kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64.rpm 42 MB/s | 8.1 MB 00:00
---------------------------------------------------------------------------------------------------------------------------
Total 77 MB/s | 156 MB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64 1/1
Installing : kernel-core-4.18.0-372.19.1.el8_6.x86_64 1/12
Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64 1/12
Installing : kernel-modules-4.18.0-372.19.1.el8_6.x86_64 2/12
Running scriptlet: kernel-modules-4.18.0-372.19.1.el8_6.x86_64 2/12
Upgrading : kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64 3/12
Running scriptlet: kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64 3/12
Upgrading : kernel-tools-4.18.0-372.19.1.el8_6.x86_64 4/12
Installing : kernel-4.18.0-372.19.1.el8_6.x86_64 5/12
Installing : kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64 6/12
Running scriptlet: kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64 6/12
Upgrading : kernel-headers-4.18.0-372.19.1.el8_6.x86_64 7/12
Installing : kernel-devel-4.18.0-372.19.1.el8_6.x86_64 8/12
Running scriptlet: kernel-devel-4.18.0-372.19.1.el8_6.x86_64 8/12
Installing : kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64 9/12
Running scriptlet: kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64 9/12
Cleanup : kernel-headers-4.18.0-372.9.1.el8.x86_64 10/12
Cleanup : kernel-tools-4.18.0-372.9.1.el8.x86_64 11/12
Cleanup : kernel-tools-libs-4.18.0-372.9.1.el8.x86_64 12/12
Running scriptlet: kernel-tools-libs-4.18.0-372.9.1.el8.x86_64 12/12
Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64 12/12
Running scriptlet: kernel-tools-libs-4.18.0-372.9.1.el8.x86_64 12/12
Verifying : kernel-4.18.0-372.19.1.el8_6.x86_64 1/12
Verifying : kernel-core-4.18.0-372.19.1.el8_6.x86_64 2/12
Verifying : kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64 3/12
Verifying : kernel-devel-4.18.0-372.19.1.el8_6.x86_64 4/12
Verifying : kernel-modules-4.18.0-372.19.1.el8_6.x86_64 5/12
Verifying : kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64 6/12
Verifying : kernel-headers-4.18.0-372.19.1.el8_6.x86_64 7/12
Verifying : kernel-headers-4.18.0-372.9.1.el8.x86_64 8/12
Verifying : kernel-tools-4.18.0-372.19.1.el8_6.x86_64 9/12
Verifying : kernel-tools-4.18.0-372.9.1.el8.x86_64 10/12
Verifying : kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64 11/12
Verifying : kernel-tools-libs-4.18.0-372.9.1.el8.x86_64 12/12
Upgraded:
kernel-headers-4.18.0-372.19.1.el8_6.x86_64 kernel-tools-4.18.0-372.19.1.el8_6.x86_64
kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64
Installed:
kernel-4.18.0-372.19.1.el8_6.x86_64 kernel-core-4.18.0-372.19.1.el8_6.x86_64
kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64 kernel-devel-4.18.0-372.19.1.el8_6.x86_64
kernel-modules-4.18.0-372.19.1.el8_6.x86_64 kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64
Complete!
- Update
GRUB 2
:
$ grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64
Found linux image: /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64
Found initrd image: /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-4f09fa5fdd3642fa85221d7c11370603
Found initrd image: /boot/initramfs-0-rescue-4f09fa5fdd3642fa85221d7c11370603.img
done
- Reboot (
$ reboot
) to/boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64
and see panic:
...
[ 0.638194] evm: Initialising EVM extended attributes:
[ 0.639435] evm: security.selinux
[ 0.640246] evm: security.ima
[ 0.640973] evm: security.capability
[ 0.641800] evm: HMAC attrs: 0x1
[ 0.643096] rtc_cmos 00:00: setting system clock to 2022-08-07 04:21:22 UTC )
[ 0.645303] md: Waiting for all devices to be available before autodetect
[ 0.646718] md: If you don't use raid, use raid=noautodetect
[ 0.648396] md: Autodetecting RAID arrays.
[ 0.649358] md: autorun ...
[ 0.650033] md: ... autorun DONE.
[ 0.650885] List of all partitions:
[ 0.651756] No filesystem could mount root, tried:
[ 0.651758]
[ 0.653332] Kernel panic - not syncing: VFS: Unable to mount root fs on unkn)
[ 0.655152] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.18.0-372.19.1.el8_6.1
[ 0.656845] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14
[ 0.659136] Call Trace:
[ 0.659754] dump_stack+0x41/0x60
[ 0.660532] panic+0xe7/0x2ac
[ 0.661292] mount_block_root+0x2be/0x2e2
[ 0.662252] ? do_early_param+0x91/0x91
[ 0.663080] prepare_namespace+0x135/0x16b
[ 0.663971] kernel_init_freeable+0x203/0x22d
[ 0.664917] ? rest_init+0xaa/0xaa
[ 0.665731] kernel_init+0xa/0xfb
[ 0.666510] ret_from_fork+0x35/0x40
[ 0.667415] Kernel Offset: 0x1ce00000 from 0xffffffff81000000 (relocation ra)
[ 0.669546] ---[ end Kernel panic - not syncing: VFS: Unable to mount root f-
It seems fairly evident that the problem is that it’s not finding the root volume, however the previous kernel (/boot/vmlinuz-4.18.0-372.9.1.el8.x86_64
) still boots fine if I select that instead. With an appropriate timeout I can pick the working kernel or I can issue this in GRUB for successful boot:
set root=(hd0)
linux /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 root=/dev/sda ro crashkernel=auto rhgb console=ttyS0,19200n8 net.ifnames=0
initrd /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img
boot
I found the numbering kind of odd insomuch that the one booting is ...9.1...
and the one failing is ...19.1...
- perhaps that’s to be expected, but odd in mind that they’re only 10
off:
Success: /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64
Failure: /boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64
In the /boot/
folder I see the following kernels and initrds present:
-rwxr-xr-x. 1 root root 10M May 10 09:04 vmlinuz-4.18.0-372.9.1.el8.x86_64
-rw-------. 1 root root 25M Jun 16 19:19 initramfs-4.18.0-372.9.1.el8.x86_64.img
-rwxr-xr-x. 1 root root 10M Aug 2 13:50 vmlinuz-4.18.0-372.19.1.el8_6.x86_64
The only glaring thing to me is that .19.
seems missing the initramfs
.
Any ideas about what may be going on here and how to resolve this panic situation with the mysterious missing root for .19.
?