Hello,
I have just installed Almalinux 9 (AlmaLinux-9.0-x86_64-minimal.iso) in proxmox vm , everthing seems all right. But as I installed it in bare metal server(Lenovo SR550) , something wrong happened.
After minimal install and reboot, system stuck in emergency mode. From output, it seems like mistakes in loading lvm volume.
Anyone can help me resolv this? It stuck me one week,thanks.
Server: Lenovo SR550, 4cpu, 32G memory
The SR550 is “not old”? On the machines that do have EFI and hence use GPT, I’ve stopped using LVM (except perhaps add PV later for KVM guests). Just plain partitions.
If the current installation is unusable anyway, then I would try that (storage scheme).
Thanks for your reply.
Use plain partitions may be a good resolution.I’ll try it.
But I want to know what cause this happening .Because we probably uses it to our production enviroment in the future .A scaleable root partitions (uses lvm) may be better for us.
Unfortunately, I just tried plain partition, the same error happened.This time the device path is
/dev/disk/by-uuid/…… does not exist
Ok, not really about LVM then. What is the model (device ID) of the storage controller? lspci -nn
That is a good point. I prefer my root tiny and then I mount additional volumes for services that do need space (e.g. database). Not as flexible as LVM, but separates core OS-files from data too. I also still use ext4 as it can shrink, unlike XFS.
All hard disks form raid5
sata controller
SATA controller [0106]: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] [8086:a182] (rev 09)
raid controller
RAID bus controller [0104]: Broadcom / LSI MegaRAID Tri-Mode SAS3508 [1000:0016] (rev 01)
Both of those do have a driver in Alma 9:
[AL9 ~]$ modprobe -c | grep 8086.*A182
alias pci:v00008086d0000A182sv*sd*bc*sc*i* ahci
[AL9 ~]$ modprobe -c | grep 1000.*0016
alias pci:v00001000d00000016sv*sd*bc*sc*i* megaraid_sas
However, ELRepo has megaraid_sas too:
[AL9 ~]$ dnf -q list kmod-megaraid_sas
Available Packages
kmod-megaraid_sas.x86_64 07.719.03.00-1.el9_0.elrepo elrepo
Does that just have support for more devices, or does it fix something too?
The version string in Alma’s copy is almost, but not quite the same:
[AL9 ~]$ modinfo megaraid_sas | grep ^version
version: 07.719.03.00-rc1
ELRepo has not only that RPM-file, but also the DUD images so that installer can load, mount, and install driver package from image with the inst.dd option.
If network works, then this added to kernel command line options would do the magic:
inst.dd=https://mirrors.maine.edu/elrepo/dud/el9/x86_64/dd-megaraid_sas-07.719.03.00-1.el9_0.elrepo.iso
(You probably have some other ELRepo mirror closer to you.)
If installer has no network, then image on USB is used. See Chapter 13. Updating drivers during installation Red Hat Enterprise Linux 9 | Red Hat Customer Portal
I didn’t look at specific messages, but I know that when I did my AL9 PVE VMs, I need to specify “host” CPU instead of “kvm64” or whatever the default is. If I don’t, I get a similar message to what you’re seeing. Not sure if this helps or not, but is easy to try.
That is probably about the x86_64-v2 requirement.
I assume that the hypervisor does not pass all CPU features to VM with that “kvm64” setting and that “host” setting passes as much as possible. The “kvm64” is thus a safe option as the VM can migrate to most hosts, while a VM on “host” setting can migrate only to hosts, who have at least all the features of the current CPU.
The OP has a bare metal host, so all of CPU is available and installer kernel would have halted on load if the CPU lacks the x86_64-v2 features.
Ah okay, I was getting myself mixed up in PVE vs Alma limitations. Thanks for correcting/clarifying!
As I tried your resolution, I found something strange that confused me.
I compare initrd.img and system modules files between Almalinux 9 and Almalinux 8.5 , I found that both of initrd.img are all have megaraid.ko
#almalinux8.5
[root@almalinux initrd]# lsinitrd initrd.img |grep mega
drwxr-xr-x 2 root root 0 Nov 9 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/scsi/megaraid
-rw-r--r-- 1 root root 69980 Nov 9 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
#almalinux9
[root@localhost ~]# lsinitrd initrd.img |grep -i megaraid
drwxr-xr-x 2 root root 0 May 18 01:05 usr/lib/modules/5.14.0-70.13.1.el9_0.x86_64/kernel/drivers/scsi/megaraid
-rw-r--r-- 1 root root 72336 May 17 16:02 usr/lib/modules/5.14.0-70.13.1.el9_0.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
but almalinux 8.5 has this driver file in system’s modules directory and it can be search by modprobe , 9 can’t
#almalinux8.5
[root@almalinux initrd]# modprobe -c | grep 1000.*0016
alias pci:v00001000d00000016sv*sd*bc*sc*i* megaraid_sas
#almalinux9
[root@localhost ~]# modprobe -c | grep 1000.*0016
[root@localhost ~]#
Above operation I did in my vms. I also search my physical server’s system file, megaraid driver file can’t be found,too.
As I tried your resolution, I found something strange that confused me.
I compare initrd.img and system modules files between Almalinux 9 and Almalinux 8.5 , I found that both of initrd.img are all have megaraid.ko
#almalinux8.5
[root@almalinux initrd]# lsinitrd initrd.img |grep mega
drwxr-xr-x 2 root root 0 Nov 9 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/scsi/megaraid
-rw-r--r-- 1 root root 69980 Nov 9 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
#almalinux9
[root@localhost ~]# lsinitrd initrd.img |grep -i megaraid
drwxr-xr-x 2 root root 0 May 18 01:05 usr/lib/modules/5.14.0-70.13.1.el9_0.x86_64/kernel/drivers/scsi/megaraid
-rw-r--r-- 1 root root 72336 May 17 16:02 usr/lib/modules/5.14.0-70.13.1.el9_0.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
but almalinux 8.5 has this driver file in system’s modules directory and it can be search by modprobe , 9 can’t
#almalinux8.5
[root@almalinux initrd]# modprobe -c | grep 1000.*0016
alias pci:v00001000d00000016sv*sd*bc*sc*i* megaraid_sas
#almalinux9
[root@localhost ~]# modprobe -c | grep 1000.*0016
[root@localhost ~]#
Above operation I did in my vms. I also search my physical server’s system file, megaraid driver file can’t be found,too.
As I tried your resolution, I found something strange that confused me.
I compare initrd.img and system modules files between Almalinux 9 and Almalinux 8.5 , I found that both of initrd.img are all have megaraid.ko
#almalinux8.5
[root@almalinux initrd]# lsinitrd initrd.img |grep mega
drwxr-xr-x 2 root root 0 Nov 9 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/scsi/megaraid
-rw-r--r-- 1 root root 69980 Nov 9 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
#almalinux9
[root@localhost ~]# lsinitrd initrd.img |grep -i megaraid
drwxr-xr-x 2 root root 0 May 18 01:05 usr/lib/modules/5.14.0-70.13.1.el9_0.x86_64/kernel/drivers/scsi/megaraid
-rw-r--r-- 1 root root 72336 May 17 16:02 usr/lib/modules/5.14.0-70.13.1.el9_0.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
but almalinux 8.5 has this driver file in system’s modules directory and it can be search by modprobe , 9 can’t
#almalinux8.5
[root@almalinux initrd]# modprobe -c | grep 1000.*0016
alias pci:v00001000d00000016sv*sd*bc*sc*i* megaraid_sas
#almalinux9
[root@localhost ~]# modprobe -c | grep 1000.*0016
[root@localhost ~]#
Above operation I did in my vms. I also search my physical server’s system file, megaraid driver file can’t be found,too.
Thanks a lot, it works!
But as I tried your resolution, I found something strange that confused me.
I compare initrd.img and system modules files between Almalinux 9 and Almalinux 8.5 , I found that both of initrd.img are all have megaraid_sas.ko
#almalinux8.5
[root@almalinux initrd]# lsinitrd initrd.img |grep mega
drwxr-xr-x 2 root root 0 Nov 9 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/scsi/megaraid
-rw-r--r-- 1 root root 69980 Nov 9 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
#almalinux9
[root@localhost ~]# lsinitrd initrd.img |grep -i megaraid
drwxr-xr-x 2 root root 0 May 18 01:05 usr/lib/modules/5.14.0-70.13.1.el9_0.x86_64/kernel/drivers/scsi/megaraid
-rw-r--r-- 1 root root 72336 May 17 16:02 usr/lib/modules/5.14.0-70.13.1.el9_0.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
but almalinux 8.5 has this driver file in system’s modules directory and it can be search by modprobe , 9 can’t
#almalinux8.5
[root@almalinux initrd]# modprobe -c | grep 1000.*0016
alias pci:v00001000d00000016sv*sd*bc*sc*i* megaraid_sas
#almalinux9
[root@localhost ~]# modprobe -c | grep 1000.*0016
[root@localhost ~]#
So what causes it ? Should I install extra raid driver instead of iso build-in driver(if it has) every time I install Almalinux9?
That is fascinating.
The installer’s initrd.img does indeed have the module (as it probably has all of them in order to function on all supported hardware).
[AL9 installer]$ lsinitrd initrd.img | grep mega
drwxr-xr-x 2 root root 0 May 18 08:05 usr/lib/modules/5.14.0-70.13.1.el9_0.x86_64/kernel/drivers/scsi/megaraid
-rw-r--r-- 1 root root 72336 May 17 23:02 usr/lib/modules/5.14.0-70.13.1.el9_0.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
The installer (kernel-5.14.0-70.13.1) should load the necessary modules and thus be able to use devices.
If we supply additional drivers, then the kernel can load them too.
How it prefers megaraid_sas from DUD, when it already has one in initrd.img? I don’t know.
In addition to use of driver during install, the kmod-* RPM from DUD should also install into the system automagickally.
I don’t have that hardware, so the initramfs*.img of installed kernel (5.14.0-70.17.1) on my machines does not have it, but system has it as it is provided by kernel-modules package:
[AL9]$ ls -l /lib/modules/5.14.0-70.17.1.el9_0.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
-rw-r--r--. 1 root root 72464 Jun 28 22:07 /lib/modules/5.14.0-70.17.1.el9_0.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz
What is interesting in your “modprobe -c” output is that it finds nothing for [1000:0016], while on my Alma 9 the ‘megaraid_sas’ is listed both by default and if I install the kmod-megaraid_sas from ELRepo.
The real question related to future installs is why the included driver fails. Is the issue only in 5.14.0-70.13.1’s module and fixed in future kernels – in which case future installers (e.g. for Alma 9.1) would behave – or will it persist. Red Hat’s bugzilla does not show any entries for “megaraid”.
I tend to (re)install with PXEboot, which allows setting custom boot options on a server for multiple machines, although latest CentOS 7 installs had to use removable media as those machines had NIC that does need driver from ELRepo.
One thing to test (but might break the system) is to remove the kmod-megaraid_sas from already installed system. If the 5.14.0-70.17.1 can boot with its own megaraid_sas module (which obviously has to be injected into initramfs after removal of kmod-package and before reboot), then the issue is only in 5.14.0-70.13.1.