Cant install EL9, EL8 works fine?

Support
1

Hello there!
I was wondering whether anyone could provide some guidance on resolving an issue with installing EL9, the issue is that, after installing EL9, I get errors stating that the device does not exist, strangely, EL8 installs and works fine.

The error mentioned is as follows:
warning: UUID=xyz does not exist.

I can live boot and mount the partitions and have checked the UUID with lsblk -l and they match what fstab has inside, I also tried to change the fstab to point to /dev/sdax rather than the UUID, and get the same issue, further, I get the same with trying to change the grub boot parameters to say root=/dev/sdax.

The setup is using a hardware raid controller, I tried to change the bios from raid to ahci to see if this was the issue, but I still get the same error.

The hardware im trying to install on is a Dell poweredge R350, bios version 1.8.1, this is the latest bios version (shipped with the hardware)

I dont quite know where to go next, any suggestions or knowledge about this issue?
Thankyou

2

Is the UUID same as given in blkid? Does the system go in kernel panic or what’s the output in console?

1 Like
3

Hi Ankesh,
I can confirm that the UUID is the same under blkid.
the system does the following when attempting to boot.

grub shows almalinux el9 (I have tried a few subversions of EL9, and all have the same issue)
upon selecting almalinux, the boot process begins.

It hangs around after dracut initqueue hook for a while, If i have a usb hard drive in, it picks up the usb as /dev/sda but does not show a /dev/sdb after for the actual harddrive, additionally, if i remove the external usb, it doesnt pick up the hard drive as /dev/sda
What flags me as strange here is that the hard drive isnt being picked up, even though I can boot from it?

next the system goes into a dracut-initqueue loop, trying to find the device by uuid (or /dev/sdx if i set it to do so)

the error in the loop is, warning: dracut-initqueue: starting timeout scripts
after around a minute I get:

warning: /dev/disk/by-uuid/uuid of my root partition does not exist
(although, I can mount the drives fine by live boot as metioned)

followed by an “entering emergency mode”

Thanks!

4

Note: that is not “real” hardware RAID if you can switch mode like that. Some call it “fakeRAID”. All the operations are implemented in driver and run by CPU. What controller does it supposedly have? (Dell Support site can usually tell from the Service Tag.)

You can boot the EL9, so issue is not the x86-64-v2 microarchitecture level that the EL9 requires (SSE4.2, etc).

Your EL9 installer apparently does access the storage device fine, so there is no total lack of driver support.

Anyway, lspci -nn tells the details about devices that one usually does use to hunt drivers with. The disk controller is of interest.

1 Like
5

Looks to me like it might be a kernel issue as you stated. Either the hardware and EL9 is not compatible or your kernel might be missing a kernel module to detect the specific raid controller.
What’s the output of lsmod? Also, for the following:

lsmod | grep -i raid
lspci -nn | grep -i raid
1 Like
6

Intdresting, I did not know that thankyou.
The raid controller is a: “PERC H355 Adapter”
I will take a look into the drivers.

Many Thanks

7

Thanks for the input, I will take a look and see if it could be a module issue!

8

The Intel chipset is “fakeRAID”, but the PERC might be “real”.

According to /usr/share/hwdata/pci.ids Broadcom / LSI (1000) produces “MegaRAID 12GSAS/PCIe Secure SAS38xx” (10e6), device ID [1000:10e6]that is known with 14 different names/subversions:

# grep -B2 -A11 "PERC H355"  /usr/share/hwdata/pci.ids 
	10e6  MegaRAID 12GSAS/PCIe Secure SAS38xx
		1000 40e0  MegaRAID 9540-2M2
		1028 2172  PERC H355 Adapter
		1028 2173  PERC H355 Front
		1028 2174  PERC H350 Mini
		1028 2177  PERC H350 Adapter
		1028 2199  PERC H350 Mini LP
		15d9 1b9d  AOC-S3816L-L16IR Storage Adapter
		15d9 1b9f  AOC-S3816L-L8IR Storage Adapter
		15d9 1c6d  AOC-S3808L-L8IR Storage Adapter
		15d9 1c6e  AOC-SLG4-2H8M2 Storage Adapter
		1d49 0505  ThinkSystem RAID 540-8i PCIe Gen4 12Gb Adapter
		1d49 0506  ThinkSystem RAID 540-16i PCIe Gen4 12Gb Adapter
		1d49 0700  ThinkSystem M.2 SATA/NVMe 2-Bay Non-Hot-Swap RAID Enablement Kit
		1d49 0701  ThinkSystem 7mm SATA/NVMe 2-Bay Rear Hot-Swap RAID Enablement Kit

In the below there is an example output from one (Alma9) Poweredge system with plain lspci, lspci -nn, lspci -v for specific device, and entry from /usr/share/hwdata/pci.ids with subsystem id

# lspci | grep -E "RAID|SATA"
00:11.5 SATA controller: Intel Corporation C620 Series Chipset Family SSATA Controller [AHCI mode] (rev 09)
00:17.0 SATA controller: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] (rev 09)
3b:00.0 RAID bus controller: Broadcom / LSI MegaRAID Tri-Mode SAS3508 (rev 01)
3c:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

# lspci -nn | grep -E "RAID|SATA"
00:11.5 SATA controller [0106]: Intel Corporation C620 Series Chipset Family SSATA Controller [AHCI mode] [8086:a1d2] (rev 09)
00:17.0 SATA controller [0106]: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] [8086:a182] (rev 09)
3b:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID Tri-Mode SAS3508 [1000:0016] (rev 01)
3c:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] [1000:005d] (rev 02)

# lspci -s 3b:00.0 -nn -v
3b:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID Tri-Mode SAS3508 [1000:0016] (rev 01)
	Subsystem: Dell PERC H840 Adapter [1028:1fc9]
	Flags: bus master, fast devsel, latency 0, IRQ 37, NUMA node 0
	Memory at ab000000 (64-bit, prefetchable) [size=1M]
	Memory at ab100000 (64-bit, prefetchable) [size=1M]
	Memory at ab400000 (32-bit, non-prefetchable) [size=1M]
	I/O ports at 7000 [size=256]
	Expansion ROM at <ignored> [disabled]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [70] Express Endpoint, MSI 00
	Capabilities: [b0] MSI-X: Enable+ Count=128 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [148] Power Budgeting <?>
	Capabilities: [158] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [168] Secondary PCI Express
	Capabilities: [254] Dynamic Power Allocation <?>
	Capabilities: [284] Vendor Specific Information: ID=0002 Rev=1 Len=100 <?>
	Capabilities: [384] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
	Capabilities: [3bc] Physical Resizable BAR
	Kernel driver in use: megaraid_sas
	Kernel modules: megaraid_sas

# grep "1028 1fc9"  /usr/share/hwdata/pci.ids 
		1028 1fc9  PERC H840 Adapter

We see that “PERC H840 Adapter” has device ID 1000:0016 and subsystem 1028 1fc9.

Your “PERC H355 Adapter” has probably device ID 1000:10e6. That device ID seems to be supported by:

# modprobe -c | grep -i 1000.*10e6
alias pci:v00001000d000010E6sv*sd*bc*sc*i* megaraid_sas

kernel module megaraid_sas. (Alas, a module listing device ID does not guarantee proper support.)

The ELRepo does have package kmod-megaraid_sas, so there is at least some functionality that the Red Hat’s version of megaraid_sas lacks.

On a functioning system one could install the package:

dnf install elrepo-release
dnf install kmod-megaraid_sas

and after reboot the ELRepo version of the module should be in use. (The installation should inject the module into initramfs so that kernel has it before it attempts to mount filesystems).

The Dell PERC hardware RAID controllers – the arrays/storage – are usually configured via the iDRAC management, if the server has that option.

2 Likes
10

So ive managed to get this all working.
A big thankyou to everyone that helped point me in the right direction.

I messed around with lsmod in emergency mode in el9 and forced an emergency mode with el8, there were definitely some missing modules raid modules.
I proceeded to live boot, mount and chroot the broken install, gave myself network and a default route then attempted to install the megaraid package, after doing all of this, I was greeted with a error about being uable to mount sysroot, oops, back to the drawing board.

Well, I believe at this point the issue is with EL9 and a driver, and EL8 works fine,
so what happens if I try and upgrade 8?

So I installed back the EL8 version, updated it, then ran through the elevate process to upgrade from EL8 to EL9, it looks like the drivers are fine when going the elevate route, the machine finished the upgrade and I can use it absolutely no issues so far.

from my now working EL9:

lsmod | grep -i raid
megaraid_sas

modinfo megaraid_sas | less
filename: /lib/modules/5.14.0-362.18.1.el9_3.x86_64/extra/megaraid_sas/megaraid_sas.ko

(el9 in the filename, interesting, so it truly is using an el9 file)

lspci -nn | grep -i raid
01:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID 12GSAS/PCIe Secure SAS38xx [1000:10e6]

modprobe -c | grep -i megaraid | grep -i 10e6
alias pci:v00001000d000010E6svsdbcsci* megaraid_sas

so, something about the differences in the installer with EL8 and EL9 for the raid controller kernel drivers?

1 Like