Stuck Changing perc 6 to perc H200 raid Controller on installed system

Support
1

Hi. I am in trouble.
This is a refreshment of an older ticket. It is too old to get noticed I think.

situation: exchanged a perc 6 sas controller for a perc H200 on a installed system.
After cloning the system partition of the old drive to a new RAID1 virtual disk, the system attempts to boot, but fails in dracut rescue mode (prompt)

question: How to change by chrooting from a live usb stick in my corrupt almalinux system.
uninstall the perc6 and install perc: H200 ?
I would really hate to have to reinstall the system only for choosing another raid controller. PS: Note that the ElRepo repository is installed on this active system.

I chrooted in the almalinux system by this command structure:

mount /dev/(drive here) /mnt
mount -o bind /sys /mnt/sys
mount -o bind /dev /mnt/dev
mount -o bind /proc /mnt/proc
chroot /mnt /bin/bash

and did:

lspci -m | grep -n SAS

It lists the SAS controller:

…
23:03:00.0 “Serial Attached SCSI controller” “Broadcom / LSI” “SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]” -r03 “Dell” “PERC H200 Integrated”
…

So, finally I was able to list the device ID of this card. And this link given by Toracat:

https://pci-ids.ucw.cz/read/PC/1000/0072

Contains the right driver Now all I need is a procedure of step by step commands that could be used to remove the old perc6 driver from the corrupt Almalinux system and replace it for this H200 driver , assuming the commands come from a ‘chrooted’ live usb stick system connecting the almalinux system drive.
By the way, I use a Debian rescue distribution on usb stick does it matter the host system is
not of same distribution as the chrooted system?

lsb_release -a

"ALMALINUX release 8.6 (sky Tiger)
release: 8.6
Codename : SkyTiger

Is there not anyone that has experienced this kind of situation?
You would save me hours of work to set up my almalinux system.

Thanks in advance. Bye. Dirk.

2

jlehtone.

if I do: lspci -nn -s 23:03:00.0

nothing is replied. empty.

update:

if i do :
lspci -m | grep -nn -s SAS

it gives the line like with lspci -m | grep -n SAS.
23:03:00.0 “Serial Attached SCSI controller” “Broadcom / LSI” “SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]” -r03 “Dell” “PERC H200 Integrated”

3

How about plain
lspci -nn

4

Hi Jlehtone,

Thanks for helping! I was very busy yesterday, late reply
Your command gives more information.
To filter out the card in the selection I combined:

lspci -nn | grep ‘SAS’
it gave:

[1000:0072] or the full line:

03:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)

https://pci-ids.ucw.cz/read/PC/1000/0072
This must be correct then I think.

5

The kernel seems to have driver ‘mpt3sas’ for the 1000:0072.
Removal of perc 6 driver should not be important.

What is important is to have the mpt3sas in the initramfs.
Simply regerating initramfs for the installed kernel might be sufficient.
With tool dracut: https://support.hpe.com/hpesc/public/docDisplay?docId=sf000081208en_us&docLocale=en_US

6

I am trying now…update comes soon

update:

My system refuses to boot in rescue mode it gives an error and returns to the grub menu.
When I try to boot the newest kernel, it stops after a while trying booting, then gives
‘dracut :/#’
I try to do the first copy instruction as method 2 discribes, but an error appears :slight_smile:
cannot stat '/boot/initr…(kernel) no such file or directory.

When i do ls, indeed i see i am not in the systemroot i see folders like ‘sysroot’ and a file: dracut-state.sh

How to reach the boot folder and copy the img file as the instruction says?

update: I see there is blkid in the sbin folder of this directory. I used blkid and I see 3 drives but not the system partition where my almalinux in installed. I am not sure I can comprehend all of this.

7

Well, I guess it is enough for this Sunday. I don’t have much alternative as this dracut mode. Maybe an ignorant question. Would it be possible to do this procedure from a chrooted takeover (via usb live stick)…?

8

If you do have installer USB, then you can boot with it, select the troubleshoot option. It will detect and mount the system partitions (if possible) and give a shell. Even tells how to chroot, IIRC.

9

Thanks Jlehtone, that opens new doors. I will try it. If the live stick is not the exact same version of almalinux, will it disturb the dracut -f tool result in any way?

10

Well, there is one problem I forgot when i started full of courage. Almalinux 8 + doesn’t see a H200 raid drive native. I would need to run this driver cd first some way.

update:

I tried to follow the instruction from the first post where you asked me to add a boot parameter. I booted the live usb, first menu I chose ’ repair an almalinux system’ from there there were only 2 options. I chose rescue. Before I did this I used ‘e’ on the keyboard (uefi system) and a new text screen appeared : parameters.
Also there was a choice to enter shell. So first i entered shell and typed:

inst.dd=https://elrepo.org/linux/dud/el8/x86_64/dd-mpt3sas-39.100.00.00-1.el8_6.elrepo.iso

then I hit the escape key and it went back to the parameter lines. Again the same line:
‘inst.dd=https://elrepo.org/linux/dud/el8/x86_64/dd-mpt3sas-39.100.00.00-1.el8_6.elrepo.iso’

Then I used ctrl+x as was written to start the system. After booting a rescue prompt screen came up. I used ‘lsblk -fm’ to see the partitions and drives.
Unlucky for me there were only the non RAID sas perc H200 partitions to be seen…

Is there any hint from here Available? I am not very experienced in handling environment variables like this, very well possible I made a mistake. How to maintain?

11

First, AlmaLinux 8 has ‘mpt3sas’ kernel module that should support 1000:0072, i.e. the H200.
Furthermore, the version of mpt3sas in Alma 8.7’s kernel-4.18.0-425.3.1 is 42.100.00.00, which sounds more recent than 39.100.00.00.
Two Alma 8.6’s kernels, 4.18.0-372.16.1 and 4.18.0-372.26.1, had mpt3sas 39.100.00.00.

The Alma’s installer should therefore support and see the H200 “out of the box”.

I did just test with installer USB (I don’t use Live images) although it was Alma 9.0’s installer and the system had only legacy BIOS. Hence on it, there too were “Troubleshoot” menu and in it “Rescue” entry. I got to edit the entry with ‘Tab’ in the legacy bootloader, corresponding to the ‘e’ for GRUB. (There were no “shell” option and I would not wanted GRUB shell anyway, just modify the boot entry.)

There was a line with words “vmlinuz”, “initrd=initrd.img”, and “inst.rescue” (among other things).
I should have added to that line the

inst.dd=https://elrepo.org/linux/dud/el8/x86_64/dd-mpt3sas-39.100.00.00-1.el8_6.elrepo.iso

that way the inst.dd=… is one of the kernel command-line parameters.

While one is editing that line, one could remove words “quiet” and “rhgb”, if they are there. Those options hide most of the bootup messages.

The inst.dd=URL assumes that the installer activates network. It does so on the install entries. I haven’t checked with the rescue mode. Some additional kernel command-line parameters might be required. (I’d peek at the regular installation boot entry for anything that contains “dhcp”.)

When one then boots the rescue entry, it should tell that it can search installed system (the primary option) or just drop one to shell. One should definitely choose the “seek & mount”.

If the kernel can access&find filesystems, then it will mount /, /boot, /home, etc under /mnt/sysroot directory and there will be instruction to run

chroot /mnt/sysroot

once you get to shell.

In the chrooted environment one sees only what is under /mnt/sysroot. That is, it looks like you are in / of your installed system. There are only programs that are installed on the system.

As second test I did boot with installer USB of CentOS Linux 7.9, in rescue mode, and successfully chrooted into the AlmaLinux 9.1 installation that was on that test machine.
I was thus able to use Alma 9’s tools, although I was running CentOS 7’s kernel.

So,

  • (Current) Alma 8 should have a driver for the controller
  • One start repair of Alma 8 setup with CentOS 7’s installer
12

Hi jlehtone,

First of all, thanks you for your time today and your attempt to help.
I spent an hour of 2 to get it to work. Unfortunately it seems only more questions rise up.

There is some pretty weird behaviour of the system here.

Ok. here is what I did. i took this iso to flash a installation 8.6 stick.

https://almalinux.cu.be/8.6/isos/x86_64/AlmaLinux-8.6-x86_64-minimal.iso

also I took this iso to flash a driver stick on a different usb stick.
(I didn’t use it, I followed your procedure.)
https://elrepo.org/linux/dud/el8/x86_64/dd-mpt3sas-39.100.00.00-1.el8_6.elrepo.iso

Up to now, I found that I do not only have a sas perc H200 driver problem (mpt3sas:0072:1000),
also after running the installer usb almalinux rescue stick has no network.
After it booted, I found pinging to 1.1.1.1 gives a no network error. Simply the installer stick
doesn’t see a network, so it will never find the iso configured in boot variables.

Talking about that, I did exactly what you described and that part worked out. I have found the line and I modified and added the part you told me to.
I know this time I used the right line, cause I removed “quiet” argument which was indeed listed as you predicted.
After running with (ctrl X) key, I saw the script running so, indeed it picked up the new line.

However, and now it comes. (logically) It didn’t pick up the driver.
Again "you have no linux system installed " when I entered rescue shell and used “1.” to let it search for a Almalinux system.
For testing I rebooted the stick a couple of times and tried again.
Now, believe me or not, after trying a couple of times, witohut any modification or setting params in boot, it booted up, I chose “1” and suddenly it DID SEE the sas vritual disk. Even without a connected network.
This is really weird inconsistent behaviour.a couple of times booting, no sas drive available then suddenly without any change it boots up and does see the sas drive. Weird.

So, I was very happy. Also you were right about the “mpt3sas” driver in almalinux 8.
I was able to chroot the system to /mnt/sysroot.
I followed the procedure method 2 on this site:
https://support.hpe.com/hpesc/public/docDisplay?docId=sf000081208en_us&docLocale=en_US
to run “dracut -f” and it seemed to work. for my kernel:
4.18.0-372.26.1.el8_6.x86_64 (8.6 skytiger)

Later I did use the same command to do it for an older kernel 8.6 version on my system to be sure.
I was hopeful.

the dracut tool finished the job and left me with a blanking cursor.
I rebooted the sytem and tried to boot my system.
Unfortunately it didn’t boot up as I was hoping. It booted to Dracut mode.
It didn’t see the sas drive…Stuck.
Errors came up, but this time errors for 2 devices:

first:
1.7537701 Warning: deprecated hardware is detected. bnx2:163B:14E4 @0000.0

and second:

Warning: DISABLED hardware is detected. mpt3sas:0072:1000 0000:03:00.0 is no longer enabled in this release.

Well, the latter one “mpt3sas:0072:1000” is a known friend. The system seems to know the sasdriver now, but does not accept it.
But why the hack almalinux 8.6 installer doesn’t see my integrated Broadcom network card is not understandable for me.
P.S the hardware is Dell T310 (with a perc H200 controller now)

That card worked since the beginning.
Before I migrated to almalinux I installed centos 8 and I don’t remember network problems.

So the result;

  1. Unstable when booting 8.6 installer. Most times “no almalinux system installed” Sometimes,: Yes, found it. I can chroot to /mnt/sysroot/

  2. After dracut -f the right image files in “/boot/initramfs-.img” i rebooted the installed almalinux system to find that SAS is not recognized with errors above.

Would it really make a difference from here to retry the process using a Centos7 installer?

13

Booting with CentOS 7 kernel would definitely offer different approach.

With the first warning message I found this: https://access.redhat.com/solutions/6663421

That links to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/considerations_in_adopting_rhel_8/index#removed-adapters_hardware-enablement

Warning about deprecation is thus not an issue. It just predicts that el9 will not have that driver.

Where the 1000:0072 is listed disabled from ‘mpt2sas’. That explains why ELRepo has one; while the ‘mpt3sas’ module in el8 kernel lists the ID, it is not for “I do support”, but for “I know this disabled model”. The ELRepo version – even with same version number – is probably different.

The installer is probably not set to activate network in rescue mode, like I suspected in previous post. Either additional kernel parameters or manually activating interface (with ip commands) might enable the network.

You have driver on USB. Did you try with that? Chapter 13. Updating drivers during installation Red Hat Enterprise Linux 8 | Red Hat Customer Portal

If that too does not occur on rescue mode, then – when in shell – one has to mount the USB, get kernel module from it, load it (with modprobe, I presume), and if controller them works, manually mount enough filesystems to be able to chroot, and install the ELRepo version of kernel module there, after which again try to recreate initrd image.

(I’m now on hypothetical side; not having done most of those in such circumstances.)

14

I’m confused now…

My main goal is to have my preinstalled system run the
mpt3sas:0072:1000
driver so it boots up, no? Should I still worry about the installer usb?
Your last tips are really over my head.

I tried to go parameter settings of the installed system.
To try " inst.dd=/dev/sdb" the id of the rpm driver usb stick.
Same result…

i think about giving this up.
What would be the easiest way to migrate this system to a fresh install?
I could copy the /etc/ folder of the active system, to have all my tweaks and settings in reserve and use config files. But i will loose much. KVM was installed. I need to copy guest linux systems that had their jobs. I created a complex network bridge to have KVM running .
And most of it I probably forgot to mention. Man that sucks.

Thanks for all your trying all the same It is much appreciated. .
I need some sleep now. Good night.

15

@leeuw

Hope you had a good night sleep. :slight_smile:

I’m sure you’ll find this tutorial video helpful. In the comment section someone using a system with H200 reported successful installation.

16

Thanks Toracat for hinting this.
What i actually tried to do, is change the driver for a already installed
running system, after changing from perc6 to H200 sas controller card.
Not installing a fresh system. I needed to keep my setup as it was, because it worked
perfect. i thought changing a controller card for another, would be a procedure that is known in the community for enterprise linux like almalinux and redhead likes.
I made a mistake.

2 years ago, when I installed centos 8 on my dell T310 -perc6 system, I used a method
like in the video. I had exactly the same problem then like this man in the video describes.
No drive to be seen to install to.
If I am not mistaken this only works when you are in the almalinux (centos8) o.s. installer.
Not in my situation of replacing a sas controller After installation.

17

After finishing the video. I am getting in doubt. On the 16 minute (0 seconds) there is the part of adding inst.dd behind the vmlinuz line. Exactly behind it. For sometimes it would not work otherwise. My menu is different on almalinux 8.6, but besides that.

i already tried the usb installer stick dracut -f procedure via usb to restore the system. It didn’t work. My active system doesn’t boot. Stops in dracut mode. No commands available.
Would this video help me really if I would pick the driver up on my recent corrupt installed system?

18

Yes.

I did test on system that requires ‘mptsas’. The el9_1 version dd-iso from ELRepo written to one USB (with ‘dd’), the Alma 9.1 installer-iso on another USB. Chose the “rescue” entry and did append the ‘inst.dd’ to the line that has the kernel options.

Just like in the video, the system listed devices as potential driver disks, although the list was empty and took couple refreshes to populate. I chose the correct system and then the RPM was available for selection. The process then offered to seek installed system as is usual for the rescue mode and did found the existing volumes on the array.

Furthermore, network was up too.

After entering the chroot one would have to install the ELRepo driver RPM for the system too.
Once system has the driver under /lib/modules/*/extra/one can run the dracut again.

After that one can use ‘lsinitrd’ to check what is written to image.
In my test system:

# lsinitrd /boot/initramfs-5.14.0-162.6.1.el9_1.x86_64.img | grep mptsas
-rw-r--r--   1 root     root          113 Oct 14 22:51 etc/depmod.d/kmod-mptsas.conf
drwxr-xr-x   2 root     root            0 Oct 14 22:51 usr/lib/modules/5.14.0-162.6.1.el9_1.x86_64/extra/mptsas
-rw-r--r--   1 root     root       147953 Oct 14 22:51 usr/lib/modules/5.14.0-162.6.1.el9_1.x86_64/extra/mptsas/mptsas.ko
-rw-r--r--   1 root     root        41652 Oct 14 22:51 usr/lib/modules/5.14.0-162.6.1.el9_1.x86_64/kernel/drivers/message/fusion/mptsas.ko.xz

The ‘extra/mptsas/mptsas.ko’ is from ELRepo and the ‘kernel/drivers/message/fusion/mptsas.ko.xz’ is from base repo. Only the former supports the card in that system.

1 Like
19

wow, very many thanks for this. I was not able to reply last days. I didn’t expect all help after last weekend. i will try this soon as possible. I will update you soon. Was there any specific reason to use 9.1? Not 8.6? I will just try both if it would not work here.

20

No reason other than that I had 9.1 handy. I did mostly skip the 8 – the lifetimes of 7 and 9 overlap enough.