How to repair / rebuild GRUB following a cross-upgrade from CentOS 7 to AlmaLinux 8?

I essentially successfully moved through a cPanel ELevate migration of CentOS 7 to AlmaLinux 8 via their script and with some patience and elbow grease. This is on a Linode server.

After dealing with a Python issue (yum remove python36) and a strange additional error that didn’t seem clear, I found that the system just wouldn’t boot without some manual intervention.

Basically I have to manually enter boot args each time GRUB loads:

set root=(hd0)
linux /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 root=/dev/sda console=ttyS0,19200n8
initrd /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img
boot

I figured I could simply repair this after my final ELevate reboot sequence, but after initiating the following grub command, but it still doesn’t seem to recognize any Linux kernel images:

$ grub2-mkconfig --output /boot/grub2/grub.cfg
Generating grub configuration file ...
done

The good news is indeed that cPanel is running fine and the server seems stable, but I’m somewhat baffled by this behavior insomuch that clearly GRUB is not able to understand the new AlmaLinux kernel locations / structure properly…

What would be the proper way to build / rebuild and repair a GRUB installation on AlmaLinux 8?

Look at the generated /boot/grub2/grub.cfg. Does it have entries for the kernels in the ‘10_linux’ section, or does it have:

insmod blscfg
blscfg

This latter “BLS” is the default. It has entries as separate files in /boot/loader/entries that look like:

title AlmaLinux (4.18.0-372.9.1.el8.x86_64) 8.6 (Sky Tiger)
version 4.18.0-372.9.1.el8.x86_64
linux /vmlinuz-4.18.0-372.9.1.el8.x86_64
initrd /initramfs-4.18.0-372.9.1.el8.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id almalinux-20220510130458-4.18.0-372.9.1.el8.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

Note that the ‘options’ uses variable ‘$kernelopts’. IIRC, early RHEL 8 had hardcoded options there in some conditions. Should not affect you, but good to check.

In the grub.cfg, just before blscfg, there is probably:

if [ -z "${kernelopts}" ]; then
  set kernelopts="..."
fi

That is, if the ‘grubenv’ does not define ‘kernelopts’, then grub.cfg does. What is in grubenv?

sudo grub2-editenv list

The ‘grub2-mkconfig’ clearly updates grub.cfg, but if content of ‘grubenv’ has precedence, then the appropriate tool is probably ‘grub2-editenv’.

/boot/grub2/grub.cfg :: 10_linux section contains:

### BEGIN /etc/grub.d/10_linux ###
insmod ext2
set root='hd1'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd0 --hint-efi=hd0 --hint-baremetal=ahci0 --hint='hd1'  06e9a827-47b4-43f9-8e01-91b3852f86a8
else
  search --no-floppy --fs-uuid --set=root 06e9a827-47b4-43f9-8e01-91b3852f86a8
fi
insmod ext2
set boot='hd1'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=boot --hint-bios=hd0 --hint-efi=hd0 --hint-baremetal=ahci0 --hint='hd1'  06e9a827-47b4-43f9-8e01-91b3852f86a8
else
  search --no-floppy --fs-uuid --set=boot 06e9a827-47b4-43f9-8e01-91b3852f86a8
fi

# This section was generated by a script. Do not modify the generated file - all changes
# will be lost the next time file is regenerated. Instead edit the BootLoaderSpec files.
#
# The blscfg command parses the BootLoaderSpec files stored in /boot/loader/entries and
# populates the boot menu. Please refer to the Boot Loader Specification documentation
# for the files format: https://www.freedesktop.org/wiki/Specifications/BootLoaderSpec/.

# The kernelopts variable should be defined in the grubenv file. But to ensure that menu
# entries populated from BootLoaderSpec files that use this variable work correctly even
# without a grubenv file, define a fallback kernelopts variable if this has not been set.
#
# The kernelopts variable in the grubenv file can be modified using the grubby tool or by
# executing the grub2-mkconfig tool. For the latter, the values of the GRUB_CMDLINE_LINUX
# and GRUB_CMDLINE_LINUX_DEFAULT options from /etc/default/grub file are used to set both
# the kernelopts variable in the grubenv file and the fallback kernelopts variable.
if [ -z "${kernelopts}" ]; then
  set kernelopts="root=/dev/sda ro crashkernel=auto rhgb console=ttyS0,19200n8 net.ifnames=0 "
fi

insmod blscfg
blscfg
### END /etc/grub.d/10_linux ###

/boot/loader/entries/ does indeed have a file that seems to correspond to what should be booting:

title AlmaLinux (4.18.0-372.9.1.el8.x86_64) 8.6 (Sky Tiger)
version 4.18.0-372.9.1.el8.x86_64
linux /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64
initrd /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id almalinux-20220510130458-4.18.0-372.9.1.el8.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

As for grubenv, yeah, I’m sure you saw that above and so:

$ grub2-editenv list
saved_entry=4f09fa5fdd3642fa85221d7c11370603-4.18.0-372.9.1.el8.x86_64
kernelopts=root=/dev/sda ro crashkernel=auto rhgb console=ttyS0,19200n8 net.ifnames=0
boot_indeterminate=1

Where do you recommend I go from here?

Note: the grub2-mkconfig does construct kernelopts from value of GRUB_CMDLINE_LINUX that is set in /etc/default/grub (and some other things).

Based on what you have shown, the grub entry during boot should contain:

linux /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 root=/dev/sda ro crashkernel=auto rhgb console=ttyS0,19200n8 net.ifnames=0

In OP you did show:

linux /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 root=/dev/sda console=ttyS0,19200n8

Was that the entry’s state before or after your edits?

Since these two mismatch, one has to ask is the entry that you select at boot really the same entry that you do look at in config?

I’m sorry, in my original post I just kinda pieced that together out of desperation while trying to get the system to boot. It wasn’t necessarily accurate and not a permanent edit, just something I paste into the console that I got to work on their remote terminal.

There is NOTHING present for/in GRUB at boot, simply grub:

grub> _

My bad. I did read “boot args” as “kernelopts”, while you mean that Grub has no entries. The GRUB does not find (or fails to parse) the grub.cfg that you do have.

I haven’t manually reinstalled GRUB in ages.

Thanks so far. Since we’re kinda hitting a roadblock here I’ve tried to expand / generalize the reach: linux - How to recover from the GRUB losing its entries and faced with only "grub> _" at boot? - Server Fault

Also notice that I expanded some things within the ServerFault posting regarding EFI since perhaps the presence of /boot/efi/EFI/almalinux/ with no grub.cfg therein is a clue to what’s going on…

It turns out that the key to resolve this issue in the context of a Linode server was to disable GRUB_ENABLE_BLSCFG and rerun grub2-mkconfig, eg:

$ nano /etc/default/grub
### set: GRUB_ENABLE_BLSCFG=false
$ grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64
Found initrd image: /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-4f09fa5fdd3642fa85221d7c11370603
Found initrd image: /boot/initramfs-0-rescue-4f09fa5fdd3642fa85221d7c11370603.img
done

Useful information, nevertheless. On a system that does use legacy, rather than UEFI, the /boot/efi is not on separate “ESP” filesystem (a FAT32), and EFI-related GRUB2-packages are not installed, but a skeleton tree does come from grub2-common:

# rpm -qf /boot/efi/EFI/almalinux
grub2-common-2.02-123.el8.alma.noarch
# ls -R /boot/efi
/boot/efi:
EFI

/boot/efi/EFI:
almalinux

/boot/efi/EFI/almalinux:

In other words, that is just a red herring.

Your post on other site shows:

error: variable `prefix’ isn’t set.

The grub.cfg uses that variable:

# grep prefix grub.cfg
elif [ -s $prefix/grubenv ]; then
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;

What does set the ‘prefix’?

That says “grub-install”, but in GRUB2 that is “grub2-install”. man grub2-install and info grub2-install tell about it, but apparently the command to run is:

grub2-install /dev/sda

The peculiar situation here on Linode with this attempt is that it doesn’t seem to work perhaps since they inject GRUB 2 themselves and the drive isn’t necessarily configured normally (?):

$ grub2-install /dev/sda
Installing for i386-pc platform.
grub2-install: warning: File system `ext2' doesn't support embedding.
grub2-install: warning: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and their use is discouraged..
grub2-install: error: will not proceed with blocklists.

Is that also possible explanation: if migration does attempt grub2-install, then that step fails.