AlmaLinux 9 - Server Key Generation was skipped because of a failed condition check

First time playing with AlmaLinux 9 coming from CentOS 7 and AlmaLinux 8 experience and I am noticing these messages in /var/log/messages on a fresh install on Vultr KVM VPS server.

Aug 29 16:46:31 almalinux9 systemd[1]: OpenSSH ecdsa Server Key Generation was skipped because of a failed condition check (ConditionPathExists=!/run/systemd/generator.early/multi-user.target.wants/cloud-init.target).
Aug 29 16:46:31 almalinux9 systemd[1]: OpenSSH ed25519 Server Key Generation was skipped because of a failed condition check (ConditionPathExists=!/run/systemd/generator.early/multi-user.target.wants/cloud-init.target).
Aug 29 16:46:31 almalinux9 systemd[1]: OpenSSH rsa Server Key Generation was skipped because of a failed condition check (ConditionPathExists=!/run/systemd/generator.early/multi-user.target.wants/cloud-init.target).
Aug 29 16:46:31 almalinux9 systemd[1]: Reached target sshd-keygen.target.
Aug 29 16:46:31 almalinux9 systemd[1]: System Security Services Daemon was skipped because all trigger condition checks failed.

All 3 services’ status show

systemctl status sshd-keygen@rsa.service --no-pager
○ sshd-keygen@rsa.service - OpenSSH rsa Server Key Generation
     Loaded: loaded (/usr/lib/systemd/system/sshd-keygen@.service; disabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/sshd-keygen@.service.d
             └─disable-sshd-keygen-if-cloud-init-active.conf
     Active: inactive (dead)
  Condition: start condition failed at Mon 2022-08-29 16:53:39 UTC; 23min ago
             └─ ConditionPathExists=!/run/systemd/generator.early/multi-user.target.wants/cloud-init.target was not met

Aug 29 16:53:39 almalinux9.local systemd[1]: OpenSSH rsa Server Key Generation was skipped because of a failed condition check (ConditionPathExists=!/run/systemd/generator.early/multi-user…ud-init.target).
Hint: Some lines were ellipsized, use -l to show in full.
systemctl status sshd-keygen@ecdsa.service --no-pager
○ sshd-keygen@ecdsa.service - OpenSSH ecdsa Server Key Generation
     Loaded: loaded (/usr/lib/systemd/system/sshd-keygen@.service; disabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/sshd-keygen@.service.d
             └─disable-sshd-keygen-if-cloud-init-active.conf
     Active: inactive (dead)
  Condition: start condition failed at Mon 2022-08-29 16:53:39 UTC; 25min ago
             └─ ConditionPathExists=!/run/systemd/generator.early/multi-user.target.wants/cloud-init.target was not met

Aug 29 16:53:39 almalinux9.local systemd[1]: OpenSSH ecdsa Server Key Generation was skipped because of a failed condition check (ConditionPathExists=!/run/systemd/generator.early/multi-us…ud-init.target).
Hint: Some lines were ellipsized, use -l to show in full.
systemctl status sshd-keygen@ed25519.service --no-pager
○ sshd-keygen@ed25519.service - OpenSSH ed25519 Server Key Generation
     Loaded: loaded (/usr/lib/systemd/system/sshd-keygen@.service; disabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/sshd-keygen@.service.d
             └─disable-sshd-keygen-if-cloud-init-active.conf
     Active: inactive (dead)
  Condition: start condition failed at Mon 2022-08-29 16:53:39 UTC; 25min ago
             └─ ConditionPathExists=!/run/systemd/generator.early/multi-user.target.wants/cloud-init.target was not met

Aug 29 16:53:39 almalinux9.local systemd[1]: OpenSSH ed25519 Server Key Generation was skipped because of a failed condition check (ConditionPathExists=!/run/systemd/generator.early/multi-…ud-init.target).
Hint: Some lines were ellipsized, use -l to show in full.

and seems it’s because of /etc/systemd/system/sshd-keygen@.service.d/disable-sshd-keygen-if-cloud-init-active.conf drop in file with

# In some cloud-init enabled images the sshd-keygen template service may race
# with cloud-init during boot causing issues with host key generation.  This
# drop-in config adds a condition to sshd-keygen@.service if it exists and
# prevents the sshd-keygen units from running *if* cloud-init is going to run.
#
[Unit]
ConditionPathExists=!/run/systemd/generator.early/multi-user.target.wants/cloud-init.target

which exists at /run/systemd/generator.early/multi-user.target.wants/cloud-init.target

cat /run/systemd/generator.early/multi-user.target.wants/cloud-init.target
# cloud-init target is enabled by cloud-init-generator
# To disable it you can either:
#  a.) boot with kernel cmdline of 'cloud-init=disabled'
#  b.) touch a file /etc/cloud/cloud-init.disabled
[Unit]
Description=Cloud-init target
After=multi-user.target

How to properly handle this?

It’s Not a Bug, It’s a Feature - No! Really, It’s! :grinning:

Let me explain how.

This Cloud-init PR, creates systemd drop-in unit to avoid race condition between sshd-keygen units and Cloud-init. - Cloud-init handles the generation of ssh host keys.

So this output:

OpenSSH {ecdsa,ed25519,rsa} Server Key Generation was skipped because of a failed condition check

comes from the condition check of the disable-sshd-keygen-if-cloud-init-active.conf
drop in unit file.

Hi!
On an AlmaLinux9 node I can recreate the behavior described above using cloud-init-22.1-5.el9.alma.1.noarch.
Can you please take a look?