Openssh-server-8.0p1-16.el8 update and HostKey

Hello Team,

I have updated Almalinux 8.6 to 8.7, which also updated Openssh-server version from “8.0p1-13.el8.x86_64” to “8.0p1-16.el8.x86_64”.

In fact, this is causing issue with “HostKey”. We don’t use HostKey duing deployment and which was failing our may applications which was needed ssh, applications like PCS, HAproxy etc.

could you please help to fix this issue.

Regards,
Dilip

Is it possible to disable hostkeys in config?

man sshd_config
man update-crypto-policies
man sshd

point to CRYPTO_POLICY in /etc/sysconfig/sshd and HostKeyAlgorithms.
Can the latter be an empty list?

Does this means, I was asked to give entry in file “/etc/sysconfig/sshd” like below
CRYPTO_POLICY=HostKeyAlgorithms
?

at present it;s commented
image

The description of option -T in man sshd shows that one should source two files when using the -T. I bet that the sshd.service does source those same files (but did not check).

The first file is from system-wide crypto policy and does define CRYPTO_POLICY as list
of command-line parameters for the sshd. There are a lot in that.

The /etc/sysconfig/sshd is the second file. If one does as comments in it say and sets

CRYPTO_POLICY=

then none of the options in system-wide crypto policy are used and sshd has only
what is in /etc/ssh/sshd_config (and builtin defaults).

[ 71.525345] cloud-init[2051]: ------------ Status of SSHD --------
[ 71.532703] cloud-init[2051]: ��� sshd.service - OpenSSH server daemon
[ 71.533639] cloud-init[2051]: Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
[ 71.534959] cloud-init[2051]: Active: activating (auto-restart) (Result: exit-code) since Thu 2023-03-16 14:51:12 UTC; 59ms ago
[ 71.536276] cloud-init[2051]: Docs: man:sshd(8)
[ 71.536974] cloud-init[2051]: man:sshd_config(5)
[ 71.537747] cloud-init[2051]: Process: 2339 ExecStart=/usr/sbin/sshd -D $OPTIONS $CRYPTO_POLICY (code=exited, status=1/FAILURE)
[ 71.538950] cloud-init[2051]: Main PID: 2339 (code=exited, status=1/FAILURE)
[ 71.539858] cloud-init[2051]: ------------ status sshd-keygen.target ----
[ 71.540830] cloud-init[2051]: ��� sshd-keygen.target
[ 71.541626] cloud-init[2051]: Loaded: loaded (/usr/lib/systemd/system/sshd-keygen.target; static; vendor preset: disabled)
[ 71.542808] cloud-init[2051]: Active: active since Thu 2023-03-16 14:51:12 UTC; 16ms ago
[ 71.543780] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: Reached target sshd-keygen.target.
[ 71.544843] cloud-init[2051]: ------------ Journalctl SSH o/p ------
[ 71.545690] cloud-init[2051]: – Logs begin at Thu 2023-03-16 14:50:02 UTC, end at Thu 2023-03-16 14:51:12 UTC. –
[ 71.546788] cloud-init[2051]: Mar 16 14:51:11 cluman.novalocal systemd[1]: Starting OpenSSH server daemon…
[ 71.547858] cloud-init[2051]: Mar 16 14:51:11 cluman.novalocal sshd[1153]: sshd: no hostkeys available – exiting.
[ 71.548962] cloud-init[2051]: Mar 16 14:51:11 cluman.novalocal systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
[ 71.550258] cloud-init[2051]: Mar 16 14:51:11 cluman.novalocal systemd[1]: sshd.service: Failed with result ‘exit-code’.
[ 71.551421] cloud-init[2051]: Mar 16 14:51:11 cluman.novalocal systemd[1]: Failed to start OpenSSH server daemon.
[ 71.552554] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: Stopped OpenSSH server daemon.
[ 71.553632] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: Starting OpenSSH server daemon…
[ 71.554683] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal sshd[2339]: sshd: no hostkeys available – exiting.
[ 71.555838] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
[ 71.557140] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: sshd.service: Failed with result ‘exit-code’.
[ 71.558282] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: Failed to start OpenSSH server daemon.
[ 71.559375] cloud-init[2051]: ------------ HostKey entry in sshd_config ----
[ 71.560308] cloud-init[2051]: # HostKey for protocol version 1
[ 71.561135] cloud-init[2051]: #HostKey /etc/ssh/ssh_host_key
[ 71.561874] cloud-init[2051]: # HostKeys for protocol version 2
[ 71.562684] cloud-init[2051]: #HostKey /etc/ssh/ssh_host_rsa_key
[ 71.563487] cloud-init[2051]: #HostKey /etc/ssh/ssh_host_dsa_key
[ 71.564291] cloud-init[2051]: ------------ Journalctl SSHD-Keygen o/p ----
[ 71.565488] cloud-init[2051]: – Logs begin at Thu 2023-03-16 14:50:02 UTC, end at Thu 2023-03-16 14:51:12 UTC. –
[ 71.566551] cloud-init[2051]: Mar 16 14:50:06 localhost.localdomain systemd[1]: Reached target sshd-keygen.target.
[ 71.567632] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: Stopped target sshd-keygen.target.
[ 71.568609] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: Stopping sshd-keygen.target.
[ 71.569878] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: Reached target sshd-keygen.target.
[ 71.571524] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: Stopped target sshd-keygen.target.
[ 71.573188] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: Stopping sshd-keygen.target.
[ 71.574806] cloud-init[2051]: Mar 16 14:51:12 cluman.novalocal systemd[1]: Reached target sshd-keygen.target.
[ 71.576516] cloud-init[2051]: ----------------

not sure in Openstack sshd-keygen.target is failing and default HostKey entry is not adding in “sshd_config” file

can I expect any update here please ?

[ 71.612356] cloud-init[1824]: ------------ Listing HostKey default file —
[ 71.613047] cloud-init[1824]: ls: cannot access ‘/etc/ssh/ssh_host_rsa_key’: No such file or directory

.
how can we make this default private key file for sshd-keygen.target to get started Or we have to disable the need of “HostKey” during installation

I have tried given entry in file “/etc/sysconfig/sshd” like below and restarted the SSHD and SSHD-genkey, but only sshd-genkey is working but default HostKey file “/etc/ssh/ssh_host_rsa_key” not created and SSHD service not started with error

CRYPTO_POLICY=HostKeyAlgorithms

SSHD - Error : -
Starting OpenSSH server daemon…
Unable to load host key: /etc/ssh/ssh_host_rsa_key
sshd: no hostkeys available – exiting.
sshd.service: Main process exited, code=exited, status=1/FAILURE
sshd.service: Failed with result ‘exit-code’.
Failed to start OpenSSH server daemon.

The default value (in el8 crypto policy) is:

CRYPTO_POLICY='-oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512 -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1- -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa'

If one does want that default, then “/etc/sysconfig/sshd” should have:

CRYPTO_POLICY=

The real question is, why the VM image does not run those services proper? Has something masked the three services?

Answer is ‘yes’, the cloud-init does disable:
Package cloud-init has file named /etc/systemd/system/sshd-keygen@.service.d/disable-sshd-keygen-if-cloud-init-active.conf – a systemd drop-in config to solve:

In some cloud-init enabled images the sshd-keygen template service may race with cloud-init during boot causing issues with host key generation.

The Cloud config examples - cloud-init 23.3.3 documentation describes ssh_keys variable. That is how cloud-init could put host keys into image.

Indeed, “ssh-keygen -A” is working as added this entry in kickstart file that created the /etc/ssh/ssh_host_rsa_key file…
Many thanks for this solution

howerver, not sure why this is was not created during the openssh installed in Open stack vm but, things are working fine in VMware VMs.

If possible, I would request to take it as bug to fix it further down releases of it.

I have tried it but no luck.