Kernel panic watchdog CPU stuck

Hi,
just updated to Alma 8.5 version 4.18.0-348.12.2 the other day and seeing kernel panics on my Hardware name: Supermicro M12SWA-TF/M12SWA-TF, BIOS 1.0c 09/10/2021 mobo.

Here’s a snippet:

Jan 21 15:50:43 verbiet kernel: watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [brave:123184]
Jan 21 15:50:43 verbiet kernel: Modules linked in: snd_seq_dummy snd_hrtimer xt_nat veth xt_CHECKSUM ipt_REJECT nf_reject
ipv4 xt_conntrack binfmt_misc ipt_MASQUERADE nf_conntrack_netlink nft_counter xt_addrtype nft_compat nft_chain_nat nf_na
t nf_conntrack uinput nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink br_netfilter tun bridge stp llc overlay sunrpc sq
uashfs vfat fat ext4 mbcache jbd2 loop intel_rapl_msr intel_rapl_common amd64_edac_mod edac_mce_amd amd_energy kvm_amd kv
m irqbypass crct10dif_pclmul crc32_pclmul ipmi_si ghash_clmulni_intel rapl ipmi_devintf ipmi_msghandler pcspkr snd_hda_co
dec_hdmi snd_hda_intel snd_intel_dspcfg snd_usb_audio snd_intel_sdw_acpi snd_hda_codec snd_hda_core snd_usbmidi_lib snd_r
awmidi snd_hwdep snd_seq snd_seq_device uvcvideo snd_pcm videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_comm
on snd_timer snd videodev joydev ccp soundcore sp5100_tco k10temp i2c_piix4 i2c_designware_platform i2c_designware_core a
cpi_cpufreq xfs libcrc32c nvidia_drm(POE)
Jan 21 15:50:43 verbiet kernel: nvidia_modeset(POE) nvidia(POE) sd_mod sg crc32c_intel ast drm_vram_helper drm_ttm_helper
ttm drm_kms_helper igb syscopyarea sysfillrect ahci sysimgblt fb_sys_fops libahci nvme dca drm libata atlantic i2c_algo

bit nvme_core t10_pi rndis_host cdc_ether usbnet mii dm_mirror dm_region_hash dm_log dm_mod fuse
Jan 21 15:50:43 verbiet kernel: CPU: 21 PID: 123184 Comm: brave Kdump: loaded Tainted: P OE --------- - - 4
.18.0-348.12.2.el8_5.x86_64 #1
Jan 21 15:50:43 verbiet kernel: Hardware name: Supermicro M12SWA-TF/M12SWA-TF, BIOS 1.0c 09/10/2021
Jan 21 15:50:43 verbiet kernel: RIP: 0010:_nv035888rm+0xa8/0xe0 [nvidia]
Jan 21 15:50:43 verbiet kernel: Code: ff ff 49 8b 4c 24 20 48 89 c2 48 89 ef 48 8d b1 48 01 00 00 4c 89 e9 e8 86 5b ff ff 66 0f 1f 44 00 00 48 89 ef e8 e8 5b ff ff <84> c0 74 8a 48 8b 75 00 48 39 5e 08 75 ea 4c 39 26 75 e5 49 8b 44
Jan 21 15:50:43 verbiet kernel: RSP: 0018:ffffb64dd0c43b98 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
Jan 21 15:50:43 verbiet kernel: RAX: 0000000000000001 RBX: ffff9680c4a2b830 RCX: ffff9680d0651d78
Jan 21 15:50:43 verbiet kernel: RDX: ffffede1f201cbc8 RSI: ffffede1f201dde7 RDI: ffff967f4b0b2d20
Jan 21 15:50:43 verbiet kernel: RBP: ffff967f4b0b2d20 R08: 0000000000000020 R09: ffff967f4b0b2d28

will watch carefully before rebooting under prior kernel. Anyone else with this supermicro Mobo to check?

The entire log is here:

https://www.iscs-i.com/download/supermicro/messages

Thanks,
Henry

Running kernel 4.18.0-372.19.1.el8_6.x86_64 and saw this today:

Message from syslogd@HOSTNAME at Aug  8 04:46:48 ...
 kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [php:52973]