Frequent kernel crashes since Jan 1

Support
1

I have an AL 8 fileserver VM (running on VMware ESXi) that primarily serves as an NFS share, with a bit of SMB and http sharing also going on. It has been rock solid for many months.

Beginning Jan 1 2024 it started crashing multiple times per hour, eventually filling up the root volume with core files.

Tracebacks vary but in all cases the failure appears to be during slab_free, e.g.

[ 2134.883197] kernel BUG at mm/slub.c:376!
[ 2134.883237] invalid opcode: 0000 [#1] SMP PTI
[ 2134.883260] CPU: 0 PID: 1319 Comm: rpc.mountd Kdump: loaded Tainted: G W ---------r- - 4.18.0-513.9.1.el8_9.x86_64
#1
[ 2134.883311] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ 2134.883374] RIP: 0010:__slab_free+0x19e/0x350

I assume something is corrupting kernel memory and this happens when the corruption is detected. But the problem did not start until the new year began. I’m currently running latest patches of 8.9, but I was on 8.8 until I upgraded this morning to see if that would fix the crashes…it did not.

Anyone else seeing this? Any ideas what might have broken by the turn of the year?

2

Booted with slub_debug=F and it looks like the checking caught a double-free. Anyone recognize this?

[ 3070.298233] =============================================================================
[ 3070.298322] BUG kmalloc-4k (Tainted: G ---------r- -): Object already free
[ 3070.298371] -----------------------------------------------------------------------------

[ 3070.298424] Slab 0x00000000a37036d6 objects=8 used=2 fp=0x0000000049976705 flags=0x17ffffc0008101(locked|slab|head|node=0|zone=2|lastcpupid=0x1fffff)
[ 3070.298500] Object 0x0000000049976705 @offset=0 fp=0x0000000000000000

[ 3070.298547] Object 0000000049976705: 00 80 39 a5 b9 8e ff ff 00 80 39 a5 b9 8e ff ff …9…9…
[ 3070.298628] Object 0000000086c11a5f: 10 80 39 a5 b9 8e ff ff 10 80 39 a5 b9 8e ff ff …9…9…
[ 3070.298678] Object 00000000fa5466b8: 20 80 39 a5 b9 8e ff ff 20 80 39 a5 b9 8e ff ff .9… .9…
[ 3070.298730] Object 000000007c208758: 30 80 39 a5 b9 8e ff ff 30 80 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.298779] Object 000000002093ffef: 40 80 39 a5 b9 8e ff ff 40 80 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.298820] Object 00000000a9ca3706: 50 80 39 a5 b9 8e ff ff 50 80 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.298860] Object 0000000018e12e01: 60 80 39 a5 b9 8e ff ff 60 80 39 a5 b9 8e ff ff .9......9…
[ 3070.298899] Object 000000003b06a7fb: 70 80 39 a5 b9 8e ff ff 70 80 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.298938] Object 000000006d02208d: 80 80 39 a5 b9 8e ff ff 80 80 39 a5 b9 8e ff ff …9…9…
[ 3070.298977] Object 00000000ab026cb7: 90 80 39 a5 b9 8e ff ff 90 80 39 a5 b9 8e ff ff …9…9…
[ 3070.299016] Object 000000007fdb4f82: a0 80 39 a5 b9 8e ff ff a0 80 39 a5 b9 8e ff ff …9…9…
[ 3070.299066] Object 00000000617b02b7: b0 80 39 a5 b9 8e ff ff b0 80 39 a5 b9 8e ff ff …9…9…
[ 3070.299108] Object 0000000092fbc88c: c0 80 39 a5 b9 8e ff ff c0 80 39 a5 b9 8e ff ff …9…9…
[ 3070.299149] Object 00000000692462c1: d0 80 39 a5 b9 8e ff ff d0 80 39 a5 b9 8e ff ff …9…9…
[ 3070.299189] Object 00000000c67c59d0: e0 80 39 a5 b9 8e ff ff e0 80 39 a5 b9 8e ff ff …9…9…
[ 3070.299231] Object 00000000fe700af6: f0 80 39 a5 b9 8e ff ff f0 80 39 a5 b9 8e ff ff …9…9…
[ 3070.299271] Object 000000006514deb5: 00 81 39 a5 b9 8e ff ff 00 81 39 a5 b9 8e ff ff …9…9…
[ 3070.299311] Object 00000000ef65ab23: 10 81 39 a5 b9 8e ff ff 10 81 39 a5 b9 8e ff ff …9…9…
[ 3070.299352] Object 000000006fa763fa: 20 81 39 a5 b9 8e ff ff 20 81 39 a5 b9 8e ff ff .9… .9…
[ 3070.299392] Object 00000000e69e0c80: 30 81 39 a5 b9 8e ff ff 30 81 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.299431] Object 000000006a681b4a: 40 81 39 a5 b9 8e ff ff 40 81 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.299470] Object 0000000009b68ae2: 50 81 39 a5 b9 8e ff ff 50 81 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.299510] Object 000000009f3b640b: 60 81 39 a5 b9 8e ff ff 60 81 39 a5 b9 8e ff ff .9......9…
[ 3070.299558] Object 00000000fbf61c83: 70 81 39 a5 b9 8e ff ff 70 81 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.299597] Object 00000000f24f8767: 80 81 39 a5 b9 8e ff ff 80 81 39 a5 b9 8e ff ff …9…9…
[ 3070.299637] Object 00000000993f7e5f: 90 81 39 a5 b9 8e ff ff 90 81 39 a5 b9 8e ff ff …9…9…
[ 3070.299685] Object 00000000c1d3b62b: a0 81 39 a5 b9 8e ff ff a0 81 39 a5 b9 8e ff ff …9…9…
[ 3070.299725] Object 000000008156c5b1: b0 81 39 a5 b9 8e ff ff b0 81 39 a5 b9 8e ff ff …9…9…
[ 3070.299764] Object 00000000980fd822: c0 81 39 a5 b9 8e ff ff c0 81 39 a5 b9 8e ff ff …9…9…
[ 3070.306530] Object 000000000ed021a1: d0 81 39 a5 b9 8e ff ff d0 81 39 a5 b9 8e ff ff …9…9…
[ 3070.307011] Object 00000000854f75fe: e0 81 39 a5 b9 8e ff ff e0 81 39 a5 b9 8e ff ff …9…9…
[ 3070.307287] Object 0000000090576e87: f0 81 39 a5 b9 8e ff ff f0 81 39 a5 b9 8e ff ff …9…9…
[ 3070.307627] Object 000000008db52523: 00 82 39 a5 b9 8e ff ff 00 82 39 a5 b9 8e ff ff …9…9…
[ 3070.307838] Object 000000001e2a87d3: 10 82 39 a5 b9 8e ff ff 10 82 39 a5 b9 8e ff ff …9…9…
[ 3070.308040] Object 00000000ec8ebe5e: 20 82 39 a5 b9 8e ff ff 20 82 39 a5 b9 8e ff ff .9… .9…
[ 3070.308245] Object 00000000c9d19e62: 30 82 39 a5 b9 8e ff ff 30 82 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.308450] Object 000000006b5c8073: 40 82 39 a5 b9 8e ff ff 40 82 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.308660] Object 00000000ff7414b9: 50 82 39 a5 b9 8e ff ff 50 82 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.308862] Object 0000000086848448: 60 82 39 a5 b9 8e ff ff 60 82 39 a5 b9 8e ff ff .9......9…
[ 3070.309117] Object 000000006e2dff3c: 70 82 39 a5 b9 8e ff ff 70 82 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.309388] Object 00000000d3634089: 80 82 39 a5 b9 8e ff ff 80 82 39 a5 b9 8e ff ff …9…9…
[ 3070.309604] Object 000000006b2f7b13: 90 82 39 a5 b9 8e ff ff 90 82 39 a5 b9 8e ff ff …9…9…
[ 3070.309815] Object 00000000ce944cae: a0 82 39 a5 b9 8e ff ff a0 82 39 a5 b9 8e ff ff …9…9…
[ 3070.310014] Object 000000005c51897b: b0 82 39 a5 b9 8e ff ff b0 82 39 a5 b9 8e ff ff …9…9…
[ 3070.310209] Object 0000000078a9b18c: c0 82 39 a5 b9 8e ff ff c0 82 39 a5 b9 8e ff ff …9…9…
[ 3070.310405] Object 000000008e9731c9: d0 82 39 a5 b9 8e ff ff d0 82 39 a5 b9 8e ff ff …9…9…
[ 3070.310606] Object 00000000df34a076: e0 82 39 a5 b9 8e ff ff e0 82 39 a5 b9 8e ff ff …9…9…
[ 3070.310809] Object 00000000f29d8df6: f0 82 39 a5 b9 8e ff ff f0 82 39 a5 b9 8e ff ff …9…9…
[ 3070.310986] Object 0000000075f29c4c: 00 83 39 a5 b9 8e ff ff 00 83 39 a5 b9 8e ff ff …9…9…
[ 3070.311162] Object 0000000054629949: 10 83 39 a5 b9 8e ff ff 10 83 39 a5 b9 8e ff ff …9…9…
[ 3070.311332] Object 00000000b153178f: 20 83 39 a5 b9 8e ff ff 20 83 39 a5 b9 8e ff ff .9… .9…
[ 3070.311511] Object 00000000325ec27f: 30 83 39 a5 b9 8e ff ff 30 83 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.311683] Object 0000000064f5848d: 40 83 39 a5 b9 8e ff ff 40 83 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.311913] Object 000000003b910835: 50 83 39 a5 b9 8e ff ff 50 83 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.312108] Object 000000001e04da79: 60 83 39 a5 b9 8e ff ff 60 83 39 a5 b9 8e ff ff .9......9…
[ 3070.312279] Object 00000000d0efb125: 70 83 39 a5 b9 8e ff ff 70 83 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.312463] Object 00000000fc46de3c: 80 83 39 a5 b9 8e ff ff 80 83 39 a5 b9 8e ff ff …9…9…
[ 3070.312634] Object 000000004f466053: 90 83 39 a5 b9 8e ff ff 90 83 39 a5 b9 8e ff ff …9…9…
[ 3070.312815] Object 0000000057fdada1: a0 83 39 a5 b9 8e ff ff a0 83 39 a5 b9 8e ff ff …9…9…
[ 3070.312998] Object 00000000b2dffdfb: b0 83 39 a5 b9 8e ff ff b0 83 39 a5 b9 8e ff ff …9…9…
[ 3070.313167] Object 0000000011eb1ba2: c0 83 39 a5 b9 8e ff ff c0 83 39 a5 b9 8e ff ff …9…9…
[ 3070.313337] Object 000000006c25bb81: d0 83 39 a5 b9 8e ff ff d0 83 39 a5 b9 8e ff ff …9…9…
[ 3070.313542] Object 000000008b2cb196: e0 83 39 a5 b9 8e ff ff e0 83 39 a5 b9 8e ff ff …9…9…
[ 3070.313715] Object 00000000b737d9ab: f0 83 39 a5 b9 8e ff ff f0 83 39 a5 b9 8e ff ff …9…9…
[ 3070.313885] Object 0000000011483c62: 00 84 39 a5 b9 8e ff ff 00 84 39 a5 b9 8e ff ff …9…9…
[ 3070.314072] Object 000000003978fad8: 10 84 39 a5 b9 8e ff ff 10 84 39 a5 b9 8e ff ff …9…9…
[ 3070.314243] Object 00000000c4e6cdd9: 20 84 39 a5 b9 8e ff ff 20 84 39 a5 b9 8e ff ff .9… .9…
[ 3070.314416] Object 000000002edb22f1: 30 84 39 a5 b9 8e ff ff 30 84 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.314604] Object 00000000b8c9623d: 40 84 39 a5 b9 8e ff ff 40 84 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.314791] Object 000000006e8a81bb: 50 84 39 a5 b9 8e ff ff 50 84 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.315031] Object 000000006ed1ae7c: 60 84 39 a5 b9 8e ff ff 60 84 39 a5 b9 8e ff ff .9......9…
[ 3070.315244] Object 0000000020c7e3e5: 70 84 39 a5 b9 8e ff ff 70 84 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.315423] Object 000000006f05bf65: 80 84 39 a5 b9 8e ff ff 80 84 39 a5 b9 8e ff ff …9…9…
[ 3070.315608] Object 000000005a895bc7: 90 84 39 a5 b9 8e ff ff 90 84 39 a5 b9 8e ff ff …9…9…
[ 3070.315783] Object 00000000c032970b: a0 84 39 a5 b9 8e ff ff a0 84 39 a5 b9 8e ff ff …9…9…
[ 3070.315982] Object 0000000097ddab9b: b0 84 39 a5 b9 8e ff ff b0 84 39 a5 b9 8e ff ff …9…9…
[ 3070.316167] Object 00000000d5691e41: c0 84 39 a5 b9 8e ff ff c0 84 39 a5 b9 8e ff ff …9…9…
[ 3070.316338] Object 00000000545c4b57: d0 84 39 a5 b9 8e ff ff d0 84 39 a5 b9 8e ff ff …9…9…
[ 3070.316519] Object 00000000d6f2eeb1: e0 84 39 a5 b9 8e ff ff e0 84 39 a5 b9 8e ff ff …9…9…
[ 3070.316706] Object 00000000f73bf62f: f0 84 39 a5 b9 8e ff ff f0 84 39 a5 b9 8e ff ff …9…9…
[ 3070.316909] Object 000000008baf1119: 00 85 39 a5 b9 8e ff ff 00 85 39 a5 b9 8e ff ff …9…9…
[ 3070.317083] Object 00000000a80babad: 10 85 39 a5 b9 8e ff ff 10 85 39 a5 b9 8e ff ff …9…9…
[ 3070.317253] Object 000000003bebd600: 20 85 39 a5 b9 8e ff ff 20 85 39 a5 b9 8e ff ff .9… .9…
[ 3070.317439] Object 00000000a0e21a78: 30 85 39 a5 b9 8e ff ff 30 85 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.317623] Object 00000000ed3d514e: 40 85 39 a5 b9 8e ff ff 40 85 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.317809] Object 00000000f6286f7d: 50 85 39 a5 b9 8e ff ff 50 85 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.317998] Object 000000001323b9cc: 60 85 39 a5 b9 8e ff ff 60 85 39 a5 b9 8e ff ff .9......9…
[ 3070.318170] Object 000000001f12c223: 70 85 39 a5 b9 8e ff ff 70 85 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.318340] Object 00000000e3a7ad94: 80 85 39 a5 b9 8e ff ff 80 85 39 a5 b9 8e ff ff …9…9…
[ 3070.318512] Object 00000000f050be74: 90 85 39 a5 b9 8e ff ff 90 85 39 a5 b9 8e ff ff …9…9…
[ 3070.318726] Object 0000000031c7599c: a0 85 39 a5 b9 8e ff ff a0 85 39 a5 b9 8e ff ff …9…9…
[ 3070.318913] Object 000000008a25fce7: b0 85 39 a5 b9 8e ff ff b0 85 39 a5 b9 8e ff ff …9…9…
[ 3070.319084] Object 0000000014abd1c3: c0 85 39 a5 b9 8e ff ff c0 85 39 a5 b9 8e ff ff …9…9…
[ 3070.319277] Object 00000000f9625859: d0 85 39 a5 b9 8e ff ff d0 85 39 a5 b9 8e ff ff …9…9…
[ 3070.319449] Object 000000009b47f58a: e0 85 39 a5 b9 8e ff ff e0 85 39 a5 b9 8e ff ff …9…9…
[ 3070.319627] Object 000000000c26e89e: f0 85 39 a5 b9 8e ff ff f0 85 39 a5 b9 8e ff ff …9…9…
[ 3070.319811] Object 00000000a45b6f10: 00 86 39 a5 b9 8e ff ff 00 86 39 a5 b9 8e ff ff …9…9…
[ 3070.319982] Object 0000000063f5ce51: 10 86 39 a5 b9 8e ff ff 10 86 39 a5 b9 8e ff ff …9…9…
[ 3070.320150] Object 0000000053477d56: 20 86 39 a5 b9 8e ff ff 20 86 39 a5 b9 8e ff ff .9… .9…
[ 3070.320317] Object 000000006d6d8126: 30 86 39 a5 b9 8e ff ff 30 86 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.320487] Object 000000003eb5e445: 40 86 39 a5 b9 8e ff ff 40 86 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.320665] Object 00000000ea9809d5: 50 86 39 a5 b9 8e ff ff 50 86 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.320849] Object 0000000087059695: 60 86 39 a5 b9 8e ff ff 60 86 39 a5 b9 8e ff ff .9......9…
[ 3070.321020] Object 00000000e39c58c8: 70 86 39 a5 b9 8e ff ff 70 86 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.321189] Object 0000000063476a5e: 80 86 39 a5 b9 8e ff ff 80 86 39 a5 b9 8e ff ff …9…9…
[ 3070.321369] Object 00000000c538cede: 90 86 39 a5 b9 8e ff ff 90 86 39 a5 b9 8e ff ff …9…9…
[ 3070.321538] Object 000000007af2bf81: a0 86 39 a5 b9 8e ff ff a0 86 39 a5 b9 8e ff ff …9…9…
[ 3070.321714] Object 0000000065812197: b0 86 39 a5 b9 8e ff ff b0 86 39 a5 b9 8e ff ff …9…9…
[ 3070.321897] Object 000000002115fe7d: c0 86 39 a5 b9 8e ff ff c0 86 39 a5 b9 8e ff ff …9…9…
[ 3070.322078] Object 00000000848ceaab: d0 86 39 a5 b9 8e ff ff d0 86 39 a5 b9 8e ff ff …9…9…
[ 3070.322246] Object 0000000087ac3b9e: e0 86 39 a5 b9 8e ff ff e0 86 39 a5 b9 8e ff ff …9…9…
[ 3070.322418] Object 0000000078fe5166: f0 86 39 a5 b9 8e ff ff f0 86 39 a5 b9 8e ff ff …9…9…
[ 3070.322587] Object 000000004f31fd27: 00 87 39 a5 b9 8e ff ff 00 87 39 a5 b9 8e ff ff …9…9…
[ 3070.322780] Object 000000007553ee97: 10 87 39 a5 b9 8e ff ff 10 87 39 a5 b9 8e ff ff …9…9…
[ 3070.322950] Object 000000007d25f964: 20 87 39 a5 b9 8e ff ff 20 87 39 a5 b9 8e ff ff .9… .9…
[ 3070.323117] Object 00000000a4781b79: 30 87 39 a5 b9 8e ff ff 30 87 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.323283] Object 0000000013dd2b16: 40 87 39 a5 b9 8e ff ff 40 87 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.323454] Object 00000000dfb0f80f: 50 87 39 a5 b9 8e ff ff 50 87 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.323621] Object 00000000e05a6336: 60 87 39 a5 b9 8e ff ff 60 87 39 a5 b9 8e ff ff .9......9…
[ 3070.323815] Object 0000000064f10ec8: 70 87 39 a5 b9 8e ff ff 70 87 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.323990] Object 000000009ab03f8a: 80 87 39 a5 b9 8e ff ff 80 87 39 a5 b9 8e ff ff …9…9…
[ 3070.324158] Object 000000001049927d: 90 87 39 a5 b9 8e ff ff 90 87 39 a5 b9 8e ff ff …9…9…
[ 3070.324324] Object 00000000290520be: a0 87 39 a5 b9 8e ff ff a0 87 39 a5 b9 8e ff ff …9…9…
[ 3070.324493] Object 00000000e3ebcbcc: b0 87 39 a5 b9 8e ff ff b0 87 39 a5 b9 8e ff ff …9…9…
[ 3070.324659] Object 00000000a604e70f: c0 87 39 a5 b9 8e ff ff c0 87 39 a5 b9 8e ff ff …9…9…
[ 3070.324856] Object 00000000e6a8fe4d: d0 87 39 a5 b9 8e ff ff d0 87 39 a5 b9 8e ff ff …9…9…
[ 3070.325024] Object 000000006c2d0ed3: e0 87 39 a5 b9 8e ff ff e0 87 39 a5 b9 8e ff ff …9…9…
[ 3070.325196] Object 00000000ab119d61: f0 87 39 a5 b9 8e ff ff f0 87 39 a5 b9 8e ff ff …9…9…
[ 3070.325381] Object 00000000d6ddad88: b3 02 6b dd fa 68 4a 15 00 88 39 a5 b9 8e ff ff …k…hJ…9…
[ 3070.325554] Object 00000000b98b4bd3: 10 88 39 a5 b9 8e ff ff 10 88 39 a5 b9 8e ff ff …9…9…
[ 3070.325740] Object 00000000f762dc0a: 20 88 39 a5 b9 8e ff ff 20 88 39 a5 b9 8e ff ff .9… .9…
[ 3070.325914] Object 000000007d31ad69: 30 88 39 a5 b9 8e ff ff 30 88 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.326086] Object 00000000c2f8ae8f: 40 88 39 a5 b9 8e ff ff 40 88 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.326258] Object 00000000a3404e99: 50 88 39 a5 b9 8e ff ff 50 88 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.326436] Object 00000000dd7a6a72: 60 88 39 a5 b9 8e ff ff 60 88 39 a5 b9 8e ff ff .9......9…
[ 3070.326608] Object 00000000a7164946: 70 88 39 a5 b9 8e ff ff 70 88 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.326786] Object 000000000e6895ad: 80 88 39 a5 b9 8e ff ff 80 88 39 a5 b9 8e ff ff …9…9…
[ 3070.326959] Object 00000000526cc2b5: 90 88 39 a5 b9 8e ff ff 90 88 39 a5 b9 8e ff ff …9…9…
[ 3070.327130] Object 00000000067a61d7: a0 88 39 a5 b9 8e ff ff a0 88 39 a5 b9 8e ff ff …9…9…
[ 3070.327302] Object 000000002b32fa3a: b0 88 39 a5 b9 8e ff ff b0 88 39 a5 b9 8e ff ff …9…9…
[ 3070.327476] Object 0000000009b0f02f: c0 88 39 a5 b9 8e ff ff c0 88 39 a5 b9 8e ff ff …9…9…
[ 3070.327648] Object 000000002811ff61: d0 88 39 a5 b9 8e ff ff d0 88 39 a5 b9 8e ff ff …9…9…
[ 3070.327826] Object 000000000f7538b9: e0 88 39 a5 b9 8e ff ff e0 88 39 a5 b9 8e ff ff …9…9…
[ 3070.328005] Object 00000000f856cac4: f0 88 39 a5 b9 8e ff ff f0 88 39 a5 b9 8e ff ff …9…9…
[ 3070.328188] Object 000000000ee03b8a: 00 89 39 a5 b9 8e ff ff 00 89 39 a5 b9 8e ff ff …9…9…
[ 3070.328361] Object 00000000e92c00f6: 10 89 39 a5 b9 8e ff ff 10 89 39 a5 b9 8e ff ff …9…9…
[ 3070.328535] Object 00000000b40f02f4: 20 89 39 a5 b9 8e ff ff 20 89 39 a5 b9 8e ff ff .9… .9…
[ 3070.328709] Object 0000000001375a76: 30 89 39 a5 b9 8e ff ff 30 89 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.328902] Object 00000000e4e8b78f: 40 89 39 a5 b9 8e ff ff 40 89 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.329077] Object 0000000001d8e0c7: 50 89 39 a5 b9 8e ff ff 50 89 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.329251] Object 0000000076e28e63: 60 89 39 a5 b9 8e ff ff 60 89 39 a5 b9 8e ff ff .9......9…
[ 3070.329427] Object 0000000098fd9505: 70 89 39 a5 b9 8e ff ff 70 89 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.329600] Object 0000000021391b64: 80 89 39 a5 b9 8e ff ff 80 89 39 a5 b9 8e ff ff …9…9…
[ 3070.329782] Object 00000000fc17d7a0: 90 89 39 a5 b9 8e ff ff 90 89 39 a5 b9 8e ff ff …9…9…
[ 3070.329955] Object 00000000a93ccaa5: a0 89 39 a5 b9 8e ff ff a0 89 39 a5 b9 8e ff ff …9…9…
[ 3070.330128] Object 00000000eb27b0e0: b0 89 39 a5 b9 8e ff ff b0 89 39 a5 b9 8e ff ff …9…9…
[ 3070.330300] Object 000000003778d016: c0 89 39 a5 b9 8e ff ff c0 89 39 a5 b9 8e ff ff …9…9…
[ 3070.330475] Object 00000000093d77b4: d0 89 39 a5 b9 8e ff ff d0 89 39 a5 b9 8e ff ff …9…9…
[ 3070.330648] Object 00000000dc8ed0ea: e0 89 39 a5 b9 8e ff ff e0 89 39 a5 b9 8e ff ff …9…9…
[ 3070.330861] Object 00000000396d4994: f0 89 39 a5 b9 8e ff ff f0 89 39 a5 b9 8e ff ff …9…9…
[ 3070.331075] Object 0000000026664d06: 00 8a 39 a5 b9 8e ff ff 00 8a 39 a5 b9 8e ff ff …9…9…
[ 3070.331254] Object 0000000033d6a090: 10 8a 39 a5 b9 8e ff ff 10 8a 39 a5 b9 8e ff ff …9…9…
[ 3070.331431] Object 000000000979c635: 20 8a 39 a5 b9 8e ff ff 20 8a 39 a5 b9 8e ff ff .9… .9…
[ 3070.331606] Object 000000009bc7389f: 30 8a 39 a5 b9 8e ff ff 30 8a 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.331801] Object 0000000023fa4f99: 40 8a 39 a5 b9 8e ff ff 40 8a 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.331986] Object 00000000708c3546: 50 8a 39 a5 b9 8e ff ff 50 8a 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.332162] Object 0000000095ef6c4f: 60 8a 39 a5 b9 8e ff ff 60 8a 39 a5 b9 8e ff ff .9......9…
[ 3070.332336] Object 00000000814abcdf: 70 8a 39 a5 b9 8e ff ff 70 8a 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.332516] Object 00000000272677f6: 80 8a 39 a5 b9 8e ff ff 80 8a 39 a5 b9 8e ff ff …9…9…
[ 3070.332691] Object 00000000cea89c50: 90 8a 39 a5 b9 8e ff ff 90 8a 39 a5 b9 8e ff ff …9…9…
[ 3070.332931] Object 000000009efbe91d: a0 8a 39 a5 b9 8e ff ff a0 8a 39 a5 b9 8e ff ff …9…9…
[ 3070.333112] Object 0000000007c4cfa7: b0 8a 39 a5 b9 8e ff ff b0 8a 39 a5 b9 8e ff ff …9…9…
[ 3070.333287] Object 00000000db5d3f7e: c0 8a 39 a5 b9 8e ff ff c0 8a 39 a5 b9 8e ff ff …9…9…
[ 3070.333465] Object 00000000d8e1794f: d0 8a 39 a5 b9 8e ff ff d0 8a 39 a5 b9 8e ff ff …9…9…
[ 3070.333662] Object 000000001b416152: e0 8a 39 a5 b9 8e ff ff e0 8a 39 a5 b9 8e ff ff …9…9…
[ 3070.333912] Object 00000000e4ca00a9: f0 8a 39 a5 b9 8e ff ff f0 8a 39 a5 b9 8e ff ff …9…9…
[ 3070.334193] Object 00000000085d6315: 00 8b 39 a5 b9 8e ff ff 00 8b 39 a5 b9 8e ff ff …9…9…
[ 3070.334486] Object 00000000c4fbcc36: 10 8b 39 a5 b9 8e ff ff 10 8b 39 a5 b9 8e ff ff …9…9…
[ 3070.334693] Object 00000000e82e5504: 20 8b 39 a5 b9 8e ff ff 20 8b 39 a5 b9 8e ff ff .9… .9…
[ 3070.334957] Object 000000006d18e02a: 30 8b 39 a5 b9 8e ff ff 30 8b 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.335133] Object 000000008f9ad2a9: 40 8b 39 a5 b9 8e ff ff 40 8b 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.335305] Object 000000001ce85d03: 50 8b 39 a5 b9 8e ff ff 50 8b 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.335511] Object 000000004386d5c9: 60 8b 39 a5 b9 8e ff ff 60 8b 39 a5 b9 8e ff ff .9......9…
[ 3070.335684] Object 00000000d8e6e4c2: 70 8b 39 a5 b9 8e ff ff 70 8b 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.335943] Object 000000005b259df9: 80 8b 39 a5 b9 8e ff ff 80 8b 39 a5 b9 8e ff ff …9…9…
[ 3070.336213] Object 00000000535c5825: 90 8b 39 a5 b9 8e ff ff 90 8b 39 a5 b9 8e ff ff …9…9…
[ 3070.336388] Object 00000000f42676e8: a0 8b 39 a5 b9 8e ff ff a0 8b 39 a5 b9 8e ff ff …9…9…
[ 3070.336562] Object 00000000acdaa19d: b0 8b 39 a5 b9 8e ff ff b0 8b 39 a5 b9 8e ff ff …9…9…
[ 3070.336768] Object 000000004faa5734: c0 8b 39 a5 b9 8e ff ff c0 8b 39 a5 b9 8e ff ff …9…9…
[ 3070.336983] Object 00000000ac6be732: d0 8b 39 a5 b9 8e ff ff d0 8b 39 a5 b9 8e ff ff …9…9…
[ 3070.337156] Object 000000009a1d16f6: e0 8b 39 a5 b9 8e ff ff e0 8b 39 a5 b9 8e ff ff …9…9…
[ 3070.337352] Object 000000007ee6a942: f0 8b 39 a5 b9 8e ff ff f0 8b 39 a5 b9 8e ff ff …9…9…
[ 3070.337529] Object 0000000027d94d01: 00 8c 39 a5 b9 8e ff ff 00 8c 39 a5 b9 8e ff ff …9…9…
[ 3070.337719] Object 0000000032e50b40: 10 8c 39 a5 b9 8e ff ff 10 8c 39 a5 b9 8e ff ff …9…9…
[ 3070.337934] Object 000000008fbbde1d: 20 8c 39 a5 b9 8e ff ff 20 8c 39 a5 b9 8e ff ff .9… .9…
[ 3070.338111] Object 000000007a696aae: 30 8c 39 a5 b9 8e ff ff 30 8c 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.338281] Object 0000000044946a37: 40 8c 39 a5 b9 8e ff ff 40 8c 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.338456] Object 00000000d78f25f1: 50 8c 39 a5 b9 8e ff ff 50 8c 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.338660] Object 0000000082ea70b8: 60 8c 39 a5 b9 8e ff ff 60 8c 39 a5 b9 8e ff ff .9......9…
[ 3070.338846] Object 0000000070e06256: 70 8c 39 a5 b9 8e ff ff 70 8c 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.339019] Object 0000000096a6fcf8: 80 8c 39 a5 b9 8e ff ff 80 8c 39 a5 b9 8e ff ff …9…9…
[ 3070.339213] Object 0000000007148aff: 90 8c 39 a5 b9 8e ff ff 90 8c 39 a5 b9 8e ff ff …9…9…
[ 3070.339386] Object 00000000b89e18d3: a0 8c 39 a5 b9 8e ff ff a0 8c 39 a5 b9 8e ff ff …9…9…
[ 3070.339558] Object 00000000604b0bd4: b0 8c 39 a5 b9 8e ff ff b0 8c 39 a5 b9 8e ff ff …9…9…
[ 3070.339764] Object 000000006fe54ecd: c0 8c 39 a5 b9 8e ff ff c0 8c 39 a5 b9 8e ff ff …9…9…
[ 3070.340004] Object 00000000dd019521: d0 8c 39 a5 b9 8e ff ff d0 8c 39 a5 b9 8e ff ff …9…9…
[ 3070.340193] Object 00000000e43670a7: e0 8c 39 a5 b9 8e ff ff e0 8c 39 a5 b9 8e ff ff …9…9…
[ 3070.340389] Object 00000000756d6578: f0 8c 39 a5 b9 8e ff ff f0 8c 39 a5 b9 8e ff ff …9…9…
[ 3070.340632] Object 00000000d90a6a05: 00 8d 39 a5 b9 8e ff ff 00 8d 39 a5 b9 8e ff ff …9…9…
[ 3070.340874] Object 000000006eee08fa: 10 8d 39 a5 b9 8e ff ff 10 8d 39 a5 b9 8e ff ff …9…9…
[ 3070.341073] Object 000000002eddfac8: 20 8d 39 a5 b9 8e ff ff 20 8d 39 a5 b9 8e ff ff .9… .9…
[ 3070.341246] Object 00000000f2780b5f: 30 8d 39 a5 b9 8e ff ff 30 8d 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.341415] Object 000000005889fd6b: 40 8d 39 a5 b9 8e ff ff 40 8d 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.341612] Object 00000000c0e1a094: 50 8d 39 a5 b9 8e ff ff 50 8d 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.341784] Object 0000000005ef524f: 60 8d 39 a5 b9 8e ff ff 60 8d 39 a5 b9 8e ff ff .9......9…
[ 3070.341954] Object 0000000013a81cc5: 70 8d 39 a5 b9 8e ff ff 70 8d 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.342142] Object 000000001aaf2bec: 80 8d 39 a5 b9 8e ff ff 80 8d 39 a5 b9 8e ff ff …9…9…
[ 3070.342316] Object 00000000708f5782: 90 8d 39 a5 b9 8e ff ff 90 8d 39 a5 b9 8e ff ff …9…9…
[ 3070.342489] Object 0000000032d04f55: a0 8d 39 a5 b9 8e ff ff a0 8d 39 a5 b9 8e ff ff …9…9…
[ 3070.342658] Object 000000001a8657ac: b0 8d 39 a5 b9 8e ff ff b0 8d 39 a5 b9 8e ff ff …9…9…
[ 3070.342857] Object 00000000e9ff5638: c0 8d 39 a5 b9 8e ff ff c0 8d 39 a5 b9 8e ff ff …9…9…
[ 3070.343030] Object 0000000080becae9: d0 8d 39 a5 b9 8e ff ff d0 8d 39 a5 b9 8e ff ff …9…9…
[ 3070.343199] Object 00000000430df7df: e0 8d 39 a5 b9 8e ff ff e0 8d 39 a5 b9 8e ff ff …9…9…
[ 3070.343406] Object 000000003f41fd61: f0 8d 39 a5 b9 8e ff ff f0 8d 39 a5 b9 8e ff ff …9…9…
[ 3070.343581] Object 00000000401411e1: 00 8e 39 a5 b9 8e ff ff 00 8e 39 a5 b9 8e ff ff …9…9…
[ 3070.343752] Object 00000000c07223ea: 10 8e 39 a5 b9 8e ff ff 10 8e 39 a5 b9 8e ff ff …9…9…
[ 3070.343943] Object 0000000045b427c0: 20 8e 39 a5 b9 8e ff ff 20 8e 39 a5 b9 8e ff ff .9… .9…
[ 3070.344136] Object 0000000021fee8bf: 30 8e 39 a5 b9 8e ff ff 30 8e 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.344309] Object 00000000087d503a: 40 8e 39 a5 b9 8e ff ff 40 8e 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.344492] Object 0000000088743067: 50 8e 39 a5 b9 8e ff ff 50 8e 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.344679] Object 00000000aee6ac5f: 60 8e 39 a5 b9 8e ff ff 60 8e 39 a5 b9 8e ff ff .9......9…
[ 3070.344898] Object 00000000869b49e6: 70 8e 39 a5 b9 8e ff ff 70 8e 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.345073] Object 0000000016e6055d: 80 8e 39 a5 b9 8e ff ff 80 8e 39 a5 b9 8e ff ff …9…9…
[ 3070.345266] Object 00000000a71b6261: 90 8e 39 a5 b9 8e ff ff 90 8e 39 a5 b9 8e ff ff …9…9…
[ 3070.345441] Object 00000000aa4c7b46: a0 8e 39 a5 b9 8e ff ff a0 8e 39 a5 b9 8e ff ff …9…9…
[ 3070.345611] Object 0000000090cb2cc7: b0 8e 39 a5 b9 8e ff ff b0 8e 39 a5 b9 8e ff ff …9…9…
[ 3070.345807] Object 0000000059ebd8e2: c0 8e 39 a5 b9 8e ff ff c0 8e 39 a5 b9 8e ff ff …9…9…
[ 3070.345980] Object 00000000560fd39d: d0 8e 39 a5 b9 8e ff ff d0 8e 39 a5 b9 8e ff ff …9…9…
[ 3070.346150] Object 00000000a1c7ab94: e0 8e 39 a5 b9 8e ff ff e0 8e 39 a5 b9 8e ff ff …9…9…
[ 3070.346340] Object 000000006d113982: f0 8e 39 a5 b9 8e ff ff f0 8e 39 a5 b9 8e ff ff …9…9…
[ 3070.346517] Object 00000000f1bbc141: 00 8f 39 a5 b9 8e ff ff 00 8f 39 a5 b9 8e ff ff …9…9…
[ 3070.346687] Object 000000006a859797: 10 8f 39 a5 b9 8e ff ff 10 8f 39 a5 b9 8e ff ff …9…9…
[ 3070.346856] Object 000000009dac5bbd: 20 8f 39 a5 b9 8e ff ff 20 8f 39 a5 b9 8e ff ff .9… .9…
[ 3070.347047] Object 000000006f85ed28: 30 8f 39 a5 b9 8e ff ff 30 8f 39 a5 b9 8e ff ff 0.9…0.9…
[ 3070.347219] Object 000000000aeb7743: 40 8f 39 a5 b9 8e ff ff 40 8f 39 a5 b9 8e ff ff @.9…@.9…
[ 3070.347389] Object 00000000dfc45ca7: 50 8f 39 a5 b9 8e ff ff 50 8f 39 a5 b9 8e ff ff P.9…P.9…
[ 3070.347582] Object 000000008a6a22b7: 60 8f 39 a5 b9 8e ff ff 60 8f 39 a5 b9 8e ff ff .9......9…
[ 3070.347756] Object 00000000a1a1a983: 70 8f 39 a5 b9 8e ff ff 70 8f 39 a5 b9 8e ff ff p.9…p.9…
[ 3070.347933] Object 00000000bc16966f: 80 8f 39 a5 b9 8e ff ff 80 8f 39 a5 b9 8e ff ff …9…9…
[ 3070.348126] Object 00000000c507ce88: 90 8f 39 a5 b9 8e ff ff 90 8f 39 a5 b9 8e ff ff …9…9…
[ 3070.348307] Object 000000001f9b8274: a0 8f 39 a5 b9 8e ff ff a0 8f 39 a5 b9 8e ff ff …9…9…
[ 3070.348480] Object 0000000097d8bd5a: b0 8f 39 a5 b9 8e ff ff b0 8f 39 a5 b9 8e ff ff …9…9…
[ 3070.348654] Object 00000000bb639ea6: c0 8f 39 a5 b9 8e ff ff c0 8f 39 a5 b9 8e ff ff …9…9…
[ 3070.348867] Object 00000000096a7f6c: d0 8f 39 a5 b9 8e ff ff d0 8f 39 a5 b9 8e ff ff …9…9…
[ 3070.349040] Object 000000004b7232a7: e0 8f 39 a5 b9 8e ff ff e0 8f 39 a5 b9 8e ff ff …9…9…
[ 3070.349210] Object 00000000429d8da1: f0 8f 39 a5 b9 8e ff ff f0 8f 39 a5 b9 8e ff ff …9…9…
[ 3070.349405] CPU: 0 PID: 1204 Comm: rpc.mountd Kdump: loaded Tainted: G ---------r- - 4.18.0-513.9.1.el8_9.x86_64 #1
[ 3070.349410] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ 3070.349412] Call Trace:
[ 3070.349450] dump_stack+0x41/0x60
[ 3070.349469] object_err.cold.113+0x20/0x34
[ 3070.349481] free_debug_processing+0x208/0x2c0
[ 3070.349486] __slab_free+0x1f7/0x350
[ 3070.349489] ? vsnprintf+0x105/0x520
[ 3070.349515] ? __free_client+0x99/0xf0 [nfsd]
[ 3070.349554] kfree+0x22e/0x250
[ 3070.349556] __free_client+0x99/0xf0 [nfsd]
[ 3070.349581] client_info_show+0x185/0x1b0 [nfsd]
[ 3070.349606] seq_read+0x163/0x420
[ 3070.349616] vfs_read+0x91/0x150
[ 3070.349625] ksys_read+0x4f/0xb0
[ 3070.349628] do_syscall_64+0x5b/0x1b0
[ 3070.349634] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[ 3070.349649] RIP: 0033:0x7f1a5f0e8d45
[ 3070.349653] Code: fe ff ff 50 48 8d 3d 32 cb 06 00 e8 35 ee 01 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 f5 49 2a 00 8b 00 85 c0 75 0f 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[ 3070.349656] RSP: 002b:00007fff90194838 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 3070.349659] RAX: ffffffffffffffda RBX: 0000559bfd31f6e0 RCX: 00007f1a5f0e8d45
[ 3070.349661] RDX: 0000000000001000 RSI: 0000559bfd345f50 RDI: 000000000000000d
[ 3070.349662] RBP: 0000000000000d68 R08: 0000000000000003 R09: 0000559bfd31d20e
[ 3070.349664] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f1a5f384860
[ 3070.349665] R13: 00007f1a5f3853a0 R14: 00000000000007ff R15: 0000559bfd31f6e0
[ 3070.349668] Disabling lock debugging due to kernel taint

3

(part 2 of dump)

[ 3070.349729] FIX kmalloc-4k: Object at 0x0000000049976705 not freed
[ 3070.349930] =============================================================================
[ 3070.350106] BUG kmalloc-64 (Tainted: G B ---------r- -): Object already free
[ 3070.350286] -----------------------------------------------------------------------------

[ 3070.350642] Slab 0x000000007fd0ea80 objects=64 used=41 fp=0x00000000984f1d98 flags=0x17ffffc0000101(locked|slab|node=0|zone=2|lastcpupid=0x1fffff)
[ 3070.350854] Object 0x00000000db53d59d @offset=1344 fp=0x0000000000000000

[ 3070.351227] Bytes b4 000000001ab59106: 00 a2 80 74 b7 8e ff ff 03 33 30 30 36 36 30 33 …t…3006603
[ 3070.351431] Object 00000000db53d59d: 35 65 35 66 39 34 39 30 2d 61 36 34 33 2d 63 64 5e5f9490-a643-cd
[ 3070.351703] Object 000000003358db78: 36 38 2d 31 62 61 61 2d 30 30 35 30 35 36 39 63 68-1baa-0050569c
[ 3070.352026] Object 00000000913faa2c: 7c 86 57 f7 cf d1 55 8c 30 2c 20 52 65 6c 65 61 |.W…U.0, Relea
[ 3070.352297] Object 0000000080b9da39: 73 65 62 75 69 6c 64 2d 31 33 30 30 36 36 30 33 sebuild-13006603
[ 3070.352526] CPU: 0 PID: 1204 Comm: rpc.mountd Kdump: loaded Tainted: G B ---------r- - 4.18.0-513.9.1.el8_9.x86_64 #1
[ 3070.352531] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ 3070.352533] Call Trace:
[ 3070.352539] dump_stack+0x41/0x60
[ 3070.352549] object_err.cold.113+0x20/0x34
[ 3070.352557] free_debug_processing+0x208/0x2c0
[ 3070.352560] __slab_free+0x1f7/0x350
[ 3070.352563] ? vsnprintf+0x105/0x520
[ 3070.352572] ? __free_client+0xa5/0xf0 [nfsd]
[ 3070.352611] kfree+0x22e/0x250
[ 3070.352615] __free_client+0xa5/0xf0 [nfsd]
[ 3070.352648] client_info_show+0x185/0x1b0 [nfsd]
[ 3070.352670] seq_read+0x163/0x420
[ 3070.352675] vfs_read+0x91/0x150
[ 3070.352680] ksys_read+0x4f/0xb0
[ 3070.352683] do_syscall_64+0x5b/0x1b0
[ 3070.352689] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[ 3070.352697] RIP: 0033:0x7f1a5f0e8d45
[ 3070.352700] Code: fe ff ff 50 48 8d 3d 32 cb 06 00 e8 35 ee 01 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 f5 49 2a 00 8b 00 85 c0 75 0f 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[ 3070.352703] RSP: 002b:00007fff90194838 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 3070.352707] RAX: ffffffffffffffda RBX: 0000559bfd31f6e0 RCX: 00007f1a5f0e8d45
[ 3070.352709] RDX: 0000000000001000 RSI: 0000559bfd345f50 RDI: 000000000000000d
[ 3070.352711] RBP: 0000000000000d68 R08: 0000000000000003 R09: 0000559bfd31d20e
[ 3070.352713] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f1a5f384860
[ 3070.352714] R13: 00007f1a5f3853a0 R14: 00000000000007ff R15: 0000559bfd31f6e0
[ 3070.352851] FIX kmalloc-64: Object at 0x00000000db53d59d not freed
[ 3070.353096] =============================================================================
[ 3070.353328] BUG kmalloc-16 (Tainted: G B ---------r- -): Object already free
[ 3070.353563] -----------------------------------------------------------------------------

[ 3070.354001] Slab 0x00000000729d7a22 objects=256 used=118 fp=0x0000000070d2e593 flags=0x17ffffc0000101(locked|slab|node=0|zone=2|lastcpupid=0x1fffff)
[ 3070.354260] Object 0x0000000070d2e593 @offset=2512 fp=0x0000000000000000

[ 3070.354708] Bytes b4 00000000784597ee: 00 00 00 00 00 00 40 73 01 00 80 15 22 0b 00 00 …@s…"…
[ 3070.354967] Object 0000000070d2e593: 76 6d 77 61 72 65 2e 63 58 0f bd 11 02 53 3b f1 vmware.cX…S;.
[ 3070.355220] CPU: 0 PID: 1204 Comm: rpc.mountd Kdump: loaded Tainted: G B ---------r- - 4.18.0-513.9.1.el8_9.x86_64 #1
[ 3070.355224] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ 3070.355226] Call Trace:
[ 3070.355230] dump_stack+0x41/0x60
[ 3070.355239] object_err.cold.113+0x20/0x34
[ 3070.355245] free_debug_processing+0x208/0x2c0
[ 3070.355249] __slab_free+0x1f7/0x350
[ 3070.355251] ? vsnprintf+0x105/0x520
[ 3070.355258] ? __free_client+0xb1/0xf0 [nfsd]
[ 3070.355292] kfree+0x22e/0x250
[ 3070.355295] __free_client+0xb1/0xf0 [nfsd]
[ 3070.355318] client_info_show+0x185/0x1b0 [nfsd]
[ 3070.355340] seq_read+0x163/0x420
[ 3070.355345] vfs_read+0x91/0x150
[ 3070.355350] ksys_read+0x4f/0xb0
[ 3070.355352] do_syscall_64+0x5b/0x1b0
[ 3070.355358] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[ 3070.355365] RIP: 0033:0x7f1a5f0e8d45
[ 3070.355369] Code: fe ff ff 50 48 8d 3d 32 cb 06 00 e8 35 ee 01 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 f5 49 2a 00 8b 00 85 c0 75 0f 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[ 3070.355371] RSP: 002b:00007fff90194838 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 3070.355374] RAX: ffffffffffffffda RBX: 0000559bfd31f6e0 RCX: 00007f1a5f0e8d45
[ 3070.355375] RDX: 0000000000001000 RSI: 0000559bfd345f50 RDI: 000000000000000d
[ 3070.355377] RBP: 0000000000000d68 R08: 0000000000000003 R09: 0000559bfd31d20e
[ 3070.355379] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f1a5f384860
[ 3070.355380] R13: 00007f1a5f3853a0 R14: 00000000000007ff R15: 0000559bfd31f6e0
[ 3070.355465] FIX kmalloc-16: Object at 0x0000000070d2e593 not freed
[ 3070.355766] =============================================================================
[ 3070.356027] BUG kmalloc-64 (Tainted: G B ---------r- -): Object already free
[ 3070.356289] -----------------------------------------------------------------------------

[ 3070.356790] Slab 0x00000000ff76cd98 objects=64 used=62 fp=0x000000008e7e5e47 flags=0x17ffffc0000101(locked|slab|node=0|zone=2|lastcpupid=0x1fffff)
[ 3070.357114] Object 0x000000008e7e5e47 @offset=64 fp=0x0000000000000000

[ 3070.357629] Bytes b4 00000000bd10447a: 64 33 64 01 00 00 00 0b 04 14 92 7b 36 36 30 33 d3d…{6603
[ 3070.357923] Object 000000008e7e5e47: 56 4d 77 61 72 65 20 4e 46 53 76 34 2e 31 20 43 VMware NFSv4.1 C
[ 3070.358209] Object 000000000ef133e0: 6c 69 65 6e 74 3a 20 56 4d 77 61 72 65 20 45 53 lient: VMware ES
[ 3070.358506] Object 000000004bee45b8: 7c 86 57 f9 cd 9f c0 8c 30 2c 20 52 65 6c 65 61 |.W…0, Relea
[ 3070.358800] Object 000000000ce85217: 73 65 62 75 69 6c 64 2d 31 33 30 30 36 36 30 33 sebuild-13006603
[ 3070.359099] CPU: 0 PID: 1204 Comm: rpc.mountd Kdump: loaded Tainted: G B ---------r- - 4.18.0-513.9.1.el8_9.x86_64 #1
[ 3070.359104] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ 3070.359106] Call Trace:
[ 3070.359113] dump_stack+0x41/0x60
[ 3070.359122] object_err.cold.113+0x20/0x34
[ 3070.359129] free_debug_processing+0x208/0x2c0
[ 3070.359134] __slab_free+0x1f7/0x350
[ 3070.359136] ? vsnprintf+0x105/0x520
[ 3070.359145] ? __free_client+0xbd/0xf0 [nfsd]
[ 3070.359180] kfree+0x22e/0x250
[ 3070.359182] __free_client+0xbd/0xf0 [nfsd]
[ 3070.359206] client_info_show+0x185/0x1b0 [nfsd]
[ 3070.359228] seq_read+0x163/0x420
[ 3070.359233] vfs_read+0x91/0x150
[ 3070.359238] ksys_read+0x4f/0xb0
[ 3070.359241] do_syscall_64+0x5b/0x1b0
[ 3070.359247] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[ 3070.359252] RIP: 0033:0x7f1a5f0e8d45
[ 3070.359256] Code: fe ff ff 50 48 8d 3d 32 cb 06 00 e8 35 ee 01 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 f5 49 2a 00 8b 00 85 c0 75 0f 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[ 3070.359269] RSP: 002b:00007fff90194838 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 3070.359272] RAX: ffffffffffffffda RBX: 0000559bfd31f6e0 RCX: 00007f1a5f0e8d45
[ 3070.359273] RDX: 0000000000001000 RSI: 0000559bfd345f50 RDI: 000000000000000d
[ 3070.359275] RBP: 0000000000000d68 R08: 0000000000000003 R09: 0000559bfd31d20e
[ 3070.359276] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f1a5f384860
[ 3070.359278] R13: 00007f1a5f3853a0 R14: 00000000000007ff R15: 0000559bfd31f6e0
[ 3070.359372] FIX kmalloc-64: Object at 0x000000008e7e5e47 not freed
[ 3070.359690] =============================================================================
[ 3070.360002] BUG nfsd4_clients (Tainted: G B ---------r- -): Object already free
[ 3070.360314] -----------------------------------------------------------------------------

[ 3070.360987] Slab 0x00000000662ca8f5 objects=22 used=20 fp=0x00000000109d339f flags=0x17ffffc0008101(locked|slab|head|node=0|zone=2|lastcpupid=0x1fffff)
[ 3070.361330] Object 0x00000000109d339f @offset=17280 fp=0x0000000000000000

[ 3070.361967] Bytes b4 00000000373f7b95: 68 c3 58 67 b7 8e ff ff 00 00 00 00 00 00 00 00 h.Xg…
[ 3070.362296] Object 00000000109d339f: 80 c3 58 67 b7 8e ff ff 80 c3 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.362631] Object 00000000c48e15b5: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.362991] Object 000000009e4f2d27: 00 00 00 00 00 00 00 00 00 30 1b 56 b9 8e ff ff …0.V…
[ 3070.363345] Object 000000002a3fff79: b0 c3 58 67 b7 8e ff ff b0 c3 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.363692] Object 0000000008d88103: 00 00 00 00 04 00 80 00 00 00 00 00 00 00 00 00 …
[ 3070.364077] Object 000000007fa552d7: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.364420] Object 000000003ee9e840: e0 c3 58 67 b7 8e ff ff e0 c3 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.364788] Object 000000003a138605: f0 c3 58 67 b7 8e ff ff f0 c3 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.365146] Object 000000005a23eb5a: 00 c4 58 67 b7 8e ff ff 00 c4 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.365548] Object 00000000890ca36a: 10 c4 58 67 b7 8e ff ff 10 c4 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.365897] Object 00000000e7fe05bc: 24 00 00 00 00 00 00 00 40 20 3e 7f b7 8e ff ff $…@ >…
[ 3070.366226] Object 00000000bfea4abc: dd b5 04 65 00 00 00 00 00 00 00 00 00 00 00 00 …e…
[ 3070.366557] Object 000000009e9883f0: 02 00 03 81 0a 88 dc 20 00 00 00 00 00 00 00 00 … …
[ 3070.366896] Object 000000006abc9bcf: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.367216] Object 00000000696b9eb3: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.367535] Object 0000000034f90868: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.367892] Object 00000000ee6119a1: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.368200] Object 00000000bf359401: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.368506] Object 00000000f216f90f: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.368835] Object 000000004d876f9e: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.369143] Object 000000006867a9bf: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.369435] Object 0000000076387378: 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 …
[ 3070.369740] Object 000000008415a74e: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.370027] Object 00000000b2f36fe2: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.370302] Object 00000000abba3065: 06 18 97 65 b7 0d 76 b8 f8 23 97 65 ec bb 4d c3 …e…v…#.e…M.
[ 3070.370590] Object 00000000f567f302: 00 00 00 00 00 00 00 00 0a 00 00 00 00 00 00 00 …
[ 3070.370894] Object 0000000060ee471e: 00 72 7c 72 b7 8e ff ff 40 00 00 00 00 00 00 00 .r|r…@…
[ 3070.371185] Object 00000000ea930993: 40 e8 5e 9e b9 8e ff ff 8b 89 9a 5c 00 00 00 00 @.^….…
[ 3070.371463] Object 000000005c2d6ddf: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.371744] Object 000000009d1caa2d: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.372058] Object 00000000662a74cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.372389] Object 000000002c342ca5: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.372667] Object 000000007576447e: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.372963] Object 000000008a6ac1be: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.373232] Object 00000000e26933eb: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.373493] Object 000000000eb1a1cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.373763] Object 0000000012163dbf: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.374020] Object 00000000d5088a88: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.374262] Object 00000000cf8b038f: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.374500] Object 000000001265a130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.374755] Object 00000000da868036: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.375125] Object 00000000eceae288: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.375446] Object 00000000fc03da9e: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.375756] Object 0000000059cc4fd9: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.375990] Object 00000000d01e595b: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.376204] Object 0000000037059629: 34 38 80 86 62 4b 17 dc 00 00 00 00 00 00 00 00 48…bK…
[ 3070.376414] Object 00000000be20fc86: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.376628] Object 00000000f3a8e36a: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 …
[ 3070.376895] Object 000000001be65e74: 80 c3 58 67 b7 8e ff ff a0 21 93 c0 ff ff ff ff …Xg…!..
[ 3070.377192] Object 0000000006ed413a: 80 c6 58 67 b7 8e ff ff 80 c6 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.377427] Object 0000000043a4897f: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.377646] Object 00000000dd449e72: 00 01 00 00 00 00 00 00 b8 c6 58 67 b7 8e ff ff …Xg…
[ 3070.377901] Object 0000000062ce9646: b8 c6 58 67 b7 8e ff ff 40 67 92 c0 ff ff ff ff …Xg…@g…
[ 3070.378105] Object 0000000077527dad: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.378306] Object 00000000811720ab: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.378539] Object 00000000160007e8: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.378737] Object 0000000006674ffd: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.378963] Object 00000000d1396c2a: 10 c7 58 67 b7 8e ff ff 10 c7 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.379176] Object 00000000253a1c45: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.379378] Object 00000000de629f7e: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.379609] Object 000000009fae531d: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.379822] Object 00000000e950138d: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.380022] Object 000000006a23c637: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.380239] Object 00000000fd6eddb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.380432] Object 0000000092e23129: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.380625] Object 00000000f7778a5a: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.380882] Object 00000000615a69c4: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.381077] Object 00000000b4a5b716: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.381265] Object 000000007ed996e6: 00 00 00 00 00 00 00 00 80 1a 86 a6 b9 8e ff ff …
[ 3070.381485] Object 000000008550cba9: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.381686] Object 000000006aa41408: e0 c7 58 67 b7 8e ff ff e0 c7 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.381885] Object 000000006e093215: f0 c7 58 67 b7 8e ff ff f0 c7 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.382110] Object 000000008f82b273: 00 c8 58 67 b7 8e ff ff 00 c8 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.382313] Object 000000001380f857: 10 c8 58 67 b7 8e ff ff 10 c8 58 67 b7 8e ff ff …Xg…Xg…
[ 3070.382557] Object 0000000013952e5f: 00 00 00 00 00 00 00 00 28 c8 58 67 b7 8e ff ff …(.Xg…
[ 3070.382851] Object 00000000ebb3a2d9: 28 c8 58 67 b7 8e ff ff 00 00 00 00 00 00 00 00 (.Xg…
[ 3070.383092] Object 00000000120999ec: e0 ff ff ff 0f 00 00 00 48 c8 58 67 b7 8e ff ff …H.Xg…
[ 3070.383323] Object 0000000076a14121: 48 c8 58 67 b7 8e ff ff f0 1c 83 c0 ff ff ff ff H.Xg…
[ 3070.383527] Object 000000006b49ca35: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.383732] Object 000000002aa9e5a7: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.383957] Object 00000000ad60968f: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.384163] Object 000000001f2cb8d7: 00 00 00 00 00 00 00 00 b0 6a b1 98 ff ff ff ff …j…
[ 3070.384391] Object 0000000027dcb137: 03 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 … …
[ 3070.384609] Object 00000000733a9950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.384866] Object 00000000490b3e4d: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.385113] Object 000000003c3dbdb1: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.385399] Object 0000000015cccde5: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
[ 3070.385663] Object 00000000eecea2b3: 00 00 00 00 00 00 00 00 5b 3d 93 c0 ff ff ff ff …[=…
[ 3070.385888] Object 00000000b23cc9e1: 40 fb 93 9a ff ff ff ff 08 c9 58 67 b7 8e ff ff @…Xg…
[ 3070.386109] Object 00000000e94ab451: 08 c9 58 67 b7 8e ff ff 00 00 00 00 00 00 00 00 …Xg…
[ 3070.386364] CPU: 0 PID: 1204 Comm: rpc.mountd Kdump: loaded Tainted: G B ---------r- - 4.18.0-513.9.1.el8_9.x86_64 #1
[ 3070.386371] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ 3070.386373] Call Trace:
[ 3070.386379] dump_stack+0x41/0x60
[ 3070.386390] object_err.cold.113+0x20/0x34
[ 3070.386397] free_debug_processing+0x208/0x2c0
[ 3070.386401] __slab_free+0x1f7/0x350
[ 3070.386408] ? client_info_show+0x185/0x1b0 [nfsd]
[ 3070.386445] kmem_cache_free+0x2d6/0x300
[ 3070.386449] client_info_show+0x185/0x1b0 [nfsd]
[ 3070.386473] seq_read+0x163/0x420
[ 3070.386479] vfs_read+0x91/0x150
[ 3070.386485] ksys_read+0x4f/0xb0
[ 3070.386488] do_syscall_64+0x5b/0x1b0
[ 3070.386495] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[ 3070.386508] RIP: 0033:0x7f1a5f0e8d45
[ 3070.386513] Code: fe ff ff 50 48 8d 3d 32 cb 06 00 e8 35 ee 01 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 f5 49 2a 00 8b 00 85 c0 75 0f 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[ 3070.386516] RSP: 002b:00007fff90194838 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 3070.386520] RAX: ffffffffffffffda RBX: 0000559bfd31f6e0 RCX: 00007f1a5f0e8d45
[ 3070.386522] RDX: 0000000000001000 RSI: 0000559bfd345f50 RDI: 000000000000000d
[ 3070.386524] RBP: 0000000000000d68 R08: 0000000000000003 R09: 0000559bfd31d20e
[ 3070.386525] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f1a5f384860
[ 3070.386527] R13: 00007f1a5f3853a0 R14: 00000000000007ff R15: 0000559bfd31f6e0
[ 3070.386684] FIX nfsd4_clients: Object at 0x00000000109d339f not freed

4

I found the source of the problem, but not the reason for the corruptions.

I had an ESXi host that went crazy and was issuing thousands of requests to the
NFS server per second. The ESXi host had an NFS share mounted (a share that
serves ISO images), but no VM was using that share, so it should have been idle.
I dismounted the share from ESXi (without issue) and the NFS traffic stopped.
I do not know why the ESXi host was generating this traffic.

The ASCII dumps of the memory records I previously posted look like strings
related to the ESXi NFS client are all over the memory and perhaps somehow
these strings got copied in a careless way that overwrote kernel memory structures?