Firewalld-0.9.11-1 update drastically slowed ipset importing?

Support
1

In late Sept 2023 an update for firewalld was released (firewalld-0.9.11-1) and we believe since that update we have seen a massive decrease in performance with regards reloading the firewall when large ipsets are in use (ie. a ipset that contains 12,000 IP addresses).

I’d be intrigued to see if anyone else has also been impacted by this bug or if anyone can help to confirm if they see the same?

Our use case: we have hundreds of AlmaLinux machines which have a custom ipset which contains 12,000 IP addresses which periodically rotate, the ipset always contains that quantity (12k). On the previous version of firewalld-0.9.3-13 when we would reload the firewall with firewall-cmd --reload it would complete in around 6 seconds - at that point firewalld would pass the ipset to the backend firewall.

Our firewalld setup is using nftables behind the scenes not iptables.

After upgrading from firewalld-0.9.3-13 to firewalld-0.9.11-1 the same reloading operation will take a matter of minutes, it can vary from 2 - 5 minutes and causes disruption to network connections which is noticeable.

If we downgrade the firewalld package it reverts to taking a mere 6 seconds to reload. Bearing in mind firewalld is ingesting our .xml ipset which contains 12k ip addresses and passing that to nftables during that time, then reloading.

If the firewall backend is swapped out (switching from nftables to iptables in the firewalld.conf) then the problem goes away. So it appears to be a bug in the latest version of firewalld + nftables when ingesting large ipsets - it has drastically slowed down the act of reloading and can cause disruption.

From our testing it is taking anywhere from 25-55x longer to reload, so quite the performance decrease. This has been tested on a variety of AlmaLinux boxes with varying specifications.

Is there anyone else out there handling ipsets which are 10k+ in combination with firewalld + nftables? If so, do you notice firewalld taking significantly longer to reload nowadays?

If it helps anyone to test this I can perhaps share our custom ipset .xml file which contains the 12k IP addresses. Thanks for your time and help.

2

I don’t use ipsets and nothing big like 10k+ anyway. Where I have something, it is nftables named set, and even then almost nothing.

However, I’m sure that I have seen “ipset slow” mentioned with either el8 or el9. Probably before el9 – not with recent FirewallD. Can’t remember whether it was ipset, kernel, or FirewallD that was primary suspect in those cases. Word “memory hog” comes to mind. Perhaps those old issues have long been solved and you have something novel.

PS. The ‘iptables’ in el8 and el9 is wrapper to ‘nft’, so use of iptables backend in either would be an extra layer of translation – but ‘ipset’ is a separate tool so what do I know?

3

Interesting, thanks for chiming in.

If it helps anyone to validate to see if you get the same results as us then I will share the ipset XML file below that we are using.

I’m hoping someone may be able to corroborate that when reloading firewalld with this ipset in place with the package firewalld-0.9.11-1 (post Sept 2023) it’ll take ages, yet the pre Sept package firewalld-0.9.3-13 will reload in seconds.

The XML contents below could be placed in a file at: /etc/firewalld/ipsets/beacon.xml

It wouldn’t let me attach a .xml so I’ve published it here:
https://drive.google.com/file/d/1JGrTS60hRTpCffnBi-dAGu04HLgXR8g0/view?usp=sharing

Then:


firewall-cmd --permanent --zone=drop --add-source=ipset:beacon

Then:


firewall-cmd --reload

If anyone is curious what these IP addresses are - they are “bad” IPs which we have witnessed trying to brute-force their way into our systems.

4

If anyone is interested this conversation has continued over on the Github Issue tracker:
https://github.com/AlmaLinux/almalinux-deploy/issues/186#issuecomment-1857895214

It appears to be becoming apparent that nftables may in fact be the issue, causing a knock-on performance hit with firewalld. After upgrading nftables the issue is only witnessed after a reload/restart of firewalld. See the thread on Github for more info.

1 Like