6.1.0 CD UI Performance and Sensor Comms Issues due to OS Netfilter Connection Tracking Max Reached

[#1]

Version

Cb Response 6.X (Large Clusters)

Issue

In 6.1, the sensor check-in pipeline was greatly improved allowing for an increase check-in interval. Many additional improvements were that also increased inter-node communications. This increase in communications (connections) required the OS to manage more connections. In certain scenarios and in larger deployments, the OS Netfilter connection tracking were getting maxed out. This is causing connection to be dropped, which is causing issues loading UI pages, resulting in the perception of slow UI performance.

Symptoms

/var/log/messages

kernel: nf_conntrack: table full, dropping packet

Solution

Workaround

Increase nf_conntrack_max setting:

1. Check current settings
   

   cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max

   or
   

   /sbin/sysctl -a|grep -i nf_conntrack_max

2. Check current connections tracked
   

   /sbin/sysctl net.netfilter.nf_conntrack_count

3. If connection max are met or close:
   1. Increase connection tracking max to 262144
      

      sysctl -w net.netfilter.nf_conntrack_max=262144

   2. Update hashsize proportionally with nf_conntrack_max change (new value should be nf_conntrack_max/4)
      

      echo 65536 > /sys/module/nf_conntrack/parameters/hashsize

4. Save changes permanently
   1. Modify or add the following line item to /etc/sysctl.conf:
      

      net.netfilter.nf_conntrack_count = 262144