EDR: Sensor backlog growing with many 503s in Nginx

[#1]

Environment

• EDR Server: All Versions

Symptoms

• Nginx shows more 503 responses for sensor communication than 200 responses
• Slow response with queries in Console
• /var/log/cb/enterprise/enterprise.log shows Datastore garbage collection extremely over threshold repeatedly

JVM GC for cb-datastore is at ###%. Current threshold is 20.0%

Cause

Datastore is overwhelmed by the amount of incoming data and this causes increased memory usage. An increase in memory usage often leads to more Java garbage collection, which can have the side effect of slowing sensor event ingestion.

Resolution

1. Increase the allotted memory for Datastore in /etc/cb/cb.conf
   • Versions 6.2.2 and earlier:
     • Change DatastoreJvmMax=10% to DatastoreJvmMax=20%
   • Versions 6.2.3 and higher:
     • Add following content to the /etc/cb/cb.conf file:

# ============================== # SECTION - CbAllocate Options # ============================== #
# Enables the dynamic memory allocation for Solr Jvm services 
EnableDynamicJvmMemoryAllocation = False 

# The maximum % of memory allowed to solr (Used when dynamic calculation is off) 
MaximumSolrMemoryPercent = 40 

# The minimum % of memory allowed to solr (Used when dynamic calculation is off) 
MinimumSolrMemoryPercent = 10 

# The maximum % of memory allowed to datastore (Used when dynamic calculation is on or off) 
MaximumDatastoreMemoryPercent = 20 

# The minimum % of memory allowed to datastore (Used when dynamic calculation is on or off) 
MinimumDatastoreMemoryPercent = 5 

2. Save and exit /etc/cb/cb.conf file
3. Restart EDR services. EDR: How to Restart Server Services
4. Run /usr/share/cb/virtualenv/bin/python -m cb.maintenance.cballocate.main -i datastore to verify changes have been made in settings.

Additional Notes

• Set 'EnableDynamicJvmMemoryAllocation = True' to turn on dynamic memory allocation.
• Once ingestion is done, the memory can be reduced back down to the original setting.
• Depending on the size of backlog, document ingestion can still take some time to go down. 
• DatastoreJvmMax has been depreciated in versions v6.2.3

Related Content

EDR: How to Restart Server Services