Just Published! Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

EDR: Aggregate sensor event queue is growing too large

EDR: Aggregate sensor event queue is growing too large

Environment

  • EDR Server: All versions
  • EDR Sensors: All Versions

Symptoms

  • WebUI console Server Dashboard shows Aggregate sensor event queue continuing to grow excessively larger.
  • Seeing these JVM GC messages orver 20% in /var/log/messages of one or moreEDR nodes
Feb 12 14:57:25 prn-cb06 cb-enterprised[4139]: cb.enterprise.tasks.server_health_monitor.indicators.garbage_collection - JVM GC for cb-datastore is at 203.7%. Current threshold is 20.0% 
Feb 12 14:58:27 prn-cb06 cb-enterprised[4139]: cb.enterprise.tasks.server_health_monitor.indicators.garbage_collection - JVM GC for cb-datastore is at 143.3%. Current threshold is 20.0%
  • /var/log/cb/datastore/debug.log shows these WARNings:
2019-02-12 14:56:01,819 - [WARN] - from org.eclipse.jetty.http.HttpParser in qtp2059904228-932 
badMessage: java.lang.IllegalStateException: too much data after closed for 
HttpChannelOverHttp@6b8eebb2{r=1,c=false,a=IDLE,uri=-}
 
  • /var/log/cb/access.log showing excessive "503" errors:
cat /var/log/cb/nginx/access.log | cut -d'"' -f3 | cut -d' ' -f2 | sort | uniq -c 

1406550 200 
1 204 
18 400 
39 402 
48 403 
827 408 
273 499 
12224 502 
232598 503       << excessive 503 errors
73882 504

Cause

SOLR may not have enough allocated memory. Try increasing JAVA memory like so on all the nodes:

Resolution

  1. For EDR Server versions 7.x see this article on how to allocate more RAM to both Datastore and SOLR JVM's: https://community.carbonblack.com/t5/Knowledge-Base/EDR-Sensor-backlog-growing-with-many-503s-in-Ngi...  increase available Java memory on all nodes:. 
  2. For earlier 6.0-6.2.x EDR servers, increase available Java memory on all nodes: 
    1. Edit /etc/cb/solr5/solr.in.sh
    2. Change 0.40 to 0.60 as so: 
XMAX=`grep MemTotal /proc/meminfo | awk '{printf("%dM", 0.40*$2/1024)}'`
To 
XMAX=`grep MemTotal /proc/meminfo | awk '{printf("%dM", 0.60*$2/1024)}'`
  1. Then restart services:
/usr/share/cb/cbluster stop 
/usr/share/cb/cbcluster start

Additional Notes

Future upgrades of the server Server from 6.2.x will get an rpmnew file for the solr.in.sh.

Related Content


Labels (1)
Tags (2)
Was this article helpful? Yes No
0% helpful (0/2)
Article Information
Author:
Creation Date:
‎09-09-2020
Views:
1830
Contributors