Environment
- EDR Server: All versions
- EDR Sensors: All Versions
Symptoms
- WebUI console Server Dashboard shows Aggregate sensor event queue continuing to grow excessively larger.
- Seeing these JVM GC messages orver 20% in /var/log/messages of one or moreEDR nodes
Feb 12 14:57:25 prn-cb06 cb-enterprised[4139]: cb.enterprise.tasks.server_health_monitor.indicators.garbage_collection - JVM GC for cb-datastore is at 203.7%. Current threshold is 20.0%
Feb 12 14:58:27 prn-cb06 cb-enterprised[4139]: cb.enterprise.tasks.server_health_monitor.indicators.garbage_collection - JVM GC for cb-datastore is at 143.3%. Current threshold is 20.0%
- /var/log/cb/datastore/debug.log shows these WARNings:
2019-02-12 14:56:01,819 - [WARN] - from org.eclipse.jetty.http.HttpParser in qtp2059904228-932
badMessage: java.lang.IllegalStateException: too much data after closed for
HttpChannelOverHttp@6b8eebb2{r=1,c=false,a=IDLE,uri=-}
- /var/log/cb/access.log showing excessive "503" errors:
cat /var/log/cb/nginx/access.log | cut -d'"' -f3 | cut -d' ' -f2 | sort | uniq -c
1406550 200
1 204
18 400
39 402
48 403
827 408
273 499
12224 502
232598 503 << excessive 503 errors
73882 504
Cause
SOLR may not have enough allocated memory. Try increasing JAVA memory like so on all the nodes:
Resolution
- For EDR Server versions 7.x see this article on how to allocate more RAM to both Datastore and SOLR JVM's: https://community.carbonblack.com/t5/Knowledge-Base/EDR-Sensor-backlog-growing-with-many-503s-in-Ngi... increase available Java memory on all nodes:.
- For earlier 6.0-6.2.x EDR servers, increase available Java memory on all nodes:
- Edit /etc/cb/solr5/solr.in.sh
- Change 0.40 to 0.60 as so:
XMAX=`grep MemTotal /proc/meminfo | awk '{printf("%dM", 0.40*$2/1024)}'`
To
XMAX=`grep MemTotal /proc/meminfo | awk '{printf("%dM", 0.60*$2/1024)}'`
- Then restart services:
/usr/share/cb/cbluster stop
/usr/share/cb/cbcluster start
Additional Notes
Future upgrades of the server Server from 6.2.x will get an rpmnew file for the solr.in.sh.
Related Content
