EDR: Server will not start - Solr read timed out

[#1]

Environment

• EDR Server: All versions (formerly CB Response)

Symptoms

• Service startup hangs on Solr
• Solr log shows

<warning>  cb.utils.solr_client - Failed request http://127.0.0.1:8080/solr/admin/cores?action=STATUS&wt=json&indexInfo=true: HTTPConnectionPool(host='127.0.0.1', port=8080): Read timed out. (read timeout=60)

• job-runner.log shows

carbonblack.inl.gov cb-sensorservices[21972]: cb.core.config.active_grid_config - Key 0 not found in hazelcast

• Event retention settings in /etc/cb/cb.conf increased beyond default settings

Cause

There are too many cores for Solr to load before timing out

Resolution

The number of active cores must be reduced manually 

1. Stop Solr

service cb-solr stop

2. Create a backup directory outside of /var/cb/data/solr*
3. Move older cores from /var/cb/data/solr*/cbevents* to the backup directory
4. If running in RHEL or CentOS 7.x, verify all services are stopped
5. Start services

service cb-enterprise start

 

Additional Notes

• The recommended number of Solr cores at any time is up to 12 cores or 30 days worth of data, whichever is smaller
• If more than 30 days of cores are required, consider cold storage or forwarding events to a SIEM

Related Content

EDR: How to enable event cold storage
carbonblack/cb-event-forwarder