Just Published! Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

EDR: SOLR cores optimization failing and causing HPROF and crashes

EDR: SOLR cores optimization failing and causing HPROF and crashes

Environment

  • EDR Server: 6.x and higher

Symptoms

  • EDR server is generating HPROF files and crashes
  • WebUI performance is poor, with searches and watchlists taking a long time to run
  • Error in the /var/log/cb/SOLR/debug.log
    • 2018-08-30 10:44:30 [3712] <warning>  [solr_optimize] Failed request http://127.0.0.1:8080/solr/cbevents_2018_08_19_1902/update?optimize=true&waitFlush=True&waitSearcher=false&wt=json&maxSegments=10: ('Connection aborted.', BadStatusLine("''",)) 

Cause

  • The solr_optimize job starts and attempts to run on non-optimized cores. The optimize will run on solr from anywhere from 30 min to 10+ hours but every time it fails.

Resolution

  1. Connect to each minion in the cluster
  2. vi /etc/cb/cb.conf
  3. Update the values:
SolrTimePartitioningOptimizeMaxSegments=40
  1. Add this value to the end of the file:
MaximumSolrMemoryPercent=60
  1. Restart the cluster
/usr/share/cb/cbcluster stop
/usr/share/cb/cbcluster start
  1. After the SOLR cores complete optimization revert the cb.conf configuration back to its original values
SolrTimePartitioningOptimizeMaxSegments=10
#SolrTimePartitioningOptimizeMaxSegments=40

Additional Notes

  • This was a change made temporarilty until the SOLR cores were able to optimize successfully

Labels (1)
Was this article helpful? Yes No
No ratings
Article Information
Author:
Creation Date:
‎11-21-2018
Views:
1055
Contributors