EDR: SOLR cores optimization failing and causing HPROF and crashes

[#1]

Environment

• EDR Server: 6.x and higher

Symptoms

• EDR server is generating HPROF files and crashes
• WebUI performance is poor, with searches and watchlists taking a long time to run
• Error in the /var/log/cb/SOLR/debug.log

  • 2018-08-30 10:44:30 [3712] <warning>  [solr_optimize] Failed request http://127.0.0.1:8080/solr/cbevents_2018_08_19_1902/update?optimize=true&waitFlush=True&waitSearcher=false&wt=json&maxSegments=10: ('Connection aborted.', BadStatusLine("''",)) 

Cause

• The solr_optimize job starts and attempts to run on non-optimized cores. The optimize will run on solr from anywhere from 30 min to 10+ hours but every time it fails.

Resolution

1. Connect to each minion in the cluster
2. vi /etc/cb/cb.conf
3. Update the values:

SolrTimePartitioningOptimizeMaxSegments=40

4. Add this value to the end of the file:

MaximumSolrMemoryPercent=60

5. Restart the cluster

/usr/share/cb/cbcluster stop

/usr/share/cb/cbcluster start

6. After the SOLR cores complete optimization revert the cb.conf configuration back to its original values

SolrTimePartitioningOptimizeMaxSegments=10

#SolrTimePartitioningOptimizeMaxSegments=40

Additional Notes

• This was a change made temporarilty until the SOLR cores were able to optimize successfully