The solr_optimize job starts and attempts to run on non-optimized cores. The optimize will run on solr from anywhere from 30 min to 10+ hours but every time it fails.
Resolution
Connect to each minion in the cluster
vi /etc/cb/cb.conf
Update the values:
SolrTimePartitioningOptimizeMaxSegments=40
Add this value to the end of the file:
MaximumSolrMemoryPercent=60
Restart the cluster
/usr/share/cb/cbcluster stop
/usr/share/cb/cbcluster start
After the SOLR cores complete optimization revert the cb.conf configuration back to its original values
SolrTimePartitioningOptimizeMaxSegments=10
#SolrTimePartitioningOptimizeMaxSegments=40
Additional Notes
This was a change made temporarilty until the SOLR cores were able to optimize successfully