Unexpected log entry or corrupt log causing Solr not to start

[#1]

Version

All versions of Cb Response

Issue

Solr hangs while starting due to a corrupt tlog file

Cause

Usually caused by Solr not being properly shut down while indexing.

Symptoms

A similar error message is found to the following example in /var/log/cb/solr/debug.log

2016-01-01 01:01:01 - [WARN] - from org.apache.solr.update.UpdateLog in coreLoadExecutor-3-thread-1
Unexpected log entry or corrupt log. Entry=1523java.lang.ClassCastException: null

Solution

1. Create a corrupt tlog directory and move all tlogs to this directory
   

   mkdir /var/cb/data/solr/cbevents/0/data/tlogcorrupt
   mv /var/cb/data/solr/cbevents/0/data/tlog/tlog.* /var/cb/data/solr/cbevents/0/data/tlogcorrupt/

   1. Repeat the process for any remaining shards
2. Stop any remaining services
   1. Standalone
      

      service cb-enterprise stop

   2. Cluster
      

      /usr/share/cb/cbcluster stop

3. Remove all cb user owned processes
   

   killall -KILL -u cb

4. Start the Services
   1. Standalone
      

      service cb-enterprise start

   2. Cluster
      

      /usr/share/cb/cbcluster start

5. Remove the tlogcorrupt folder once services have started
   

   rm -rf /var/cb/data/solr/cbevents/0/data/tlogcorrupt

   1. Repeat the process for any remaining shards

Important Note(s)

The tlog are raw documents created for recovery purposes. Since these are now corrupt, they are no longer useful. If they are left behind, solr will take up more disk space than what is specified in /etc/cb/cb.conf.