Just Published! Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

Cb Response startup fails to proceed past "Waiting for cb-solr to build the terms dictionary"

Cb Response startup fails to proceed past "Waiting for cb-solr to build the terms dictionary"

Version
Cb Response 6.1.0, 6.1.1

Topic
Cb Response fails to start (standalone or cluster environment).  Start proceeds until indexing and then fails to proceed past and eventually returns to command prompt:

Starting cb-solr:                                          [  OK  ]

   Waiting for cb-solr to build the terms dictionary.

   Depending on index size this may take a while.............................

<returns to command prompt without error>

Review of the /var/log/cb/solr/debug.log (on standalone server or minion(s) which failed) shows the following error:

[ERROR] - from org.apache.solr.core.SolrCore in main

null:org.apache.solr.common.SolrException: Found multiple cores with the name [cbevents_<timestamp1>], with instancedirs [/var/cb/data/solr5/cbevents/cbevents_<timestamp2>] and [/var/cb/data/solr5/cbevents/cbevents_<timestamp3>]

    at org.apache.solr.core.CoreContainer.checkForDuplicateCoreNames(CoreContainer.java:537)

This is due to a rare issue with Solr core rollover identified as CB-13790.  This defect is addressed in the 6.1.2 release.   This issue can be addressed using the steps below.

Steps

  1. Locate the currently active core (the writer core)
    a.  Execute:  find /var/cb/data/solr5/cbevents -type f -exec grep -H 'name=writer' {} \;

          b.  example output:  /var/cb/data/solr5/cbevents/cbevents_2017_07_13_1456/core.properties:name=writer

               this tells us that "cbevents_2017_07_13_1456" is the current valid core and newer cores are invalid.  If the writer core is the most recent core, do not proceed and log a support ticket.

     2.  Check all later cores to confirm that they are invalid/empty

          a.  Execute:   du -h --max-depth=1 /var/cb/data/solr5/cbevents

          b.  Any directories which start with "cbevents_", have a timestamp after the writer core identified in Step 1, and show only 24k of space used should be deleted

          c.  Delete empty/invalid cores using:  rm -rf /var/cb/data/solr5/cbevents_<timestamp>.

               Be certain you are only deleting cores that occur AFTER the writer core and ones that reflect only 24k of space used, deleting a core with data will result in data loss.

     3.  Restart the standalone server/cluster

          a.  For standalone server:  service cb-enterprise restart

          b.  For cluster (from master): 

                    /usr/share/cb/cbcluster stop  

                    /usr/share/cb/cbcluster start

Important Note(s)

Do not proceed to deleting cores (Step 2c) unless you are certain this Solution is appropriate and the cores are empty.  If you are not certain, please log a support ticket for assistance, reference this KB article to indicate you are encountering CB-13790.

Related Content

Cb Response server fails to start.

Labels (1)
Was this article helpful? Yes No
100% helpful (1/1)
Article Information
Author:
Creation Date:
‎07-14-2017
Views:
1554
Contributors