Just Published! Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

Cb Response server fails to start. Starting process hangs on 'Starting cb-solr' while 'Waiting for cb-solr to build the terms dictionary'.

Cb Response server fails to start. Starting process hangs on 'Starting cb-solr' while 'Waiting for cb-solr to build the terms dictionary'.

Version
Cb Response v6.x.


Issue

Cb Response server fails to start (standalone or cluster environment). Starting process hangs on 'Starting cb-solr' while 'Waiting for cb-solr to build the terms dictionary'.

Starting cb-solr:                                          [  OK  ]

Waiting for cb-solr to build the terms dictionary.

Depending on index size this may take a while.............................

    <May return to command prompt without error>

Symptoms

Following message is found in '/var/log/cb/solr/debug.log' file:

[ERROR] - from org.apache.solr.core.SolrCore in main

null:org.apache.solr.common.SolrException: Found multiple cores with the name [writer], with instancedirs [cbevents_<timestamp1>] and [cbevents_<timestamp2>]


Cause

Known issue related to a failure in the Solr event core rollover, discussed in the following document: Cb Response startup fails


Per the referenced document, 'This is due to a rare issue with Solr core rollover identified as CB-13790. This defect is scheduled to be addressed in the next product release (currently targeted for: 6.1.2).  For this issue specifically there are two 'writer' cores. Normally there is only one 'writer' core, and it should be the core with the most recent timestamp.

Solution

Running the command from Step 1a in the referenced document will identify the 'writer' core, in this case there will be two:

$ find /var/cb/data/solr5/cbevents -type f -exec grep -H 'name=writer' {} \;

./cbevents_2017_09_04_2003/core.properties:name=writer

./cbevents_2017_07_28_1615/core.properties:name=writer

Perform following steps to remove the oldest of the two 'writer' cores:

1) From the find command above, unload the oldest 'writer' core:

(Warning: Alerts generated from process data in this core will lead to 404 pages)

curl "http://localhost:8080/solr/admin/cores?action=UNLOAD&core=cbevents_2017_07_28_1615"

2) Verify the core is unloaded:

curl "http://localhost:8080/solr/admin/cores?action=STATUS&indexInfo=true&indent=true&wt=json" | grep cbevents_2017_07_28_1615

3) Remove the directory when core is no longer loaded

rm -rf /var/cb/data/solr5/cbevents/cbevents_2017_07_28_1615

4) Restart cb-enterprise services (or cbcluster restart)

Related Content

Cb Response startup fails.

Labels (2)
Was this article helpful? Yes No
No ratings
Article Information
Author:
Creation Date:
‎10-30-2017
Views:
1507
Contributors