Just Published! Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

EDR: Corrupted CBmodules on a Minion Stop the Cluster Startup

EDR: Corrupted CBmodules on a Minion Stop the Cluster Startup

Environment

  • EDR Servers: 7.6 and higher

Symptoms

  • Cluster does not complete a start-up due to one minion's service failure between cb-solr and cb-coreservices.
  • The Solr logs may contain one of the following entries:
"IndexNotFoundException: no segments* file found LockValidatingDirectoryWrapper"

"SolrCore 'cbmodules' is not available due to init failure: Error opening new searcher"

 

Cause

The cbmodules indexing has become corrupt.

Resolution

Remove the contents of /var/cb/data/solr/cbmodules/ for that one minion.  The Primary server maintains the official copy of the cbmodules indexing and re-populates the minion's cbmodule indexing if they were removed.  This synchronizes the cbmodules indexing across the cluster and the cluster should start.

Additional Notes

  • Just remove the contents of the cbmodules directory, not the directory itself.
  • Consider making a backup of the directory before deleting it just in case it is needed.  It can be removed once the cluster starts and the Primary server properly updates the minion cbmodules directory. 
tar -cvf cbmodulesmeta.tar /var/cb/data/solr/modules/
  • The /var/cb/data/solr/cbmodules directory contains the metadata and indexing for the binary files.  The actual binary files are stored in another directory /var/cb/data/cbmodules/ and are unique on each server;  They should not be removed for this fix.
  • The replication.properties file is normal in /var/cb/data/solr/cbmodules/data/ directory.

Related Content


Labels (1)
Tags (2)
Was this article helpful? Yes No
No ratings
Article Information
Author:
Creation Date:
‎07-01-2022
Views:
64
Contributors