Just Published! Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

Carbon Black: cbcluster script hangs at service startup

Carbon Black: cbcluster script hangs at service startup

Version

This solution applies to Carbon Black 5.0 Patch 2 (5.0.0.150428.1537) and v5.1 (150625.0500).

 

Issue

The /usr/share/cb/cbcluster script hangs during a cbcluster service start.

 

Symptoms

Attempting to start the cluster on the master may cause the script to hang. Some services on the Master will report as dead or not running, while the minions are unaffected.

 

Solution

As of this writing (08/06/2015), a permanent solution does not exist. There are currently three scenarios that cause the cluster start script to hang:

 

  • Scenario 1: CbTools process hanging after a cluster stop and the hidden file /etc/cb/.scrub_cbmodules exists:

 

1. On the Master server, ensure that there is not a hidden scrub_cbmodules file:

 

ls -la /etc/cb/.scrub_cbmodules

 

 

1a. If the above file exists, please remove it.

 

rm -f /etc/cb/.scrub_cbmodules

 

 

2. On the Master server, ensure there is not a CbTools process running:

 

ps -ef|grep CbTools

 

 

2a. If the above process is running, kill it manually.

 

kill -9 <PID> (where PID is left over CbTools process on the MASTER)

 

 

3. Start the cluster and ensure it is no longer hanging.

 

/usr/share/cb/cbcluster start

 

 

  • Scenario 2: RabbitMQ is out of sync on each node.

Please follow solution 871 to resolve the issue.

 

Scenario 3: Workaround using cb-enterprise start.

 

Steps

 

1. On the Master server, stop and start the cluster:

/usr/share/cb/cbcluster stop 

/usr/share/cb/cbcluster start

 

2. When the script hangs, verify all services are running on all minions:

service cb-enterprise status

 

3. Once they are, break out of the script on the Master server:

Ctrl + c

 

4. On the Master server, run these commands to remove PID and LOCK files, then start the cb-rabbitmq service:

rm -rf /var/run/cb/rabbitmq/pid

rm -rf /var/lock/subsys/cb-rabbitmq

service cb-rabbitmq start

 

5. Verify all services are running and then test CB UI functionality:

service cb-enterprise status

 

 

Important Note(s)

Please create a support case if you've encountered this issue for tracking and diagnosis.

Labels (1)
Was this article helpful? Yes No
No ratings
Article Information
Author:
Creation Date:
‎08-28-2015
Views:
647
Contributors