Version
This solution applies to Carbon Black 5.0 Patch 2 (5.0.0.150428.1537) and v5.1 (150625.0500).
Issue
The /usr/share/cb/cbcluster script hangs during a cbcluster service start.
Symptoms
Attempting to start the cluster on the master may cause the script to hang. Some services on the Master will report as dead or not running, while the minions are unaffected.
Solution
As of this writing (08/06/2015), a permanent solution does not exist. There are currently three scenarios that cause the cluster start script to hang:
- Scenario 1: CbTools process hanging after a cluster stop and the hidden file /etc/cb/.scrub_cbmodules exists:
1. On the Master server, ensure that there is not a hidden scrub_cbmodules file:
ls -la /etc/cb/.scrub_cbmodules
1a. If the above file exists, please remove it.
rm -f /etc/cb/.scrub_cbmodules
2. On the Master server, ensure there is not a CbTools process running:
ps -ef|grep CbTools
2a. If the above process is running, kill it manually.
kill -9 <PID> (where PID is left over CbTools process on the MASTER)
3. Start the cluster and ensure it is no longer hanging.
/usr/share/cb/cbcluster start
- Scenario 2: RabbitMQ is out of sync on each node.
Please follow solution 871 to resolve the issue.
Scenario 3: Workaround using cb-enterprise start.
Steps
1. On the Master server, stop and start the cluster:
/usr/share/cb/cbcluster stop
/usr/share/cb/cbcluster start
2. When the script hangs, verify all services are running on all minions:
service cb-enterprise status
3. Once they are, break out of the script on the Master server:
Ctrl + c
4. On the Master server, run these commands to remove PID and LOCK files, then start the cb-rabbitmq service:
rm -rf /var/run/cb/rabbitmq/pid
rm -rf /var/lock/subsys/cb-rabbitmq
service cb-rabbitmq start
5. Verify all services are running and then test CB UI functionality:
service cb-enterprise status
Important Note(s)
Please create a support case if you've encountered this issue for tracking and diagnosis.
