node8: File "/usr/lib/python2.6/site-packages/haigha/channel.py", line 293, in add_synchronous_cb
node8: self.close_info['reply_text'] )
node8: ChannelClosed: ('channel %d is closed: %s : %s', 1, 404, "NOT_FOUND - home node 'rabbit@CB-SERVER-CLUSTER-HEAD-NODE' of durable queue 'alliance.data' in vhost '/' is down or inaccessible")
Note: References to node refer to master and minions
In any clustered environment (not just Carbon Black), RabbitMQ relies on the last node that was stopped to be started first. If the RabbitMQ service hangs during a routine cbcluster stop, attempts at starting the cluster may potentially fail due to RabbitMQ's design properties.
Stop the cluster from the master:
Kill all cb owned processes on all nodes (starting with the master):
killall -KILL -u cb
Start the cb-rabbitmq service on all nodes (starting with the master):
service cb-rabbitmq start
Note: RabbitMQ will not start until the last node to stop is started
Stop cb-rabbitmq service one by one on each node:
service cb-rabbitmq stop
Warning: do NOT continue to the next node until RabbitMQ fully stops.