RabbitMQ Failing to Start with Unreadable Error "booetr"

[#1]

Version
6.X

Issue

Services will not start. RabbitMQ is failing due to permissions issue

Symptoms

/var/log/cb/solr/debug.log

The below error displays broken messages for rabbitmq

node0: Process execution failure(1):

eode0: {(n"oi neirtr otre rlmoigngaetri npgr eisne ndto)_ booetr"r,o{r:b a<d0a.r2g.,0[>{

node0: rl_prim_loader,check_file_result,3,[]},{init,get_boot,1,[]},{init,get_boot,2,[]},{init,do_boot,3,[]}]}}

node0: init terminating in do_boot ()

node0: Traceback (most recent call last):

node0:   File "/opt/jenkins/build/workspace/build-cbent-signed-staging/code/coreservices/src/cb/utils/exceptions.py", line 62, in decorator

node0:   File "/usr/lib/python2.6/site-packages/cb/maintenance/cbstartup/main.py", line 152, in main

node0:   File "/usr/lib/python2.6/site-packages/cb/maintenance/cbstartup/main.py", line 98, in run

node0:   File "/opt/jenkins/build/workspace/build-cbent-signed-staging/code/coreservices/src/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 74, in execute

node0:   File "/opt/jenkins/build/workspace/build-cbent-signed-staging/code/coreservices/src/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 79, in _install_builtin_plugins

node0:   File "/opt/jenkins/build/workspace/build-cbent-signed-staging/code/coreservices/src/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 99, in _enable_plugin

node0:   File "/opt/jenkins/build/workspace/build-cbent-signed-staging/code/coreservices/src/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 116, in _run_process

node0: Exception: Process execution failure(1):

eode0: {(n"oi neirtr otre rlmoigngaetri npgr eisne ndto)_ booetr"r,o{r:b a<d0a.r2g.,0[>{

node0: rl_prim_loader,check_file_result,3,[]},{init,get_boot,1,[]},{init,get_boot,2,[]},{init,do_boot,3,[]}]}}

node0: init terminating in do_boot ()

node0:

node0: Exec failure (1)

node0:

Cause
Permissions have changed, possibly after a security update

Solution

1) Shut down the services

Single instance:
services cb-enterprise stop
Clustered:
/usr/share/cb/cbcluster stop

2) Change the permissions:

     chmod 775 /var/lib/rabbitmq

3) Kill any hanging processes from the failed start (run on minions as well for clustered)

     killall -KILL -u cb

4) Restart the services

Single instance:
services cb-enterprise start
Clustered:
/usr/share/cb/cbcluster start

Related Content(s)

You may continue to see a similar error for enabled plugins. Error: {cannot_write_enabled_plugins_file,"/etc/cb/rabbitmq/enabled_plugins",eacces}

Rabbitmq Process Won't Start With "enabled_plugins" Error