Version
All versions of Carbon Black.
Issue
Unable to start cb-enterprise or cbcluster successfully.
Symptoms
When starting cb-enterprise or cbcluster, the following error is displayed during cb-rabbitmq start up:
[root@cbserver ~]# service cb-enterprise start
Starting cb-pgsql service: [ OK ]
Starting cb-redis: [ OK ]
Starting cb-rabbitmq: [ OK ]
Error: <type 'exceptions.TypeError'> cannot concatenate 'str' and 'tuple' objects
Traceback (most recent call last):
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1/code/coreservices/src/cb/utils/exceptions.py", line 57, in decorator
File "/usr/lib/python2.6/site-packages/cb/maintenance/cbstartup/main.py", line 134, in main
File "/usr/lib/python2.6/site-packages/cb/maintenance/cbstartup/main.py", line 84, in run
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1/code/coreservices/src/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 110, in execute
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1/code/coreservices/src/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 132, in _nothing_more_we_can_do_about_network_error
NetworkError: Error: <type 'exceptions.TypeError'> cannot concatenate 'str' and 'tuple' objects
Another symptom may be that the cb-redis service has not started successfully, as indicated by a service cb-enterprise status:
[root@CentOS66 ~]# service cb-enterprise status
cb-pgsql (pid 52145) is running...
cb-redis dead but subsys locked
Cause
An updated version of the packages:
selinux-policy
selinux-policy-targeted
python-requests
python-urlib3
dependencies is preventing cb-rabbitmq and cb-redis from successfully starting.
Solution
We are working to resolve this permanently. In the meantime, the workaround is to downgrade the above package versions:
- Stop the services:
If a standalone installation:
service cb-enterprise stop
If a clustered installation:
/usr/share/cb/cbcluster stop
- Download the relevant packages (for CentOS 6.6 official mirrors):
http://mirror.centos.org/centos-6/6/extras/x86_64/Packages/python-requests-1.1.0-4.el6.centos.noarch...
http://mirror.centos.org/centos-6/6/extras/x86_64/Packages/python-urllib3-1.5-7.el6.centos.noarch.rp...
http://vault.centos.org/6.5/cr/i386/Packages/selinux-policy-3.7.19-260.el6.noarch.rpm
http://vault.centos.org/6.5/cr/i386/Packages/selinux-policy-targeted-3.7.19-260.el6.noarch.rpm
Note: One can use wget or curl to directly download the file to their Carbon Black Server or use a transfer protocol (i.e. scp) to copy from another host:
curl -O http://mirror.centos.org/centos-6/6/extras/x86_64/Packages/python-requests-1.1.0-4.el6.centos.noarch...
curl -O http://mirror.centos.org/centos-6/6/extras/x86_64/Packages/python-urllib3-1.5-7.el6.centos.noarch.rp...
curl -O http://vault.centos.org/6.5/cr/i386/Packages/selinux-policy-3.7.19-260.el6.noarch.rpm;
curl -O http://vault.centos.org/6.5/cr/i386/Packages/selinux-policy-targeted-3.7.19-260.el6.noarch.rpm;
Note: If the installation is a clustered environment, all minions will also need these downgraded versions.
- On the Carbon Black Server (and all minions, if running a clustered environment), use the rpm utility to downgrade the version:
rpm -Uvh --force selinux-policy-3.7.19-260.el6.noarch.rpm selinux-policy-targeted-3.7.19-260.el6.noarch.rpm python-requests-1.1.0-4.el6.centos.noarch.rpm python-urllib3-1.5-7.el6.centos.noarch.rpm
If a standalone installation:
service cb-enterprise start
If a clustered installation:
/usr/share/cb/cbcluster start
- Verify the services are all started, including all minions:
service cb-enterprise status
/usr/share/cb/cbcluster status
