Unable to start cb-enterprise or cbcluster: cannot concatenate 'str' and 'tuple' objects

[#1]

Version

All versions of Carbon Black.

Issue

Unable to start cb-enterprise or cbcluster successfully.

Symptoms

When starting cb-enterprise or cbcluster, the following error is displayed during cb-rabbitmq start up:

[root@cbserver ~]# service cb-enterprise start

Starting cb-pgsql service:                                 [  OK  ]

Starting cb-redis:                                            [  OK  ]

Starting cb-rabbitmq:                                      [  OK  ]

Error: <type 'exceptions.TypeError'> cannot concatenate 'str' and 'tuple' objects

Traceback (most recent call last):

  File "/opt/jenkins/builds/workspace/build-cbent-release-5.1/code/coreservices/src/cb/utils/exceptions.py", line 57, in decorator

  File "/usr/lib/python2.6/site-packages/cb/maintenance/cbstartup/main.py", line 134, in main

  File "/usr/lib/python2.6/site-packages/cb/maintenance/cbstartup/main.py", line 84, in run

  File "/opt/jenkins/builds/workspace/build-cbent-release-5.1/code/coreservices/src/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 110, in execute

  File "/opt/jenkins/builds/workspace/build-cbent-release-5.1/code/coreservices/src/cb/maintenance/cbstartup/actions/init_rabbitmq.py", line 132, in _nothing_more_we_can_do_about_network_error

NetworkError: Error: <type 'exceptions.TypeError'> cannot concatenate 'str' and 'tuple' objects

Another symptom may be that the cb-redis service has not started successfully, as indicated by a service cb-enterprise status:

[root@CentOS66 ~]# service cb-enterprise status

cb-pgsql (pid  52145) is running...

cb-redis dead but subsys locked

Cause

An updated version of the packages:

selinux-policy

selinux-policy-targeted

python-requests

python-urlib3

dependencies is preventing cb-rabbitmq and cb-redis from successfully starting.

Solution

We are working to resolve this permanently. In the meantime, the workaround is to downgrade the above package versions:

1. Stop the services:
   
   If a standalone installation:
   

   service cb-enterprise stop

   
   If a clustered installation:
   

   /usr/share/cb/cbcluster stop

   
   
   
2. Download the relevant packages (for CentOS 6.6 official mirrors):
   

http://mirror.centos.org/centos-6/6/extras/x86_64/Packages/python-requests-1.1.0-4.el6.centos.noarch...

http://mirror.centos.org/centos-6/6/extras/x86_64/Packages/python-urllib3-1.5-7.el6.centos.noarch.rp...

http://vault.centos.org/6.5/cr/i386/Packages/selinux-policy-3.7.19-260.el6.noarch.rpm

http://vault.centos.org/6.5/cr/i386/Packages/selinux-policy-targeted-3.7.19-260.el6.noarch.rpm

     Note: One can use wget or curl to directly download the file to their Carbon Black Server or use a transfer protocol (i.e. scp) to copy from another host:

curl -O http://mirror.centos.org/centos-6/6/extras/x86_64/Packages/python-requests-1.1.0-4.el6.centos.noarch...

curl -O http://mirror.centos.org/centos-6/6/extras/x86_64/Packages/python-urllib3-1.5-7.el6.centos.noarch.rp...

curl -O http://vault.centos.org/6.5/cr/i386/Packages/selinux-policy-3.7.19-260.el6.noarch.rpm;

curl -O http://vault.centos.org/6.5/cr/i386/Packages/selinux-policy-targeted-3.7.19-260.el6.noarch.rpm;

     Note: If the installation is a clustered environment, all minions will also need these downgraded versions.

• On the Carbon Black Server (and all minions, if running a clustered environment), use the rpm utility to downgrade the version:

rpm -Uvh --force selinux-policy-3.7.19-260.el6.noarch.rpm selinux-policy-targeted-3.7.19-260.el6.noarch.rpm python-requests-1.1.0-4.el6.centos.noarch.rpm python-urllib3-1.5-7.el6.centos.noarch.rpm

• Start the Services:

If a standalone installation:

service cb-enterprise start

If a clustered installation:

/usr/share/cb/cbcluster start

• Verify the services are all started, including all minions:

service cb-enterprise status

/usr/share/cb/cbcluster status