Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

Cb Response: Services are failing to start and nothing is logged

Cb Response: Services are failing to start and nothing is logged

Environment

  • Cb Response: All Versions

Symptoms

  • A service is not starting, nothing is logged in that service's startup logs or any other logs. Supervisord does not list starting the service in supervisord.log
  • You may receive a connection refused error during startup

Starting cb-supervisord: [ OK ]

Starting cb-pgsql: [ OK ]

Starting cb-datagrid: [ OK ]

Starting cb-redis: [FAILED]

Starting cb-rabbitmq: [ OK ]

Error 111 connecting localhost:6379. Connection refused.

Traceback (most recent call last):

File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/coreservices/src/cb/utils/exceptions.py", line 70, in decorator

File "/usr/lib/python2.6/site-packages/cb/maintenance/cbstartup/main.py", line 165, in main

File "/usr/lib/python2.6/site-packages/cb/maintenance/cbstartup/main.py", line 111, in run

File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/coreservices/src/cb/maintenance/cbstartup/actions/init_banning.py", line 20, in execute

File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/coreservices/src/cb/core/banning/blacklist.py", line 28, in __init__

File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/coreservices/src/cb/core/banning/list_version.py", line 13, in __init__

File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/coreservices/src/cb/utils/cache.py", line 148, in __init__

File "/opt/jenkins/build/workspace/build-cbent-signed-docker-staging/code/coreservices/src/cb/utils/cache.py", line 141, in _register_script

File "/usr/lib/python2.6/site-packages/redis/client.py", line 1503, in register_script

return Script(self, script)

File "/usr/lib/python2.6/site-packages/redis/client.py", line 2035, in __init__

self.sha = registered_client.script_load(script)

File "/usr/lib/python2.6/site-packages/redis/client.py", line 1494, in script_load

return self.execute_command('SCRIPT', 'LOAD', script, **options)

File "/usr/lib/python2.6/site-packages/redis/client.py", line 397, in execute_command

connection.send_command(*args)

File "/usr/lib/python2.6/site-packages/redis/connection.py", line 306, in send_command

self.send_packed_command(self.pack_command(*args))

File "/usr/lib/python2.6/site-packages/redis/connection.py", line 288, in send_packed_command

self.connect()

File "/usr/lib/python2.6/site-packages/redis/connection.py", line 235, in connect

raise ConnectionError(self._error_message(e))

ConnectionError: Error 111 connecting localhost:6379. Connection refused.

Cause

This typically occurs when a server has been upgraded, restored or migrated and some piece of data is out of sync.

Resolution

  • Verify that the port is listening:

nc -v localhost port#

  • Verify you can ping the server listed, so for an error like 'Error 111 connecting localhost:6379', run:

ping localhost
  • Verify there are no other instances of that service running:

ps -ef | grep 'service name

  • Try starting the service as the cb user manually, so to start the redis server, run:

    • 6.2+

sudo -u cb /usr/bin/redis-server /etc/cb/redis.conf 
    • 6.1.2 and below
sudo -u cb /usr/sbin/redis-server /etc/cb/redis.conf 

     If that starts, check that the template for that service in /etc/cb/supervisord.templates/ matches the path run to start the service. If it does not, update the path.

Labels (1)
Was this article helpful? Yes No
No ratings
Article Information
Author:
Creation Date:
‎01-12-2018
Views:
2383
Contributors