Environment
- EDR Server: 6.x and Higher
- CentOS: 7.8
Symptoms
- CB Response Services does not start.
- /var/log/messages shows
Jul 20 13:29:34 dscb001 cb-enterprise[1114]: raise NetworkError("Timeout while trying to connect to RabbitMQ")
Jul 20 13:29:34 dscb001 cb-enterprise[1114]: pyrabbit2.http.NetworkError: Timeout while trying to connect to RabbitMQ
Jul 20 13:29:34 dscb001 systemd[1]: cb-enterprise.service: control process exited, code=exited status=1
- /var/log/cb/solr/debug.log shows:
2020-07-21 14:33:16,491 - [ERROR] - from com.carbonblack.cbfs.solr.CbEventPublisher in qtp1529306539-20
RabbitMQConnection failed, event notifications will be disabled
java.util.concurrent.TimeoutException: null
2020-07-21 14:33:21,593 - [WARN] - from com.rabbitmq.client.impl.ForgivingExceptionHandler in AMQP Connection 127.0.0.1:5004
An unexpected connection driver error occured (Exception message: Socket closed)
Cause
- Unreachable DNS configured in /etc/resolv.conf
- hostname not correctly configured in /etc/hosts
- No internet access | Air-gaped system
Resolution
- Add correct hostname to /etc/hosts file
- To check hostname of the node, run
Example:
[root@myhostname] $ hostname
myhostname
- Check the hostname is configured in the /etc/hosts file. If not, add the hostname to the file.
Example:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1 myhostname
# {{BEGIN - CB Server}}
# The block within these BEGIN/END tags is managed by Carbon Black Enterprise
# Server configuration tools. DO NOT EDIT within BEGIN/END comment tags as the
# content of this section may get automatically regenerated.
127.0.0.1 CB-SERVER-CLUSTER-HEAD-NODE
# {{END - CB Server}}
- Check how many DNS are configured and are they reachable.
Example:
[root@localhost cbevents_2020_07_20_0555]# nslookup
> server
Default server: 10.40.40.1
Address: 110.40.40.1#53
Default server: 8.8.8.8
Address: 8.8.8.8#53
cat /etc/resolv.conf
# Generated by NetworkManager
search localdomain
nameserver 10.40.40.1
nameserver 8.8.8.8
Additional Notes
- Removing 8.8.8.8 (not reachable) on an air-gapped system, resolved rabbitmq stuck issue and services resumed.
- Checking source of unreachable DNS will need to be traced, or problem may reoccur after network manager restart.
- Check DHCP if configured.
Related Content
