EDR: How to Change Server IP Addresses for a Cluster (Multi-Node)

[#1]

Environment

• EDR Server: 5.X, 6.X, 7.X Cluster Implementations

Objective

To change the server IP or FQDN on an EDR Server.

Resolution

1. Update Sensors from the UI:

Please verify the chosen settings Server URL and Port(https://<url>:<port>). There is only one shot to update this setting correctly. A sensor will checkin with to the current server address and update that information locally (in the registry and hosts file for Windows sensors). The next time the sensor tries to checkin, the new server url and port will be added to the registry and it will no longer attempt to connect to the old address. This means that if this information is updated incorrectly, each sensor within the edited group will use the incorrect Server URL and Port and will need to be changed manually via the settings on the sensor (or by performing Step 3 under 'Update Remaining Sensors'). Changing the group setting in the UI back will only work for sensors which have not checked in yet, all others will not get that setting reverted back automatically.

If any questions or concerns occur during this process or with these instructions, please reach out to VMWare Carbon Black Support for assistance.

• Open the WebUI, then navigate to Administration > Sensors and select Edit Settings.
• Change the Server Address to the new IP while keeping the same port number.
  • Support's recommendation is to keep the default sensor communication port, 443. There are additional configuration changes that need to be made in /etc/cb/cb.conf and /etc/cb/nginx/conf.d/ in order for the sensors to communicate on a custom port.
• Repeat the step above for each sensor group if they exist.

2. Update Master Node from the Command Line:

• Stop Cluster Services : 

  /usr/share/cb/cbcluster stop

• Change the server IP using normal OS commands for configuring the network interface

• Update the IP address of the master node in Postgres: (NOTE: This should also be repeated for each Minion in the Cluster IE: Minion 1 is node_id=1)

In 7.5.0 and later product versions:

/usr/share/cb/cbservice cb-pgsql start
psql -d cb -p 5002 -c "UPDATE cluster_node_sensor_addresses SET address='<NEW IP>' WHERE node_id=0;"
/usr/share/cb/cbservice cb-pgsql stop

• In product versions 7.4.0 to 7.5.0:

/usr/share/cb/cbservice cb-pgsql start
psql -d cb -p 5002 -c "UPDATE cluster_node_sensor_addresses SET address='<NEW IP>' WHERE id=0;"
/usr/share/cb/cbservice cb-pgsql stop

• In product versions prior to 7.4.0:

service cb-pgsql start
psql -d cb -p 5002 -c "UPDATE cluster_node_sensor_addresses SET address='<NEW IP>' WHERE id=0;"
service cb-pgsql stop

3.To update the IP addresses of the minion nodes if applicable (IE: Minion 1 is Node 1): If not done via pgsql above Node URL's can be modified via the console in Settings - Server Nodes

• Update /etc/cb/cluster.conf to match new IPs for all relevant nodes

• Update /etc/sysconfig/iptables to accept traffic from new minion IPs if applicable

• Update /etc/hosts to redirect to match new IPs for all relevant nodes

4. Run these commands after all online sensors have checked in:

• Update Minions from the Command Line:
  1. Change the server IP using normal OS commands for configuring the network interface if applicable
  2. Update /etc/cb/cb.conf to match new master IP
     1. Update psql DatabaseURL value
     2. Update Redis RedisHost value
  3. Update /etc/cb/cluster.conf to match new IPs for new master IP (and minions if applicable)
  4. Update /etc/sysconfig/iptables to accept traffic from new master IP (and minions if applicable)
     • If CB managed firewall run the following command on master and minions

       /usr/share/cb/cbcheck firewall -a

  5. Update /etc/hosts to match new IPs for new master IP (and minions if applicable)
• Start services on the Master Node from the Command Line:
  1. Start EDR Cluster :

/usr/share/cb/cbcluster start

2. Verify that ~/.ssh/known_hosts has been updated
3. Update Remaining Sensors
   • For any clients that did not check-in before the server address is changed, please modify the SensorBackendServer setting on the individual sensor.
     • For Windows this is located at HKLM\SOFTWARE\CarbonBlack\config\SensorBackendServer
     • For MacOs and Linux, this value is stored in /var/lib/cb/sensorsettings.ini

If any questions are encountered during this process, please contact VMWare Carbon Black Support for assistance.

Additional Notes

• There will be a delay after changing this setting as all endpoints would need to check in first to pull down the new server IP / FQDN.
• Any offline endpoints that don't check in will either need to have sensor reinstalled or registry edited to point to the new server. 
• If using custom certificates, the hosts files on sensors will be updated once registry changes have been made (either via sync with new Sensor Group settings or manually changed on the sensor).  If this does not take effect, a restart of sensor services should generate a new hosts file and allow connection to the server's new IP address.

Related Content

EDR: Change Server IP or Domain Name Address (Single Node)