Environment
- EDR ( Formerly known as CB Response) Server: Version 6.2.4 and Higher
Symptoms
'sensor_comm_failures' messages are filling the /var/log/messages file. Example message:
sensor_comm_failures: {"sensor_timestamp": "2019-03-03T10:27:18.199000+00:00", "timestamp": "2019-03-03T10:27:39.402712+00:00", "sensor_id": 50016, "server_url": "https://12.345.67.89:443/data/storefile/check/50016", "failure_code": -2147014836}Cause
Logging of sensor communications failures has changed with the version 6.2.4 release. Prior to version 6.2.4 sensor communications failures were recorded in Postgres, with version 6.2.4 we are now writing them out to a log file. We make use of the rsyslog service running on the Linux system to handle the logging to this file.
Resolution
Additional Notes
- The additional logging with 'sensor_comm_failures' may cause delays with the 'sensorservices' process that handles sensor checkins, as it waits on rsyslog to accept additional messages before moving on.
- Adding the CoreServicesRecordSensorDiagnostics parameter does have the side effect that sensor diagnostics sent during sensor checkin are no longer being recorded, these include sensor comm failures and other sensor health metrics. It does not affect the core use of the product, but does limit visibility into any sensor related issues. Many of these metrics are however reported on the sensor details page. This change will persist through a restart as it is in the cb.conf. To enable the diagnostics data, simply remove that entry from cb.conf and restart services.
Related Content
