Environment
- EDR Linux Sensor: 7.1.0
- Linux: All Supported Versions
Symptoms
- Sensor occasionally creates large core dumps
- Message reporting cbdaemon is generating core dump is reported in /var/log/messages file:
systemd[1]: Started Process Core Dump (PID 1234/UID 0).
systemd[1]: cbdaemon.service: Main process exited, code=killed, status=11/SEGV
cbdaemon.service: Killing process 12234 (ECStateEngine) with signal SIGKILL.
systemd[1]: cbdaemon.service: Killing process 12334 (event_collector) with signal SIGKILL.
systemd-coredump[607]: Failed to compress (unnamed temporary file): No space left on device
systemd-coredump[607]: Process 2225 (cbdaemon) of user 0 dumped core
Cause
Issue with how the sensor handles file descriptors - CB-37984
Resolution
- This issue will be fixed in sensor version 7.1.1
- As a workaround, core dumps on the sensor can be disabled.
- Stop the cbdaemon service by executing
- Create a wrapper script for cbdaemon for systemd to execute which will disable coredumps for cbdaemon.
Put the following commands into /usr/sbin/cbdaemon.sh and make the file executable
#!/bin/bash
############
echo "disabling coredumps"
ulimit -c 0
echo "starting cbdaemon"
exec /usr/sbin/cbdaemon
- Edit the cbdaemon service unit file /etc/systemd/system/cbdaemon.service to call this wrapper script rather than cbdaemon directly.
- Change the line
ExecStart=/usr/sbin/cbdaemon
To
ExecStart=/usr/sbin/cbdaemon.sh - Reload the service configuration by executing
Additional Notes
- An inordinate amount of messages similar to the following may be reported in /var/log/messages file:
kernel: WARNING: CPU: 35 PID: 2258 at /src/workspace/kernel_event_collector_module/build/gcc73-relwithdebinfo.1/3.10.0-957/process-tracking.c:336 ec_process_tracking_update_process+0x27b/0x470 [cbsensor_2_3_859574]
kernel: WARNING: CPU: 3 PID: 381 at /src/workspace/kernel_event_collector_module/build/gcc73-relwithdebinfo.1/3.10.0-957/process-tracking.c:474 ec_process_tracking_report_exit+0xd3/0xe0
kernel: WARNING: CPU: 38 PID: 11610 at /src/workspace/kernel_event_collector_module/build/gcc73-relwithdebinfo.1/3.10.0-957/process-tracking.c:474 ec_process_tracking_report_exit+0xd3/0xe0 [cbsensor_2_3_859574]
kernel: WARNING: CPU: 34 PID: 14941 at /src/workspace/kernel_event_collector_module/build/gcc73-relwithdebinfo.1/3.10.0-957/process-tracking.c:336 ec_process_tracking_update_process+0x27b/0x470 [cbsensor_2_3_859574]