Environment
Symptoms
The cbdaemon is consuming elevated memory or CPU for an extended period of time without relief. This article addresses resource consumption that does not return to normal levels.
Cause
There have been different reasons for elevated memory and CPU over the years. All deployment scenarios cannot be anticipated, so we welcome diagnostic information when an issue occurs. Each issue is addressed after a case is opened and a bug report submitted.
Resolution
System resources can be elevated for several reasons, consider the following steps to resolve the issue:
1) Determine if there are other 3rd party Anti-Virus or security products running. Since security products use similar techniques to identify problems, they need to add rules to allow each other to operate. For Linux v6.2+, add rules to the 3rd party AV to allow Carbon Black to operate. The security product must allow CB to access:
/var/opt/carbonblack/response/*
/etc/init.d/cbdaemon
/usr/sbin/cbdaemon
/opt/carbonblack/response/*
/etc/sysconfig/modules/cbresponse.modules
2) Once exclusions are in place and resources remain elevated, take a sensordiag
while resources are elevated and open a support case. Cases help us determine the community impact. If the problem has already been reported, there may be an up-to-date workaround.
sudo /opt/carbonblack/response/bin/sensordiag.sh
3)
Temporary workaround for memory consumption while we work on the bug reported in step 2.
Implement one of the temporary workarounds below to limit the use of memory resources. Try on a test device prior to pushing to additional devices. Please keep us updated.
a. Chron a cbdaemon restart during off hours.
or
b. Limit cbdaemon memory usage.
% systemctl stop cbdaemon
% vi /etc/systemd/system/cbdaemon.service
Add the following line to the [Service] section
[Service]
MemoryMax=<limit-in-bytes>
% systemctl daemon-reload
% systemctl start cbdaemon
~
For example: MemoryMax=388M
4)
Temporary workaround for CPU consumption while we work on the bug reported in step 2.
Consider installing cpulimit and configuring.
% yum install epel-release
% yum install cpulimit
% cpulimit -p ${pidof cbdaemon} --limit 20 &
Additional Notes
- Linux OER states "The sensor typically uses 50-100 MB of virtual memory."
- Linux OER recommends the endpoint should have at least 1 GB of physical memory.
- Conflicts between security products is a common reason for escalated resources.
- Elevated Memory and CPU is expected to be elastic, meaning resource usage may spike but after time return to normal.
- Normally cbdaemon may need more resources while the endpoint is busy processing data or network connections.
Related Content
