EDR: No Free Disk Space Prevents Services From Starting

[#1]

Environment

• EDR (formerly CB Response) Server: 5.x and Higher

Symptoms

• df -h shows 100% of disk space used on one or more drives
• CbDiags will not generate 

Cause

Files within the server has caused the environment to run out of disk space.

Resolution

1. Free up space on the disk 
   1. Remove common large files from the server: https://community.carbonblack.com/t5/Knowledge-Base/EDR-Disk-On-Server-Full-Due-To-Large-Files/ta-p/...
   2. Check modulestore disk usage: 

du -h /var/cb/data/ --max-depth=1

3. If the modulestore directory is consuming excessive disk, purge old modules: https://community.carbonblack.com/t5/Knowledge-Base/How-To-Purge-Storefile-Modulestore-Binary-Data/t...
4. Determine if cbevents or cbmodules is taking up space: 

• For 5.x

du -h /var/cb/data/solr --max-depth=1

• For 6.x

du -h /var/cb/data/solr5 --max-depth=1

• For 7.x

du -h /var/cb/data/solr6 --max-depth=1

5. If cbevents data is filling the disk, purge the older events: 
   • For version 6.x and higher: https://community.carbonblack.com/t5/Knowledge-Base/CB-Response-How-To-Purge-CbEvent-Process-Data-in...
   • For version 5.x: https://community.carbonblack.com/t5/Knowledge-Base/CB-Response-How-To-Purge-CbEvent-Process-Data-in...
6. If cbmodules data is filling the disk: https://community.carbonblack.com/t5/Knowledge-Base/EDR-How-To-Purge-Modulestore-Binary-Files/ta-p/7...
7. Determine if a certain process log file is taking up space:

du -h /var/log/cb/ --max-depth=1

• Remove .tmp files older than 7 days

find <path_to_process_logs> -name *.tmp -mtime +7 -delete

• Remove .gz log files older than 7 days:

find <path_to_process_logs> -name *.gz -mtime +7 -delete

8. If the server is still out of space, determine which drive partition is out of space

df -h

9. From the affected partition, start at the root folder and work through the largest folder paths to clear data using the du command with the --max-depth=1 flag to list disk usage of each folder

du -h / --max-depth=1

2. Once space has been cleared, reset rabbitMQ: https://community.carbonblack.com/t5/Knowledge-Base/EDR-How-to-reset-Mnesia-for-RabbitMQ/ta-p/64848
3. Restart services - https://community.carbonblack.com/t5/Knowledge-Base/EDR-How-to-Restart-Server-Services/ta-p/41294

Related Content

EDR: Datastore logs consuming excessive disk space
Cb Response: Configurations to mitigate out of disk issues in 6.1 and Higher
Cb Response: Configurations to mitigate out of disk issues in 5.x