Just Published! Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

EDR: No Free Disk Space Prevents Services From Starting

EDR: No Free Disk Space Prevents Services From Starting

Environment

  • EDR (formerly CB Response) Server: 5.x and Higher

Symptoms

  • df -h shows 100% of disk space used on one or more drives
  • CbDiags will not generate 

Cause

Files within the server has caused the environment to run out of disk space.

Resolution

  1. Free up space on the disk 
    1. Remove common large files from the server: https://community.carbonblack.com/t5/Knowledge-Base/EDR-Disk-On-Server-Full-Due-To-Large-Files/ta-p/...
    2. Check modulestore disk usage: 
du -h /var/cb/data/ --max-depth=1
  1. If the modulestore directory is consuming excessive disk, purge old modules: https://community.carbonblack.com/t5/Knowledge-Base/How-To-Purge-Storefile-Modulestore-Binary-Data/t...
  2. Determine if cbevents or cbmodules is taking up space: 
  • For 5.x
du -h /var/cb/data/solr --max-depth=1
  • For 6.x
du -h /var/cb/data/solr5 --max-depth=1
  • For 7.x
du -h /var/cb/data/solr6 --max-depth=1
  1. If cbevents data is filling the disk, purge the older events: 
  2. If cbmodules data is filling the disk: https://community.carbonblack.com/t5/Knowledge-Base/EDR-How-To-Purge-Modulestore-Binary-Files/ta-p/7...
  3. Determine if a certain process log file is taking up space:
du -h /var/log/cb/ --max-depth=1
  • Remove .tmp files older than 7 days
find <path_to_process_logs> -name *.tmp -mtime +7 -delete
  • Remove .gz log files older than 7 days:
find <path_to_process_logs> -name *.gz -mtime +7 -delete
  1. If the server is still out of space, determine which drive partition is out of space
df -h
  1. From the affected partition, start at the root folder and work through the largest folder paths to clear data using the du command with the --max-depth=1 flag to list disk usage of each folder
du -h / --max-depth=1
  1. Once space has been cleared, reset rabbitMQ: https://community.carbonblack.com/t5/Knowledge-Base/EDR-How-to-reset-Mnesia-for-RabbitMQ/ta-p/64848
  2. Restart services - https://community.carbonblack.com/t5/Knowledge-Base/EDR-How-to-Restart-Server-Services/ta-p/41294

Related Content


Labels (1)
Tags (2)
Was this article helpful? Yes No
0% helpful (0/1)
Article Information
Author:
Creation Date:
‎01-28-2019
Views:
3526
Contributors