EDR: How to Clear Disk Space Safely

[#1]

Environment

• EDR Server: All Versions

Objective

Disk has filled or is close to filling for /var. (95% or higher) 

Resolution

These files are safe to delete. 

1. Delete any HPROF files that are taking up space

   find /var/log/cb -type f -iname *.hprof* -delete -exec echo "Deleted" {} \;

2. Delete old logs that may not have been removed

   find /var/log/cb/ -mindepth 1 -type f -mtime +7 -delete -exec echo "Deleted" {} \;

3. Truncate any deleted files that still have an open file descriptor taking up space

   for i in `find /proc/*/fd -ls 2> /dev/null | awk '/deleted/ {print $11}';`; do truncate -s 0 $i; done

Please read and confirm before running

1. If you are using the event forwarder and outputting to a file, these are not automatically cleaned up. Delete any older than the days you want to keep, where mtime +7 will remove the files older than 7 days. If you wish to keep the files, move to a separate storage location instead. See: EDR: Disk Full Due to JSON Files for a suggested cron job. 

   find /var/cb/data -type f -mtime +7 -iname *event_bridge* -delete \; -exec echo "Deleted" {} \;

2. Modulestore: Remove physical binaries (.zip) that are stored locally older than your retention days, or manually set how long you want to keep. See ** in additional notes before running
   • Option 1 - Use retention day setting

     d=$(grep 'MaxEventStoreDays' /etc/cb/cb.conf | awk -F '=' '{print$2}';); `find /var/cb/data/modulestore/ -type f -mtime +$d -iname *.zip* -delete \; -exec echo "Deleted" {} \;`;

   • Option 2 - Manually set days, where mtime +60 will delete older than 60 days

     find /var/cb/data/modulestore/ -type f -mtime +60 -iname *.zip* -delete \; -exec echo "Deleted" {} \;

Event/Binary Data - WARNING

1. Determine what data directory is taking up the most space under /var/cb/data/. The first output is your total directory size

   declare -a arr=("/var/cb/data" "/var/cb/data/solr*/" "/var/cb/data/modulestore/" "/var/cb/data/solr*/cbmodules/" "/var/cb/data/solr*/cbevents" "/var/cb/data/solr*/cbevents/cbevents_*"); for d in ${arr[@]}; do du -h $d --max-depth=0;done

   • /var/cb/data/modulestore. This is your physical binaries. See the last section for clean up
   • /var/cb/data/solr*/cbmodules. This is your binary metadata (signature, publisher, product name, etc.) - Binary search page
   • /var/cb/data/solr*/cbevents/cbevents_*. This is your process events - Process Search page
2. If cbmodules is large, please review an enable the cronjob if not done already. By default, these are not removed. EDR: How to Enable Automated Cbmodule Purging
   • Do NOT delete anything within this path. This is a single Solr core holding all binary metadata
3. For cbevents, first determine if you have cold cores stored

   find /var/cb/data/solr/cbevents -mindepth 1 -maxdepth 1 -type d '!' -exec test -e "{}/core.properties" ';' -print
   

   • If these cores need to be kept, they should be moved to a new storage location outside of the /var/cb partition
   • If these cores can be deleted, run the following, replacing with the core directory name:

     rm -rf /var/cb/data/solr/cbevents/cbevents_<put_core_date_here>

4. If space is still needed, confirm the oldest active core and remove it. 

   find /var/cb/data/solr/cbevents/cbevents_* -type f -printf '%T+ %p\n' | sort | head -n 1 | awk -F "/" '{print$1,$7}'

   • This will delete the event data that was captured during the time of that core being hot (writer). Solr will need to be stopped first as the data is loaded into memory. Replacing with the core directory name. 

     /usr/share/cb/cbservice cb-solr stop
     rm -rf /var/cb/data/solr/cbevents/cbevents_<put_core_date_here>
     /usr/share/cb/cbservice cb-solr start

Restart Services

• If you have experienced a disk full situation, Solr and other services may not recover gracefully and ingestion can stop. Restarting services insures that the product is able to continue working as it should

  Standalone:
  /usr/share/cb/cbservice cb-enterprise restart
  
  Cluster via Primary:
  /usr/share/cb/cbcluster stop && /usr/share/cb/cbcluster start

Additional Notes

• By default the product will delete the oldest core once disk space hits 90% on the partition holding /var/cb/. It can go over slightly as the job runs every 15 min.
• During optimization (starts at midnight), it is normal for disk space to get close to 100% full. This is due to the copying of the index files into smaller segments to optimize the performance and space of the core. Once optimization is complete, the unoptimized core is removed and disk space will return to normal. Optimization can take a few hours depending on the size of the core and memory allocated to Solr. 

Article Reference:

** var/cb/data/modulestore holds your physical binaries. These are pulled when downloading a binary zip file from the console. If you are sharing binaries with Alliance, these can be safely removed. 

• These can be kept around for a few reasons
  • You are not sharing with Alliance. The files will stay locally for download purposes, however this means you are holding onto old binaries that may not be useful any longer and take up disk space
  • KeepAllModuleFiles is set to 1. This setting will not allow the code to delete the local binaries. A clean up job will be needed outside the product
  • A binary file was set to uploaded in Postgres, however they were not successfully removed. 
  • If you wish to keep these, it's suggested to create a separate partition that is mounted to /var/cb/data/cbmodules to not affect Event and Binary metadata available disk space. 
• Removing these does not affect the binary search page
• Removing these will generate a 404 if you try to download the file itself from the binary page. 

Related Content

EDR: How to Enable Automated Cbmodule Purging 
EDR: How to Remove a Binary Document From Solr (cbmodules)