5.x no disk space available on Cb Response server due to cbevent purge settings in past

[#1]

Version

Cb Response 4.x, 5.x

Note: For 6.x follow this guide instead: No Free Disk Space Prevents Cb Response Services From Starting

Issue

The cb-enterprise service will not start with no disk space available on Cb Response server due to purge settings in past

Symptoms

df –h shows that the /var/cb is 100% used and only a small disk space is left and not enough to allow a service to start.

Cause

There’s a defect on version of 4.2.3 to v5.1.0 where it doesn't apply default values for purging. This is addressed on a hotfix but requires the CB server to be on at least v5.0.0 patch 3. CB v5.0.0 patch 4 and 5.1.0 patch 1 will both include the hotfix.

Solution

1. Verify Cb Events is taking up disk space
   1. Check current disk space:
      

      df -h

   2. Check folder size in /var/cb/data since most of the disk usage is here:
      

      du -h /var/cb/data/ --max-depth=1

   3. Check folder size in /var/cb/data/solr since most of the disk usage is here under the data folder:
      

      du -h /var/cb/data/solr --max-depth=1

2. Check if environment settings are incorrect

   1. Verify that OER disk requirements are being followed:
      Carbon Black Response v5.2 - Operating Environment Requirements

   1. Check expected number of days to keep, so we know what to put in the curl command later:
      

      grep MaxEventStore /etc/cb/cb.conf

   2. Verify /etc/cb/cb.conf has values that are consistant with OER/Defaults:
      

      MaxEventStoreSizeInDocs=60
      MaxEventStoreSizeInPercent=70
      MaxEventStoreDays=30
      # MaxEventStoreSizeInMB=100000

3. Delete old storefiles to make enough room to start services: How To Purge Storefile (Modulestore Binary) Data
4. Purge event data to retention settings: How To Purge CbEvent (Process) Data in Cb Response
5. Correct EventPurgeEarliestTime if the date is in the past:
   

   Note: this issue could occur between v4.2.3 – v.5.1.0, but incorrect settings could still be present after upgrading

   psql cb -p 5002 -c 'select * from cb_settings;'

   1. Run this to remove the future date if present on EventPurgeEarliestTime
      

      psql cb -p 5002 -c "delete from cb_settings where key like 'EventPurge%';"

   2. Verify that the events_purge job is running correctly
      

      /usr/bin/python -m cb.maintenance.job_runner -s events_purge >> /var/log/cb/job-runner/startup.out 2>&1

   3. Run this to verify that the date is now correct (past date):
      

      psql cb -p 5002 -c 'select * from cb_settings;'

6. Start the main CB service and verify the UI is accessible:
   

   service cb-enterprise start