Version
Cb Response 4.x, 5.x
Note: For 6.x follow this guide instead: No Free Disk Space Prevents Cb Response Services From Starting
Issue
The cb-enterprise service will not start with no disk space available on Cb Response server due to purge settings in past
Symptoms
df –h shows that the /var/cb is 100% used and only a small disk space is left and not enough to allow a service to start.
Cause
There’s a defect on version of 4.2.3 to v5.1.0 where it doesn't apply default values for purging. This is addressed on a hotfix but requires the CB server to be on at least v5.0.0 patch 3. CB v5.0.0 patch 4 and 5.1.0 patch 1 will both include the hotfix.
Solution
- Verify Cb Events is taking up disk space
- Check current disk space:
df -h
- Check folder size in /var/cb/data since most of the disk usage is here:
du -h /var/cb/data/ --max-depth=1
- Check folder size in /var/cb/data/solr since most of the disk usage is here under the data folder:
du -h /var/cb/data/solr --max-depth=1
- Check if environment settings are incorrect
Verify that OER disk requirements are being followed:
Carbon Black Response v5.2 - Operating Environment Requirements
- Check expected number of days to keep, so we know what to put in the curl command later:
grep MaxEventStore /etc/cb/cb.conf
- Verify /etc/cb/cb.conf has values that are consistant with OER/Defaults:
MaxEventStoreSizeInDocs=60
MaxEventStoreSizeInPercent=70
MaxEventStoreDays=30
# MaxEventStoreSizeInMB=100000
- Delete old storefiles to make enough room to start services: How To Purge Storefile (Modulestore Binary) Data
- Purge event data to retention settings: How To Purge CbEvent (Process) Data in Cb Response
- Correct EventPurgeEarliestTime if the date is in the past:
Note: this issue could occur between v4.2.3 – v.5.1.0, but incorrect settings could still be present after upgrading
psql cb -p 5002 -c 'select * from cb_settings;'
- Run this to remove the future date if present on EventPurgeEarliestTime
psql cb -p 5002 -c "delete from cb_settings where key like 'EventPurge%';"
- Verify that the events_purge job is running correctly
/usr/bin/python -m cb.maintenance.job_runner -s events_purge >> /var/log/cb/job-runner/startup.out 2>&1
- Run this to verify that the date is now correct (past date):
psql cb -p 5002 -c 'select * from cb_settings;'
- Start the main CB service and verify the UI is accessible:
service cb-enterprise start |