Just Published! Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

5.x no disk space available on Cb Response server due to cbevent purge settings in past

5.x no disk space available on Cb Response server due to cbevent purge settings in past

Version

Cb Response 4.x, 5.x

Note: For 6.x follow this guide instead: No Free Disk Space Prevents Cb Response Services From Starting

Issue

The cb-enterprise service will not start with no disk space available on Cb Response server due to purge settings in past

Symptoms

df –h shows that the /var/cb is 100% used and only a small disk space is left and not enough to allow a service to start.

Cause

There’s a defect on version of 4.2.3 to v5.1.0 where it doesn't apply default values for purging. This is addressed on a hotfix but requires the CB server to be on at least v5.0.0 patch 3. CB v5.0.0 patch 4 and 5.1.0 patch 1 will both include the hotfix.

Solution

  1. Verify Cb Events is taking up disk space
    1. Check current disk space:
      df -h
    2. Check folder size in /var/cb/data since most of the disk usage is here:
      du -h /var/cb/data/ --max-depth=1
    3. Check folder size in /var/cb/data/solr since most of the disk usage is here under the data folder:
      du -h /var/cb/data/solr --max-depth=1
  2. Check if environment settings are incorrect
    1. Verify that OER disk requirements are being followed:
      Carbon Black Response v5.2 - Operating Environment Requirements

    1. Check expected number of days to keep, so we know what to put in the curl command later:
      grep MaxEventStore /etc/cb/cb.conf
    2. Verify /etc/cb/cb.conf has values that are consistant with OER/Defaults:
      MaxEventStoreSizeInDocs=60
      MaxEventStoreSizeInPercent=70
      MaxEventStoreDays=30
      # MaxEventStoreSizeInMB=100000
  3. Delete old storefiles to make enough room to start services: How To Purge Storefile (Modulestore Binary) Data
  4. Purge event data to retention settings: How To Purge CbEvent (Process) Data in Cb Response
  5. Correct EventPurgeEarliestTime if the date is in the past:
    Note: this issue could occur between v4.2.3 – v.5.1.0, but incorrect settings could still be present after upgrading
    psql cb -p 5002 -c 'select * from cb_settings;'
    1. Run this to remove the future date if present on EventPurgeEarliestTime
      psql cb -p 5002 -c "delete from cb_settings where key like 'EventPurge%';"
    2. Verify that the events_purge job is running correctly
      /usr/bin/python -m cb.maintenance.job_runner -s events_purge >> /var/log/cb/job-runner/startup.out 2>&1
    3. Run this to verify that the date is now correct (past date):
      psql cb -p 5002 -c 'select * from cb_settings;'
  6. Start the main CB service and verify the UI is accessible:
    service cb-enterprise start
Labels (1)
Was this article helpful? Yes No
No ratings
Article Information
Author:
Creation Date:
‎08-06-2015
Views:
1191
Contributors