EDR: How is Data Retention Determined?

[#1]

Environment

• EDR Server:  All Supported Versions 

Question

How is Data Retention Determined?

Answer

Event retention is based on the first hit configuration in /etc/cb/cb.conf. 

• MaxEventStoreDays: Default is set to 30 days. If event data is older than 30 days the oldest event core will be removed or converted to cold storage if enabled.
• MaxEventStoreSizeInMB: Similar to MaxEventStoreSizeInPercent, if the size of the total event store hits this parameter, the oldest event cores will be removed or converted to cold storage until the size is back under the limit.
• MaxEventStoreSizeInPercent: Default is 90%. Determines if the partitioned disk size where the event cores reside has hit the total disk percentage. This is to avoid filling the disk, causing the EDR  Server to no longer function. The oldest core will be deleted or converted to cold storage if enabled.
• MinAvailableSizeInMB: This is set when AlwaysDeleteColdPartitions is set to False. This parameter when set will remove the oldest warm core instead of converting to cold storage if conditions are met.
• SolrTimePartitioningActivePartitions: This setting controls the number of actively searched Solr partitions that remain in the query index i.e warm cores. 
• SolrTimePartitioningMinutes: This setting controls how many minutes a partition will be written to before moving to the next writer core.

Additional Notes

• The server checks every 15 minutes to determine if data needs to be purged. 
• By default event cores consist of 3 days of data with a total of 10 cores when 30 days of retention is set. When purging, the oldest 3 days of data will be removed.
• For cold core storage CB Response: How to enable event cold storage
• If you have the cb-event-forwarder installed, follow this article to avoid hitting the disk size configs early. CB Response: Disk Full Due to JSON Files
• Physical Binaries stored in /var/cb/data/modulestore can take up disk space causing the disk size configs to purge cores early. CB Response: How To Purge Modulestore Binary Files
• See other server configuration settings in CB Response Server Configuration Guide

Related Content

Data Storage
EDR: How to adjust maximum Solr core sizes
EDR: How To Purge CbEvent (Process) Data in 6.x and Higher
EDR: Low data retention on the Server/Cluster with adequate space
EDR: Modulestore Filling Disk When Alliance Sharing Is Disabled
EDR: How To Purge Modulestore Binary Files
EDR: Disk Full Due to JSON Files