Just Published! Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

Pre 5.1.1 - Psql Reaching Maximum Record Count Interupting Watchlist Hits

Pre 5.1.1 - Psql Reaching Maximum Record Count Interupting Watchlist Hits

Version

Cb Response Versions Prior to 5.1.1

Issue

Events that match a Watchlist's query are not being tagged

Symptoms

If an event is not tagged appropriately as a Watchlist hit, the subsequent symptoms can be:

1. No Email Alert for a Watchlist hit

2. No Syslog message for a Watchlist hit

3. No Alerts created for a Watchlist hit

4. The Watchlist's page "Hit Count Over Time" graph is empty.

5. The Watchlist's page is empty, meanwhile the "Search" button shows events can be found.

The following "integer out of range" message can be seen in the /var/log/cb/job-runner/job-runner.log logs showing a failure for the watchlist_search job:

Aug 31 03:20:38 [17417] <err> [watchlist_search]
... _connection_for_bind self._assert_is_active() File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 213, in _assert_is_active % self._rollback_exception InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (DataError) integer out of range 'UPDATE watchlist_entries SET search_timestamp=%(search_timestamp)s, total_hits=%(total_hits)s WHERE watchlist_entries.id = %(watchlist_entries_id)s' {'watchlist_entries_id': 21, 'total_hits': 2247215139, 'search_timestamp': '2015-08-31T07:20:02.522703Z'}

Cause

The Postgres database's watchlist_entries table  on the Carbon Black Master server reached a maximum limit number of records.  The 32-bit integer data type for the 'id' column is too small for the affected environment.

Solution

Upgrade

This issue is fixed in 5.1.1. Upgrade to the latest version.

Modify Columns

Alternately, the moduleinfo_events column id can be updated from int the data type bigint:

Warning: This will not fully resolve the issue since watchlist_entries columns total_hits and total_tags should not be altered as it requires a code change to work properly

1. Login to the Master CB server as root.

2. Run the following "ALTER TABLE" commands:

psql -d cb -p 5002 -c "ALTER TABLE moduleinfo_events ALTER COLUMN id SET DATA TYPE bigint;"

3. No need to restart any services.  Monitor the /varlog/cb/job-runner/job-runner.log logs to make sure the watchlist_search job is no longer failing, and is tagging events appropriately. After a few minutes, check the UI for new events that are now tagged.

If you are receiving this error for another column, these are save to modify to bigint without updating to 5.1.1:

vt_write_events id

sensor_event_diagnostics id

moduleinfo_events id

sensor_queued_data_stats id

sensor_activity id

If the integer out of range error is for a field other than the above fields, an upgrade will be required. In addition to the above fields, these are automatically updated to bigint in the 5.1.1 release:

Note: These fields can not be modified in versions prior to 5.1.1 since they require a code change to operate correctly

sensor_registrations id

watchlist_entries total_hits,total_tags

sensor_driver_diagnostics id

sensor_comm_failures id

sensor_component_statuses id

sensor_resource_statuses id

sensor_uninstall_attempts id

sensor_upgrade_attempts id

watchlist_entry_user_associations watchlist_entry_id

watchlist_notifications id,watchlist_entry_id

Labels (1)
Was this article helpful? Yes No
No ratings
Article Information
Author:
Creation Date:
‎09-16-2015
Views:
611
Contributors