Version
Cb Response Versions Prior to 5.1.1
Issue
Events that match a Watchlist's query are not being tagged
Symptoms
If an event is not tagged appropriately as a Watchlist hit, the subsequent symptoms can be:
1. No Email Alert for a Watchlist hit
2. No Syslog message for a Watchlist hit
3. No Alerts created for a Watchlist hit
4. The Watchlist's page "Hit Count Over Time" graph is empty.
5. The Watchlist's page is empty, meanwhile the "Search" button shows events can be found.
The following "integer out of range" message can be seen in the /var/log/cb/job-runner/job-runner.log logs showing a failure for the watchlist_search job:
Aug 31 03:20:38 [17417] <err> [watchlist_search]
... _connection_for_bind self._assert_is_active() File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 213, in _assert_is_active % self._rollback_exception InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (DataError) integer out of range 'UPDATE watchlist_entries SET search_timestamp=%(search_timestamp)s, total_hits=%(total_hits)s WHERE watchlist_entries.id = %(watchlist_entries_id)s' {'watchlist_entries_id': 21, 'total_hits': 2247215139, 'search_timestamp': '2015-08-31T07:20:02.522703Z'} |
Cause
The Postgres database's watchlist_entries table on the Carbon Black Master server reached a maximum limit number of records. The 32-bit integer data type for the 'id' column is too small for the affected environment.
Solution
Upgrade
This issue is fixed in 5.1.1. Upgrade to the latest version.
Modify Columns
Alternately, the moduleinfo_events column id can be updated from int the data type bigint:
Warning: This will not fully resolve the issue since watchlist_entries columns total_hits and total_tags should not be altered as it requires a code change to work properly
1. Login to the Master CB server as root.
2. Run the following "ALTER TABLE" commands:
psql -d cb -p 5002 -c "ALTER TABLE moduleinfo_events ALTER COLUMN id SET DATA TYPE bigint;" |
3. No need to restart any services. Monitor the /varlog/cb/job-runner/job-runner.log logs to make sure the watchlist_search job is no longer failing, and is tagging events appropriately. After a few minutes, check the UI for new events that are now tagged.
If you are receiving this error for another column, these are save to modify to bigint without updating to 5.1.1:
vt_write_events id sensor_event_diagnostics id moduleinfo_events id sensor_queued_data_stats id sensor_activity id |
If the integer out of range error is for a field other than the above fields, an upgrade will be required. In addition to the above fields, these are automatically updated to bigint in the 5.1.1 release:
Note: These fields can not be modified in versions prior to 5.1.1 since they require a code change to operate correctly
sensor_registrations id watchlist_entries total_hits,total_tags sensor_driver_diagnostics id sensor_comm_failures id sensor_component_statuses id sensor_resource_statuses id sensor_uninstall_attempts id sensor_upgrade_attempts id watchlist_entry_user_associations watchlist_entry_id watchlist_notifications id,watchlist_entry_id |
