Access official resources from Carbon Black experts
5.2.x
Cluster has a growing backlog. Sensors checking in but not submitting data
/var/log/cb/enterprise/enterprise.log:
2016-03-01 23:31:01 [14244] <err> cb.enterprise.tasks.throttle_calc - Error updating sensor data throttle. Trying again in 10 seconds.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cb/enterprise/tasks/throttle_calc.py", line 45, in work
File "/usr/lib/python2.6/site-packages/cb/core/throttle/throttler.py", line 28, in update
File "/usr/lib/python2.6/site-packages/cb/core/throttle/throttle_rates.py", line 22, in update
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/rate.py", line 24, in update
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/rate.py", line 45, in update
File "/usr/lib/python2.6/site-packages/cb/core/throttle/pressured_rate.py", line 22, in _calc_rate
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/rate_pressure.py", line 33, in update
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/pressures/datastore_cache_pressure.py", line 34, in _update
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/pressures/datastore_cache_pressure.py", line 53, in _get_stats
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/stats/query.py", line 43, in collect_stats
File "/usr/lib/python2.6/site-packages/cb/core/stats/query_source.py", line 114, in _collect_stats
File "/usr/lib/python2.6/site-packages/cb/core/stats/query_source.py", line 117, in _query_redis
File "/usr/lib/python2.6/site-packages/redis/client.py", line 1387, in hgetall
return self.execute_command('HGETALL', name)
File "/usr/lib/python2.6/site-packages/redis/client.py", line 397, in execute_command
connection.send_command(*args)
File "/usr/lib/python2.6/site-packages/redis/connection.py", line 306, in send_command
self.send_packed_command(self.pack_command(...
2016-03-01 23:31:01 [14244] <err> cb.enterprise.tasks.throttle_calc -
...*args))
File "/usr/lib/python2.6/site-packages/redis/connection.py", line 288, in send_packed_command
self.connect()
File "/usr/lib/python2.6/site-packages/redis/connection.py", line 235, in connect
raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting 10.107.74.45:6379. Connection refused.
2016-03-01 23:32:11 [14244] <err> cb.enterprise.tasks.throttle_calc - Error updating sensor data throttle. Trying again in 10 seconds.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cb/enterprise/tasks/throttle_calc.py", line 45, in work
File "/usr/lib/python2.6/site-packages/cb/core/throttle/throttler.py", line 28, in update
File "/usr/lib/python2.6/site-packages/cb/core/throttle/throttle_rates.py", line 22, in update
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/rate.py", line 24, in update
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/rate.py", line 45, in update
File "/usr/lib/python2.6/site-packages/cb/core/throttle/pressured_rate.py", line 22, in _calc_rate
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/rate_pressure.py", line 33, in update
File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
self.gen.throw(type, value, traceback)
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/rate_pressure.py", line 63, in _all_stop_context
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/rate_pressure.py", line 33, in update
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/pressures/datastore_cache_pressure.py", line 34, in _update
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/throttle/pressures/datastore_cache_pressure.py", line 53, in _get_stats
File "/opt/jenkins/builds/workspace/build-cbent-release-5.1.1/code/coreservices/src/cb/core/stats/query.py", line 43, in collect_stats
File "/usr/lib/python2.6/site-packages/cb/core/stats/query_source.py", line 114, in _collect_stats
File "/usr/lib/python2.6/site-packages/cb/core/stats/query_sources/datastore_solr_client_stats.py", line 8, in next_time_interval
File "/usr/lib/python2.6/site-packages/cb/core/stats/query_source.py", line 41, in next_time_interval
KeyError: 'num_bytes_pushed'
Sensor eventlog throttle may end up in a permanent bad state if Redis connection is lost and restored
Restart services:
service cb-enterprise restart
/usr/share/cb/cbcluster stop
/usr/share/cb/cbcluster start
This issue is resolved in 5.3.1 and 6.x. Please check 5.3.1 release notes for more information on CB-8472.