Version
Cb Response 5.x, 6.x
Issue
Bandwidth at branches fully utilized by Cb Response
Symptoms
Nginx /var/log/cb/nginx/access.log:
<IPAddress> - - [13/Apr/2017:03:49:24 -0500(0.758)] "POST /data/storefile/submit/<SensorID> HTTP/1.1" 500 186 "-" "" "-" "-" |
Nginx /var/log/cb/nginx/error.log:
2017/04/17 03:15:32 [crit] 14296#0: *6263373 open() "/var/cb/nginx/client_temp/9/56/0000339569" failed (13: Permission denied), client: <IPAddress>, server: , request: "POST /data/storefile/submit/<SensorID> HTTP/1.1", host: “<SensorIPAddress>” |
Cause
The Cb Response Server is unable to accept binaries from sensors since it is unable to write to its working directory /var/cb/nginx/client_temp. Sensors continue to attempt to upload the binary to the server causing network bandwidth problems. This can occur after a server move where permissions were not correctly copied over.
Solution
- Stop cluster:
/usr/share/cb/cbcluster stop |
- Check for hanging processes:
- Move client_temp so that nginx would regenerate the directory:
mv /var/cb/nginx/client_temp /var/cb/nginx/client_temp_old |
- Start cluster:
/usr/share/cb/cbcluster start |
- Verify that minions are receiving 200's for storefile submits:
tail -f /var/log/cb/nginx/access.log |