EDR:Job <cron name> was not run because cb-enterprise is stopped.

[#1]

Environment

• VMware Carbon Black EDR Server: All Versions

Symptoms

One of the following messages are logged in /var/log/cb/job-runner/job-runner.log when a cron job fails to start:

• "Job watchlist_search was not run because cb-enterprise is stopped."
• "Job modulestore_purge was not run because cb-enterprise is stopped."
• "Job solr_optimize was not run because cb-enterprise is stopped."

Cause

This can occur when cb-enterprise is truly stopped or in the process of stopping. This can also occur in some cases even when running "service cb-enterprise status" or "systemctl status cb-enterprise" with root permissions and it returns all services in a RUNNING state.
 

Resolution

In one scenario, running an strace with the cb-supervisord status command logged a failed attempt to open the file /etc/profile.d/lang.sh due to a permissions issue:

open("/etc/profile.d/lang.sh", O_RDONLY) = -1 EACCES (Permission denied)

In this case, the directory permissions did not allow execution of the lang.sh and caused the service cb-supervisord status command to return nothing. This condition then caused the cron job to stop executing due. To correct this the directory permissions were modified to octal 755 with this command:

chmod 755 /etc/profile.d

Additional Notes

• There might be several other causes for the /sbin/service wrapper to not redirect to the /bin/systemctl command. Troubleshooting this is usually possible by prepending an strace command to the service you are trying to start, stop, or retrieve status information from. For example, this would trace the execution of a "service cb-supervisord status" command and send it to the file strace.txt: 

sudo -u cb strace /sbin/service cb-supervisord status &> strace.txt

• On RHEL/CentOS 7 or newer operating systems the /sbin/service script is a wrapper for backwards compatibility and should redirect to systemctl to interact with services. EDR internally uses /sbin/service commands for some functions and expects the redirect to systemctl to be working properly. If this redirect is failing, the /sbin/service command might fail to function properly and not return any status information. By default, the job-runner cb.maintenance cron jobs rely on a check to see if cb-enterprise is running to ensure the cron jobs do not spend unnecessary time executing when EDR is stopped or in the process of stopping. This check involves three factors:

  1. cb-supervisord status is not in a stopped state (i.e. "/sbin/service cb-supervisord status" returns properly)
  2. cb-enterprise in not currently in the process of stopping
  3. the -f --force option is not enabled

• By default, if either #1 or #2 are true, the cron job will exit with the described error message. It is possible to workaround the issue by skipping the checks #1 and #2 with the -f or --force option added to the specific cron job command found in /etc/cb/cron/cb.cron.template. 
• If #2 is false because the all services are in the RUNNING state, then the issue is with the status of cb-supervisord not returning properly.