Environment
- EDR Server: 7.x and Higher
Objective
Enable Solr debug logging to troubleshoot long running queries
Resolution
- Edit /etc/cb/solr/log4j2.xml
- Delete the comment lines shown in red, both lines above (<--) and below (-->) the debug logging line are removed
<--
<AsyncLogger name="com.carbonblack.cbfs.solr.handler.CbSearchRequestHandlers" level="DEBUG" />
-->
- Reproduce the search issue. If this is a watchlist timeout, the job will run every 10 minutes
- After reproduction, use the following command to back up the debug and restore normal logging
mv /etc/cb/solr/log4j2.xml /etc/cb/solr/log4j2.xml.debug && cp /etc/cb/solr/log4j2.xml.template /etc/cb/solr/log4j2.xml
Additional Notes
- No service restart is required for Solr to pick up the new logging settings
- Since this setting is trying to find long running search queries, logs for troubleshooting should be collected no sooner than 30 minutes after reproducing the issue.
- Debug logging will list all queries active and completed every 10 seconds
- Restarting the services will also cause EDR to replace the existing log4j2.xml with the log4j2.xml.template
- For Solr debugging to survive a restart of the services the log4j2.xml.template can be modified