Carbon Black Cloud: How to Collect Sensor Performance Logs Manually (Windows)

[#1]

Environment

• Carbon Black Cloud Sensor: All Supported Versions
• Microsoft Windows: All supported versions

Objective

Steps to manually collect Process Monitor (Procmon) Logs, Windows Performance Recorder (WPR) ETL Trace, and Sensor Diagnostic logs for troubleshooting issues sensor performance issues

Resolution

Prerequisites

1. RepCLI Authentication must be enabled. If RepCLI Authentication was not enabled during the initial sensor install then RepCLI Authentication can be enabled on existing sensor installations
2. Create a folder where all logs will be saved. For the purposes of this document, this location will be referenced as c:\temp although the c:\temp file location can be replaced with whatever location you have specified for saving the log files. 
3. Download download ProcmonLowAlt.exe.zip at the bottom of https://community.carbonblack.com/t5/Knowledge-Base/All-Products-How-to-Collect-a-low-Altitude-Procm... or download Procmon directly from Microsoft and configure as per Option 2
4. Ensure wpr.exe exists in C:\Windows\System32\ 

NOTE: If C:\Windows\System32\wpr.exe does not exist, download Debugging Tools for Windows and at the "Select the features you want to download" install prompt deselect all other options except "Windows Performance Toolkit".  WPR.exe will download to C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit by default. Once downloaded copy wpr.exe to C:\Windows\System32\ 

 

Sensor Active:
Reproduce the behavior when Sensor is Active

1. Open a command line prompt using "Run as Administrator".
2. Run the following command to start the WPR Trace (ETL) Log and reset counters depending on the estimated time needed to reproduce

Less than 5 Minutes 

C:\temp>wpr -start GeneralProfile -start CPU -start Registry -start FileIO -start DiskIO -start Minifilter -start Network -filemode

More than 5 Minutes 

C:\temp>wpr -start GeneralProfile -start CPU -start Registry -start FileIO -start DiskIO -start Network -filemode

3. Reproduce the behavior
   1. Document exact steps taken to reproduce to provide to Support
   2. Note local machine time when testing started, and timezone of machine
   3. Note local time when issue occurred
4. Collect the wpr-active.etl, counters.txt, and psc_sensor.zip 

A. C:\Program Files\Confer>repcli counters > C:\temp\counters.txt -- Change to desired location 
B. C:\temp>wpr -stop c:\temp\wpr-active.etl
C. C:\Program Files\Confer>repcli capture c:\temp -- Change to desired location
Collecting diagnostic data (this may take a few minutes)...
....
Captured diagnostic data in c:\temp\psc_sensor.zip

5. Rename counters.txt to wpr-active-counters.txt, and psc_sensor.zip to wpr-active-psc_sensor.zip
6. Run the following commands to collect the Procmon, Sensor logs, Counter logs :

A. C:\WINDOWS\system32>cd c:\program files\confer
B. C:\Program Files\Confer>sc qprotection cbdefense --Result will show either ANTIMALWARE LIGHT or None. If none skip to step H.
C. C:\Program Files\Confer>repcli bypass 1
D. C:\Program Files\Confer>repcli registerProtectedSvcs 0 
E. C:\Program Files\Confer>repcli stopCbServices
F. C:\Program Files\Confer>sc start cbdefense
G. C:\Program Files\Confer>repcli bypass 0
H. C:\Program Files\Confer>repcli deletepolicy 1DED7E47-CE4C-448E-AD01-6F4AC3CE7F5D
I. C:\Program Files\Confer>repcli resetcounters

7. Launch Procmon.exe
8. Start collection in Procmon (CTRL+E)
9. Reproduce the behavior
   1. Document exact steps taken to reproduce to provide to Support
   2. Note local machine time when testing started, and timezone of machine
   3. Note local time when issue occurred
10. Stop collection in Procmon (CTRL+E) and save the log file
    1. In the Save pop-up window, Select "Events to save: All events" and "Format: PML"
    2. Change the file name to procmon-active.PML
    3. Click "Save" and save to C:\temp 
11. Run commands to collect counters.txt and psc_sensor.zip  

A. C:\Program Files\Confer>repcli counters > C:\temp\counters.txt -- Change to desired location
B. C:\Program Files\Confer>repcli capture c:\temp -- Change to desired location
Collecting diagnostic data (this may take a few minutes)...
....
Captured diagnostic data in c:\temp\psc_sensor.zip 

12. Rename counters.txt to procmon-active-counters.txt, and psc_sensor.zip to procmon-active-psc_sensor.zip

Sensor Bypass:
Reproduce the behavior when Sensor is in Bypass

1. Place the sensor into bypass mode:

 C:\Program Files\Confer>repcli bypass 1

2. Run the following command to start the WPR Trace (ETL) Log and reset counters depending on the estimated time needed to reproduce

Less than 5 Minutes 

C:\temp>wpr -start GeneralProfile -start CPU -start Registry -start FileIO -start DiskIO -start Minifilter -start Network -filemode

More than 5 Minutes 

wpr -start GeneralProfile -start CPU -start Registry -start FileIO -start DiskIO -start Network -filemode

3. Reproduce the behavior
   1. Document exact steps taken to reproduce to provide to Support
   2. Note local machine time when testing started, and timezone of machine
   3. Note local time when issue occurred
4. Collect the wpr-bypass.etl 

C:\temp>wpr -stop c:\temp\wpr-bypass.etl

5. Launch Procmon.exe
6. Start collection in Procmon (CTRL+E)
7. Reproduce the behavior
   1. Document exact steps taken to reproduce to provide to Support
   2. Note local machine time when testing started, and timezone of machine
   3. Note local time when issue occurred
8. Stop collection in Procmon (CTRL+E) and save the log file
   1. In the Save pop-up window, Select "Events to save: All events" and "Format: PML"
   2. Change the file name to procmon-bypass.PML
   3. Click "Save" and save to C:\temp
9. Run commands to restore sensor:

A. C:\Program Files\Confer>repcli registerProtectedSvcs 1 -- Unnecessary if was skipped in step Sensor Active Step 6
B. C:\Program Files\Confer>repcli stopCbServices
NOTE: If using a sensor version prior to 3.6.0.1897, edit cfg.ini and remove "UnregisterProtected=True"
C. C:\Program Files\Confer>sc start cbdefense
D. C:\Program Files\Confer>repcli bypass 0

10. Go to C:\temp, zip the files below and rename zip as perf-logs.zip
    1. wpr-active.etl
    2. wpr-active-counters.txt
    3. wpr-active-psc_sensor.zip
    4. wpr-bypass.etl
    5. procmon-active.PML
    6. procmon-active-counters.txt
    7. procmon-active-psc_sensor.zip
    8. procmon-bypass.PML
11. Upload the zip to CB Vault
12. Once the upload completes, please comment on the support case that the data is available for review

Additional Notes

• Process Monitor records a large amount of information, please make sure to provide an accurate time stamp in step 7, which will help expedite troubleshooting
• The WPR Trace cannot be collected at the same time as a Procmon Log
• The repcli unlock <uninstall-code> command is not needed for deleting a policy, only for adding/updating a policy.
• Both Sensor Service (repmgr stack) and File Filter Driver (ctifile) stack info are required to troubleshoot sensor performance issues. The steps above will ensure that Sensor Service (repmgr stack) info is included in Procmon Logs, but LowAltProcmon will be needed to ensure that File Filter Driver (ctifile) stack information is included in the procmon capture. 
• If Repcli Repro (Sensor 3.8+) or Sensor Capture Script is used then the above steps are not required to capture Sensor Service (repmgr stack) info; However if Repcli Repro is used procmon.exe must be downloaded directly from Microsoft as the ProcmonLowAlt.zip attached to All Products: How to Collect a low Altitude Procmon Capture cannot be used as this version of procmon is not signed by a valid publisher. RepCLI Repro cannot only invoke procmon when it has been signed by a valid publisher. 

Related Content

Carbon Black Cloud: How to Collect Sensor Performance Logs with Sensor Capture Script
Carbon Black Cloud: How to Enable RepCLI Authentication on Existing Sensors
Carbon Black Cloud: How To Collect Sensor Logs Locally (Windows)
All Products: How to Collect a low Altitude Procmon Capture