Carbon Black Cloud: How to Collect Sensor Performance Logs with Sensor Capture Script

[#1]

Environment

• Carbon Black Cloud Sensor: All Supported Versions
• Windows: All Supported Versions

Objective

Steps to collect Process Monitor (Procmon) Logs, Windows Performance Recorder (WPR) ETL Trace, and Sensor Diagnostic logs with Sensor Capture Script for troubleshooting issues sensor performance issues

Resolution

These instructions should only be used at the direction of VMWare Carbon Black Support team. Please open a Support Case and the Sensor Capture Script will be provided to collect the logs below if needed. 

Prerequisites

1. RepCLI Authentication must be enabled. If RepCLI Authentication was not enabled during the initial sensor install then RepCLI Authentication can be enabled on existing sensor installations
2. Create a folder where all logs will be saved. For the purposes of this document, this location will be referenced as C:\temp although the C:\temp file location can be replaced with whatever location you have specified for saving the log files. 
3. Download the cbc-sensor-capture.ps1.zip attached to the Support Case
4. Copy cbc-sensor-capture.ps1 to C:\temp
5. Download download ProcmonLowAlt.exe.zip at the bottom of https://community.carbonblack.com/t5/Knowledge-Base/All-Products-How-to-Collect-a-low-Altitude-Procm... or download Procmon directly from Microsoft and configure as per Option 2. 
6. Unzip procmon and copy to C:\temp

NOTE: For sensor 3.8 and above, RepCLI Repro can be used to collect this data, but Procmon.exe must be downloaded directly from Microsoft as the ProcmonLowAlt.zip attached to https://community.carbonblack.com/t5/Knowledge-Base/All-Products-How-to-Collect-a-low-Altitude-Procm... cannot be used as this version of procmon is not signed by a valid publisher. RepCLI Repro can only invoke procmon when it has been signed by a valid publisher.

7. Ensure wpr.exe exists in C:\Windows\System32\ 

NOTE: If C:\Windows\System32\wpr.exe does not exist, download Debugging Tools for Windows and at the "Select the features you want to download" install prompt deselect all other options except "Windows Performance Toolkit".  WPR.exe will download to C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit by default. Once downloaded copy wpr.exe to C:\Windows\System32\ 

Sensor Active
Reproduce the behavior when Sensor is Active

1. Open Command Prompt using the "Run As Administrator" option
2. Change Directory to C:\temp 

C:\WINDOWS\system32>cd to C:\temp
C:\temp>

3. Run the following command to reset sensor counters and retain PSC events 

C:\temp>Powershell -ExecutionPolicy bypass -f cbc-sensor-capture.ps1 keepevents

4. When the Enter the uninstall code to unlock restricted RepCLI commands.: prompt is presented, enter the uninstall code
5. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the behavior
   1. Document exact steps taken to reproduce to provide to Support
   2. Note local machine time when testing started, and timezone of machine
   3. Note local time when issue occurred
6. Once behavior is fully reproduced, press Enter to exit the capture
7. A zip will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip). Prefix with "keepevents-active" (i.e. keepevents-active-hostname-YYYYDDMMHHMMSS.zip)
8. Run the following command to start wpr trace depending on the estimated time needed to reproduce

More than 5 minutes

C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 wpr

Less than 5 minutes 

C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 minifilter

9. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the behavior
10. Once behavior is fully reproduced, press Enter to exit the capture
    1. Document exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
11. A folder will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip). Prefix with "wpr-active" (i.e. wpr-active-hostname-YYYYDDMMHHMMSS.zip)

NOTE: If the files are too large, the script may be unable to zip the folder. In this case, please manually compress the folder into a zip file. 

NOTE: If the issue takes longer than 10 minutes to reproduce the issue, then the resulting log files may take up excessive disk space 

12. Run the following command to start procmon

C:\temp>powershell -executionpolicy bypass -f c:\temp\cbc-sensor-capture.ps1 procmon 

13. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the behavior
14. Once behavior is fully reproduced, press Enter to exit the capture
15. A zip will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip). Prefix with "procmon-active" (i.e. procmon-active-hostname-YYYYDDMMHHMMSS.zip)

NOTE: If the files are too large, the script may be unable to zip the folder. In this case, please manually compress the folder into a zip file
NOTE: If the issue takes longer than 10 minutes to reproduce the issue, then the resulting log files may take up excessive disk space 

Sensor Bypass
Reproduce the behavior when Sensor is in Bypass

1. Run the following command to start wpr trace depending on the estimated time needed to reproduce

More than 5 minutes

C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 wpr bypass

Less than 5 minutes 

C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 minifilter bypass

2. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the issue
3. Once issue fully reproduced, press Enter to exit the capture
   1. Document exact steps taken to reproduce to provide to Support
   2. Note local machine time when testing started, and timezone of machine
   3. Note local time when issue occurred
4. A folder will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip). Prefix with "wpr-bypass" (i.e. wpr-bypass-hostname-YYYYDDMMHHMMSS.zip)
5. Run the following command:

C:\temp>powershell -executionpolicy bypass -f c:\temp\cbc-sensor-capture.ps1 procmon bypass

6. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the issue
7. Once the behavior has been reproduce, press Enter to exit the capture
8. A folder will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip). Prefix with "procmon-bypass" (i.e. procmon-bypass-hostname-YYYYDDMMHHMMSS.zip)
9. Go to C:\temp and zip the files below and rename zip as perfcapture-logs.zip
   1. keepevents-active-hostname-YYYYDDMMHHMMSS.zip
   2. wpr-active-hostname-YYYYDDMMHHMMSS.zip
   3. procmon-active-hostname-YYYYDDMMHHMMSS.zip
   4. wpr-bypass-hostname-YYYYDDMMHHMMSS.zip
   5. procmon-bypass-hostname-YYYYDDMMHHMMSS.zip
10. Upload the zip files created in C:\temp to CB Vault

Additional Notes

• The ProcmonLowAlt.zip attached to All Products: How to Collect a low Altitude Procmon Capture was modified so that the configuration steps and reboot typically required if procmon is downloaded directly from Microsoft are not necessary; however, the modified version of procmon included in ProcmonLowAlt.zip has not been signed 
• The WPR Trace cannot be collected at the same time as a Procmon Log
• Sensor Capture Script resets counters by default unless skipreset is specified
• The keepevents parameter requires the uninstall code even if "Require code to uninstall sensor" is not enabled on the sensor policy
• The PSC events should be collected separately from procmon and wpr if possible

Related Content

Carbon Black Cloud: How to Enable RepCLI Authentication on Existing Sensors
Carbon Black Cloud: How to Enable Authentication for RepCLI Utility During Sensor Install