Threat Report: Exposing Malware in Linux-Based Multi-Cloud Environments | Download Now

Collecting logs for System Hang Issues [Sensor - Cb Response]

Collecting logs for System Hang Issues [Sensor - Cb Response]

Version

Cb Response 4.x, 5.x, 6.x

Topic

This document describes how to collect Cb Response sensor diagnostics when a system suddenly hangs.

Check here if one of these solutions fit your issue better:

Steps

Follow steps below for a given OS platform to collect sensor diagnostics to diagnose the system hang.

Windows

Note: If a Cb Protection Agent is installed, you may need to disable the Cb Tamper Protect Updater to gain read access to the Diagnostics folder on the Windows platform

  1. Answer the following questions in your case:
    1. Is this a reproducible scenario and if so, what are the steps to reproduce the system hang?
      a. Was the host performing any special actions when the hang occurred? For example, were any backups, or large file transfers being performed?
    2. How many systems are affected?
    3. If this is affecting multiple systems is there anything in common between them such as OS, sensor version, function (ex. Web Server, Database Server,NFS)? How to find the Carbon Black sensor version
    4. Are there any other security applications/real-time scanners installed?
      a. If so, are Sensor exclusions in place? Anti-Virus Exclusions for Cb Response Sensor
      b. Is the Cb Protection Agent installed, and if so, what version?
      c. Is the Cb Defense (confer) Sensor installed, and if so, what version?
  2. Follow the Operating System vendor's Instructions to configure a memory dump:
    Note: By default, a Windows system is configured to only save a mini-dump memory file when a BSOD occurs. This may not be enough to diagnose and resolve the issue. Either a kernel (minimal) or a complete (preferred) memory dump is requested
    1. Windows Vista and up, 2K, 2K8 R2 and up
    2. Windows XP, 2K3
  3. Follow the Operation System vendor's instructions on Forcing a System Crash from the Keyboard: Forcing a System Crash from the Keyboard | Microsoft Docs
    Warning: To diagnose the issue, we will need a manually generated a Blue Screen of Death (BSOD) using a keyboard shortcut while the hang is experienced
  4. Enable Enable verbose logging of the Cb Response sensor service
    Warning: Be careful the additional logging does not fill up disk
  5. Once the hang/freeze is experienced, manually trigger a BSOD using the keyboard shortcut
  6. Once the system reboots, compress the c:\Windows\memory.dmp file (default location for memory dump files)
  7. Diagnostic utility to collect Carbon Black endpoint logs
    IMPORTANT: when complete, don't forget to remove verbose logging
  8. Export the Application and System logs from Windows Event Viewer (in .evt or .evtx format): How to Export Windows Event Logs
  9. Send the compressed memory.dmp, sensor diagnostics (tgz archive), Application and System Windows Event logs using instructions in the Uploading Files section below

MacOS

  1. Answer the following questions in your case:
    1. Is this a reproducible scenario and if so, what are the steps to reproduce the system hang?
      a. Was the host performing any special actions when the hang occurred? For example, were any backups, or large file transfers being performed?
    2. How many systems are affected?
    3. If this is affecting multiple systems is there anything in common between them such as OS, sensor version, function (ex. Web Server, Database Server,NFS)? How to find the Carbon Black sensor version
    4. Are there any other security applications/real-time scanners installed?
      a. If so, are Sensor exclusions in place? Anti-Virus Exclusions for Cb Response Sensor
      b. Is the Cb Protection Agent installed, and if so, what version?
      c. Is the Cb Defense (confer) Sensor installed, and if so, what version?
  2. Run the following command in a terminal session:
    sudo /Applications/CarbonBlack/sensordiag.sh
  3. This will create an archive of logs in the current working directory with the sensordiag_(hostname)_(date).tgz filename format
  4. Send the sensordiag_(hostname)_(date).tgz file using instructions in the Uploading Files section below

Linux

  1. Answer the following questions in your case:
    1. Is this a reproducible scenario and if so, what are the steps to reproduce the system hang?
      a. Was the host performing any special actions when the hang occurred? For example, were any backups, or large file transfers being performed?
    2. How many systems are affected?
    3. If this is affecting multiple systems is there anything in common between them such as OS, sensor version, function (ex. Web Server, Database Server,NFS)? How to find the Carbon Black sensor version
    4. Are there any other security applications/real-time scanners installed?
      a. If so, are Sensor exclusions in place? Anti-Virus Exclusions for Cb Response Sensor
      b. Is the Cb Protection Agent installed, and if so, what version?
      c. Is the Cb Defense (confer) Sensor installed, and if so, what version?
  2. If the /var/crash directory does not have the recent crash dump, you will need to prepare your system for collecting a crash dump prior to reproducing the kernel panic.
    Note: Please refer to the Operating System vendor's instructions. For example, Chapter 30 of the Red Hat Deployment Guide​ has the instructions on installing and configuring the kdump service for RHEL 6
  3. Once kdump is configured, reproduce the kernel panic or wait for one to occur before proceeding
  4. Run the following command in a terminal session as root:
    /opt/cbsensor/sensordiag.sh
  5. This will create an archive of logs in the current working directory with the sensordiag_(hostname)_(date).tgz filename format and will include the /var/crash directory
  6. If possible, take a VM Snapshot while it is running and collect the .vmsn file
    Note: if the .vmsn file is small (~1.5MB), this means the VM is configured to store the memory in a separate file. If so, please also send the .vmem file
  7. Send the sensordiag_(hostname)_(date).tgz and .vmsn/.vmem file(s) using instructions in the Uploading Files section below

Uploading Files

If under 25 MB, files can be attached to your case. Otherwise upload the collected data at Cb Vault

Labels (1)
Was this article helpful? Yes No
0% helpful (0/1)
Article Information
Author:
Creation Date:
‎02-25-2016
Views:
3849
Contributors