We’re migrating product documentation to docs.vmware.com, starting with Carbon Black Cloud. Learn more.

[Cb Response] Important Notice Related to BSOD Issues with 6.1.2 Windows Sensor

[Cb Response] Important Notice Related to BSOD Issues with 6.1.2 Windows Sensor

[UPDATED 4/23/18 at 10:30AM (ET)]

An update has been provided to our UeX post around upcoming general availability of the 6.1.6 Windows sensor: Potential Stability Issue for 6.1.4 Windows Sensor.

[UPDATED 3/27/18 at 5:41PM (ET)]

An update has been provided to our UeX post around the 6.1.4-win issue: Potential Stability Issue for 6.1.4 Windows Sensor.

[UPDATED 3/23/18 at 7:30PM (ET)]

**Important Note** A small number of customers have reached out to support indicating that they have experienced a situation in which some hosts running 6.1.4 have become unresponsive after sensor upgrade. For more details see our UeX post here Potential Stability Issue for 6.1.4 Windows Sensor

[UPDATED 3/23/18 at 8:50AM (ET)]

General Availability of 6.1.4-win for cloud environments is now available. For more details see our UeX post here: [Cb Response Cloud] Cloud Update Notification: 6.1.4 Windows Sensor

[UPDATED 3/16/18 at 4:50PM (ET)]

General Availability of 6.1.4-win (on-prem) is now available. For more details see our UeX post here: [Cb Response] Announcing General Availability of 6.1.4 Windows Sensor

[UPDATED 3/15/18 at 7:00PM (ET)]

General Availability of 6.1.4-win (on-prem) has been pushed to Friday March 16th

[UPDATED 3/14/18 at 11:07AM (ET)]

All,

The Cb Response team is getting set to release our 6.1.4 Windows sensor as generally available. Below are the targeted on-prem/cloud release dates for our upcoming 6.1.4-win Cb Response sensor:

Environment

Latest Version

Next Update

Next Version

On-Prem

6.1.2-win

March 15th

March 16th

6.1.4-win

Cloud

6.1.2-win

March 22nd

6.1.4-win

~ The Cb Response team

[UPDATED 2/8/18 at 11:13AM (ET)]

All,

Our internal testing of the 6.1.3 Windows sensor identified issues and limitations with suppression and reporting command line information for processes that we believe to have a critical impact on expected behavior. As such, the 6.1.3 Windows sensor will not be made available for Controlled Distribution as previously stated.

We are working to resolve these known issues in our follow up 6.1.4 Windows sensor release that will be made available as soon as we can provide it. Please open a case with our support team if you believe to be experiencing this issue.

The Cb Response team

[UPDATED 2/1/18 at 8:53AM (ET)]

All,

We are planning to make our 6.1.3 windows sensor available for controlled distribution next week. The sensor release will contain improvements to prevent a bug check from occurring should this same issue be observed.

The development team is still working to fully ascertain why this failed driver initialization is occurring. This bugcheck issue is sporadic. This does not impact a particular Windows OS version more so than others. The bug check is typically observed at the time of installation or very shortly thereafter. If no issues have been observed with the 6.1.2 sensor in your environment there is no need to revert back to an older sensor version. We encourage testing on 6.1.3 and future maintenance releases as they become available as we continue to make improvements to address this issue and others.

The Cb Response Team

[UPDATED 12/21/17]

Hi All,

We have learned that installing the Windows 6.1.2 Response sensor may lead to a stop error (Blue Screen of Death) for some environments.  

The issue is due to an unexpected failure during driver startup that results in the DriverEntry() function exiting early, preventing a full initialization of the driver. We believe this issue could potentially be related to other security products running on the endpoint. There are potential scenarios where one of these products could prevent access to certain system directories, such as ‘Windows’ and / or ‘Windows\System32’. This would result in a driver initialization failure, triggering a  cleanup code path that is usually called only after the driver has been fully initialized. Since the driver has not been fully initialized, the sensor ends up attempting to use uninitialized objects which results in a bug check.

For customers experiencing this issue, or believe to be a candidate, we recommend using the previous 6.0.3 windows sensor until a fix can be provided. We are working to address this issue in a follow up maintenance release that will be made available as early as possible. As always, be sure to test the newest version of the sensor on a representative subset of computers before deploying it to the entire enterprise. 

Thanks,

The Cb Response Team

Labels (1)
Comments

hello,

is this the criticial process died bsods we've reported (on resume from hibernate) or something else? case 00089792  (may or may not be related to CBR)

thanks.

Could you clarify:"For customers experiencing this issue, or believe to be a candidate"

What would make a customer more likely to be a candidate?

any updates. or everyone too busy testing the meltdown/spectre business?

Do we have any update on this?

How does one determine if they are a candidate for the BSOD?  This is now much more important since this is the only 6.1 sensor tested for the Spectre/Meltdown patch, see OS Testing for Meltdown/Spectre Patches

well this is going really well. ['radio silence' for a month] rhuang

is there a date for 6.1.3 for windows then ?

Agreed, it's now been 4 weeks...  I also like the dig at administrators at the end of the missive:

"As always, be sure to test the newest version of the sensor on a representative subset of computers before deploying it to the entire enterprise."

Clearly it's our fault.

Hi All,

The Cb Response team has made changes in our upcoming 6.1.3 Windows sensor release to prevent a bugcheck (blue screen) from occurring for this scenario. Unexpected failure during driver startup may still occur, however users can confirm if this is an issue from the Cb Response console under the Sensor tab of the left panel.

A status alert will display next to the affected ComputerName. Users can hover over the status alert to receive a tooltip with more information on the sensor alert reported. For users experiencing this issue, the tooltip will contain a message indicating that the driver has failed.  We are continuing to work on fully addressing this issue and will provide additional changes in a follow-up maintenance release.

The 6.1.3 Windows sensor release is expected to be made general availability in early February 2018.

Thanks,

The Cb Response Team

hello,

ok don't get me wrong we tested progressively on the 6.1.2 sensor for about 50-100 pcs [of 1000] and didn't find the issue, but management told us to pull the update for mass deployment [to the 1000 ] and we went with 6.0.3 (that we also tested).

> the tooltip will contain a message indicating that the driver has failed.  We are continuing to work on fully addressing this issue and will provide additional changes in a follow-up maintenance release.

1) mm, tooltip doesn't sound ideal at all - can we set up a CBER query to alert on (preferably a query not an API call, but if the former is not possible the latter will do) . or get CB to publish a threat feed for 'sensor issues' we can alert on for the potential sensor faults like the above.

2) alternatively - can CB list the winevent process driver error load event id and details so we can Winevt SIEM alert on it [we've had the driver not loading issues on another EDR product and that's how we picked it up there]

3) or is this readily identifiable if i export all sensors from the UI to csv.

4) Also can you clarify if you mean driver failed means 'sensor doesn't functions for EDR purposes correctly - e.g. no events or binary or net grabbing' or BSOD again. (that may essentially disqualify that version from testing or deployment for some people unless there is a clear way to bulk identify affected sensors that we may wish to downgrade ).

5) finally is there any more information on specific product or feature incompatibilities and the root cause for this (e.g. incompat with what other sec products and features) ?

re winevt bit:

e.g. for our old edr product we'd get these:

windows log Event ID - 7000

Event source - Service Control Manager

Event type - Classic

Event category name  - System.Errors.Services

Event description - The EcatServiceDriverXXX service failed to start due to the following error:  %%3758161940

You can export to CSV under the sensors tab - select "Export ALL Hosts".  Then look under column AJ for "sensor health message".

But like all health messages, this appears to be fleeting.  As soon as the agent reboots normally, it seems that would be cleared.

thanks. ok, how about via the API?

'fleeting' and reboot are relative terms depending on the environment and users it can mean 'a month'

I still think CB needs to very specifically expand on the above.

I can't believe nobody's answered your question in 9 days.  I've asked this question to support and I'm surprised they can't give it to me, either.

If you're not running Windows 10 with Secure Boot, you can use this 6.0.3.70821 version:  Carbon Black, Inc. Web App - Links

Apparently support has to get the latest 6.0.3 link from engineering... once they get that I'll post it here.

Support has an early-access version of 6.1.3 available.  Contact support directly for the link if you're a candidate or like beta-testing software on your endpoints.

1.  For us, this seems to manifest itself when the user on 6.1.2 clicks on "Shutdown" or "Restart" from their Start menu.  During the Shutdown process, it blue screens.  (The write-up above seems to indicate just during the bootup process). 

2.  Initial testing with the handful of users it had occurred with shows 6.1.3 doesn't blue screen under the same situations.  No new issues seem to have popped up since - let's keep our fingers crossed.

3.  Occasionally the driver does not load.  This is observed under the Sensor's detail page (screenshot) showing a historical view of it happening. 

4.  We're looking at writing a rule, likely with Bit9, to identify if a new memory.dmp file occurs, so we can try and proactively identify users with this issue.  If nothing else, we can help our service desk with identifying bad user experiences in the future.

I wish things like this could be communicated better.  I just happened to stumble on this while looking for something else on the website...after I just deployed 10k 6.1.2 sensors.  We did not see any issues in our testing but now I'm wondering how to proceed and have some questions.

1. Are there any specifics around what ise causing the issue?  Is it a conflict with some other application that we should be aware of? 

2. How likely is this to occur?  Was this just sporadic BSODs are fairly pervasive in some environments? A particular Windows version?

3. If I havent seen any issues so far, should I still plan to revert back to an older version until the 6.1.3 comes out and is tested?  Or should I be ok to stay on 6.1.2 until the new version comes out?

Without much guidance on this it's hard to say what is the best course of action.  Maybe it would be good in cases like this to have our System Engineers reach out directly with this information?  And at least pull these impacted versions from the cloud installations to keep people from pushing them out as well.  Just a couple of thoughts. 

All,

We are planning to make our 6.1.3 windows sensor available for controlled distribution next week. The sensor release will contain improvements to prevent a bug check from occurring should this same issue be observed.

The development team is still working to fully ascertain why this failed driver initialization is occurring. This bugcheck issue is sporadic. This does not impact a particular Windows OS version more so than others. The bug check is typically observed at the time of installation or very shortly thereafter. If no issues have been observed with the 6.1.2 sensor in your environment there is no need to revert back to an older sensor version. We encourage testing on 6.1.3 and future maintenance releases as they become available as we continue to make improvements to address this issue and others.

The Cb Response Team

Any update on 6.1.3 release?

Please see update in original post from today (Feb 8).  Hope this helps.

Any information available for a tentative release date for 6.1.4?

I spoke to our SE a week or so back about this and he said they were shooting for mid March time-frame for release of the 6.1.4 sensor.  I havent heard an update from him since then. 

Article Information
Author:
Creation Date:
‎12-21-2017
Views:
4623
Contributors