Browse your product documentation including release notes and installers
I wanted to provide an update on our support for the new 10.13.2 macOS. Apple has made changes in macOS 10.13.2 that has impacted Cb Response kernel extensions. Support for this new macOS will be provided in upcoming Cb Response macOS sensor releases, versions 5.2.13 and 6.1.3. We plan to provide a controlled distribution release of these sensors for on-prem customers on January 16th, 2018 with general availability slated for January 29th, 2018. More information for participating in our controlled distribution for these sensors will be provided in early January.
Product Owner - Cb Response
This is not an acceptable timeline, given the fact that many companies, including ours, require both Mac and Windows OS patches to be installed within 14 days of release (InfoSec policy). How do we stay compliant with OS patching AND CB Response sensor install support?
Yes, I'm very aware of the CBR Platform Support Policy and your SLA's are not sufficient to support our environment. Furthermore, the major release of Mac OS 10.13 did not even meet your own 45-day SLA. High Sierra was release on Sept 26, and you came out with a GA sensor on Nov 20. That's 56 days!!! A full 11 days/24% beyond your documented SLA.
Also, any documented SLA to me means "in extreme cases, is the longest you should wait, and we expect to hit the mark sooner than that on average". Your SLA timeframe should NOT be the average. It should be the maximum.
Again, I go back to our requirement for patching the OS. Internally, we have 14 days to hit this mark. That is our internal SLA, which means we're usually done earlier. CB Response needs to do even better than that, like some of your competitors. CB is making it difficult for us to keep you as a vendor in good standing.
As disco.boy mentions - pointing to your own SLA's, which you're way beyond at this point, is pretty embarrassing. Especially when this post stating full release on 1/29/18 represents almost a full doubling of your 30 day "target" for point releases. I guess, our biggest issue is simply that these releases are usually available from Apple in the developer program for upwards of 30 days before release (it obviously varies greatly...) so having to have an additional 30 days beyond that is hard to swallow. This release cadence is not likely to get slower, so we really need CarbonBlack to keep up with the two major OS's (Windows and macOS). Having to hold OS updates that fix security issues so we can wait for our security software to support it is... not a great position to be in.
(thanks for pointing that out... I didn't even see the Jan 29 GA date until you pointed it out. )
Like mbezzo said, you're doubling your own 30-day SLA to almost 60 days.
I'm at a loss for words.
Unacceptable. A patch published on 6 December not being supported until 29 January is not a good security practice.
When combined with a 10x performance decrease of the hash banning functionality in the 6.1.2 sensor that I reported in October 2017, I really wonder about your OSX support priorities. I can't recommend continuing with Carbon Black if this trend continues.
Enjoying that the SLA post was deleted. Hope that means they will rethink their support schedule.
First off, I want to thank you for voicing your opinions and concerns here, and confirm that we are listening and we hear you.
We do understand how our customers adopt these new OS releases and the importance of having support very soon after they become GA, maybe even before during the beta process. A while back we made changes to our process which allowed us to deliver support quickly after an OS update was released. Due to a number of factors we have unfortunately slide in the wrong direction, and we know we need to address it. And this is what we are in the process of doing. You should expect to hear more from us in the near future as to adjusted SLA's and improved processes to ensure we meet them.
While the High Sierra release provided us some unexpected challenges, we have an obligation to you, our customers, to deliver support in a reasonable time frame. In this case, we have not, and for that we apologize.
You have our commitment to improving this process and reducing SLAs to ensure you are able to keep your systems current and secure.
I appreciate the response, however we need more information immediately on how you plan on addressing this issue. Apologies and vague promises are not what your customers are asking for - we are asking for action.
We can no longer consider Carbon Black a security tool if it prevents are ability to maintain patch releases. There are at least eight kernel exploits patched with 10.13.2.
Apple has been on a yearly major upgrade cadence for almost seven years and the minor cadence has not changed. This shouldn't be a surprise to your company and your internal process comes across as misguided, if not firmly in opposition to your company motto.
Thanks for the fast response mbilancieri! I'm encouraged that you recognize 45 days isn't where your customers want you to be aiming for.
Looking forward to your announcement.
There was a comment that was deleted above, but it mentioned that it would be hard for other security vendors to have a quick turnaround time for support after a new OS X release. I just chatted with CrowdStrike and they said they usually have a release of their Falcon sensor a day or two after a new major OS X version is released. They didn't have to make any changes for 10.13.2 support, but when High Sierra (10.13) was released they had released a supported sensor for it on 09/20 (before High Sierra release), and confirmed support to their customers the day after release on 09/26. That is what I would like to see.
Please don’t let CrowdStrike and other Vendors show you how it’s done. Be the forefront and ahead of the game. I’m not a coder and stricly an end user, but I chose Carbon Black because I felt the coding and support should be top notch. Please step up.
Apple have the special program for early access to future software.
"Developer Beta" available long before release.
The point for this is to help all the software developers be inline with their product release.
for example macOS High Sierra Developer Beta was releases in mid-summer.
I would suggest Carbon Black to change your new software support model from reactive to proactive.
Reactive is not the word I would like to use for my security product.
Proactive sounds much better.
With a growing MacOS population, deployment of Protection and Response was something that we were likely going to have in our 2018 roadmap. These issues with SLAs may have us looking in a different direction. I could understand if the adherence to SLAs was trending in the right direction, but this is a big concern having to go back to to teams saying we can't patch (or may have to remove the products all together until a supported release is available).
This is not an acceptable timeframe. Our macOS population is small but its a high-risk population - pretty much everybody who's job title starts with a C and they are easily discovered for purposes of phishing etc. Just because the sensor count is small does not imply this is less important, the reverse is true. Windows systems have a much slower upgrade cycle than Macs so I can wait for support for the latest build of Win10 for 60 days but macOS support has to be available *prior* to the public release of the patch from Apple. Your SLA's that are not being met today need changing - please target a major revamp of this process urgently.
We have had to pull CB from all Macs at this point which does not make me happy at all, not least of which is that I'm paying for something I cannot use.
We're a 93% Mac shop here - so definitely not a small part of our population.
I think your point on being prior to the public release is a good one.
Ideally, their policy would change to having a controlled distribution agent ready 14 days after a Apple beta being published, with a general availability CB agent ready within 48 hours after Apple's release.
Not an acceptable time frame here either, granted we have only about 50 MACs across our population but they are all being used by highly visible and important folks who will not tolerate a crash of a MAC because an agent we are using so badly supports the OS and usually results in a complete loss of the installation. At the very minimum the agent should deny the upgrade from occurring when an attempt is made to upgrade to an unsupported GA.
CB should be using the Beta Program from Apple to build their tools and agents then have within 48 hours of a GA a release of a supported agent.
In follow up to my post above, I've posted a blog that provides transparency into the process and timeline for certifying High Sierra. It also outlines changes we are planning to implement to help reduce our SLAs for support of new OS. Cb Response macOS High Sierra Support Process
Your comments and feedback are, of course, always welcomed.
Thanks Michael I appreciate that it appears CB has been reading and taking feedback. The timeline of course shows that the agent only had a two week lifespan where support was released, then immediately obsoleted by the next apple release :-). Definitely looking forward to having zero day support for macOS.
Heads up that developers already have beta copies of OSX 10.13.3 expected to have general availability released at the end of January 2018. Any commitment date for CB Response sensor supporting 10.13.3?
User error resulted in deletion of previous post with the following questions. Apologies for that. Responses below.
1. Why didn't CB test on the Golden Master build beginning September 14th? This would have given additional lead time and was functionally similar to the GA released September 25th.
We did get the GM candidate in and started testing on September 17. We just didn't list all releases in the table above to help simplify.
2. Why didn't CB release a beta throughout this time to allow customers to give feedback? Chances are the security vulnerability would have been found between June and September.
A CD release was made available on October 16. We will be looking to provide earlier releases in the future for customers to begin use in their labs.
3. On December 11th, Apple released 10.13.3 beta 1. Has testing began on this version?
10.13.3 is in house and testing will begin this week
4. Should we assume based on your September 27th GA testing, that your validation on December 20th will only take a day?
December 20 is start of our internal deployment. QA testing is complete at that point. We are assessing risks with delivering CD to customers prior to end of our internal deployment.
5. Should we also assume a January 8th-12th release window for 10.13.2 support?
6.1.3 CD for 10.13.2 support is currently planned for Jan 16. GA is planned for Jan 29. 5.2.13 is expected shortly after that. As mentioned, we are assessing risks of delivering CD sooner.
Thanks for the update and glad my questions weren't just blindly silenced.
So it dawns on me that the problems CB has been having with supporting OSX 10.13.2/10.13.3 are related to the Intel KPTI bug.
macOS 10.13.3 fixes the Intel KPTI issue | Hacker News
There were significant changes to the OSX kernel as a result of that bug.
There were significant kernel changes related to KPTI patching in 10.13.2 which caused a side-effect that resulted in the incompatibility with CbR. At this time, we are unable to confirm this theory with 100% certainty until Apple publishes more information. While the incompatibility between CbR and 10.13.2 itself is well understood, the side-effect that caused it is, in our estimation, most likely linked to that fix.
Today our test group received a new Beta OS X release and encountered a new Kernel Panic on the latest OS X 10.12.6 beta build 16G1210 using CarbonBlack Response sensor v18.104.22.168.2.
I understand CarbonBlack receives these OSX Beta builds directly so you are likely already aware of this new Kernel Panic which we are seeing on OSX 10.12.6 beta build 16G1210 with the latest CB Response sensor.
We're also seeing this with the new version of Response (v22.214.171.124.2) as well on 10.12.6 beta build 16G1210. Defense 126.96.36.199 doesn't seem to cause issues, but we're seeing kernel panics related to cbsystemproxy. Are the dates in this original post still relevant?
New community post from CB Staff about 10.12.6 Beta Kernel Panic
[Cb Response] Important Notice Related to Latest MacOS 10.11 and 10.12 Betas
Thank you for your continued patience as we work to provide you with a mac sensor to support the 10.13.2 macOS. Internal testing confirmed that the recent 10.11 and 10.12 security updates cause Kernel Panic issues with the previous 6.1.3 and 5.2.13 sensors provided for CD release. We are in the process of updating these sensors to contain support for these security updates as well before making these sensors generally available. As such, we are delaying the Jan 29th date for generally availability but plan to make these sensors available in early February. More information around updated timelines will be provided to this post as it becomes available.