Carbon Black Cloud: Unable to upgrade or install due to existing system extension (macOS)

[#1]

Environment

• Carbon Black Cloud Sensor: 3.5.1.x and Higher
• Apple macOS: 10.15 and Higher (Catalina

Symptoms

• Install of 3.5.1.x or higher Sensor fails
• Error message displayed in preinstall log indicates failure to unload installed System Extension

  failed to uninstall system extension
  program terminated with error code: 4096

• Checking on Systemextensionsctl shows Carbon Black installed

  sudo Systemextensionsctl list
  
  --- com.apple.system_extension.endpoint_security
  enabled active teamID bundleID (version) name [state]
  * * 7AGZNQ2S2T com.vmware.carbonblack.cloud.se-agent.extension (3.5.2fc76/3.5.2fc76) com.vmware.carbonblack.cloud.se-agent.extension [activated enabled]

Cause

Previously installed System Extension is unable to be uninstalled without first disabling System Integrity Protection (SIP)
NOTE: Using newly added scripts this can be worked around without disabling SIP. This can be done using Resolution 1 below. Resolution 2 can still be used if Resolution 1 cannot be used. 

Resolution

• This is the optimal solution using scripts that don't require SIP to be disabled. The scripts can be found here.
  1. Download and extract the "VMware CBC Mitigation bundle.zip" to only the affected endpoints. Note: Running this tool on a healthy endpoint is not intended and will result in undefined behavior.
  2. As a root user, execute the driver_remediation.sh script. If a clean uninstall of the endpoint is desired, run with the -u/--uninstall flag. Otherwise, default behavior is to clean up the old system extension to allow for a sensor upgrade to take place afterwards.
  3. The script will unload the old system extension. Due to OS requirements, there may be a popup requesting user permission for the unload. User credentials should be inputted, and the script will continue.
  4. After a successful execution of the script, the old system extension will be in the [Terminated waiting to uninstall on reboot] state. A reboot is not required, and sensor upgrade or uninstall can immediately be re-attempted. 
• Previous Solution requiring SIP to be disabled. Does not need to be used if the Resolution above has been followed. 
  1. Check Systemextensionsctl for com.vmware.carbonblack.cloud.se-agent.extension

     sudo Systemextensionsctl list | grep carbonblack

  2. Disable System Integrity Protection (SIP)
  3. Once rebooted in normal mode with SIP disabled, check Systemextensionsctl for com.vmware.carbonblack.cloud.se-agent.extension again

     sudo Systemextensionsctl list | grep carbonblack

  4. Manually uninstall system extension

     sudo systemextensionsctl uninstall 7AGZNQ2S2T com.vmware.carbonblack.cloud.se-agent.extension

  5. Verify com.vmware.carbonblack.cloud.se-agent.extension is no longer present

     sudo Systemextensionsctl list | grep carbonblack

  6. Attempt installation of desired Sensor version, collecting most recent /tmp/preinstall-<Timestamp>.log
  7. Enable System Integrity Protection (SIP)

Additional Notes

• Version will display with 'fc' included (i.e., 3.5.1fc23, 3.5.1fc31, or 3.5.2fc76) and is normal
• If the system is on macOS12+, we have success stories resolving the error by adding the "RemoveableSystemExtensions" setting in the System Extensions payload specifically targeting com.vmware.carbonblack.cloud.se-agent.extension.
• If installation still fails, run through step 5 above, outputting to a file with the name of the device and please open a case with Carbon Black Technical Support

Related Content

[Carbon Black Cloud] macOS Big Sur and Later Documentation List (v3.5.1)
https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protectio...
Carbon Black Cloud: Script tool for unloading installed System Extension (not requiring SIP disablem...