App Control: Agents Out of Date due to Upgrade Waiting or Upgrade Scheduled Status

[#1]

Environment

• App Control Server: All Supported Versions
• App Control Agent: All Supported Versions

Symptoms

• Agent Policy Status shows "Approvals out of Date"
• Computer Details shows CL version behind the Current server CL version
• Agent upgrades have stalled in an "Upgrade Waiting" or "Upgrade Scheduled" status

Cause

There could be multiple causes why agent upgrades could stall, but primarily it's related to low system resources available on the application server

Resolution

I.  Increase the Private Memory in IIS
II. Too many Agent Upgrade Requests are pending or prioritized, and the requests need to be cleared.

• From the console per agent:

1. For each agent's stuck in "Upgrade Waiting" or "Upgrade Scheduled"  > Go to the computer details page
2. On the right under "Action" > Select "Remove Agent Upgrade Request"

• From SQL for all agents:

1. Login to the application server as the Carbon Black Service Account.
2. Stop the Carbon Black App Control Server and Reporter services.
3. Launch SQL Server Management Studio and execute the following script:

   use das  
   set nocount on    
   -- --------------------------------------------------------------------------------------------  
   -- Description:   
   -- 1. Remove prioritization of upgrades; and expire selected hosts session  
   -- --------------------------------------------------------------------------------------------  
   -- Step 1: declare variables  
   	declare @cnt int;  
   	declare @max int;  
   	declare @tbl table (id int identity, host_id int)    
   			
   -- Step 2: get hosts to remove prioritization of upgrades
   	INSERT INTO @tbl (host_id)  
   	SELECT host_id FROM dbo.hostmain h WITH (NOLOCK)
   	WHERE deleted = 0 AND upgrade_state < 0 -- forced upgrade requested through the menu (0x80000000)
   
   	SELECT @max = max(id), @cnt= 1 from @tbl ;    
   			
   -- Step 3: loop; each host (and expiring the host)  
   	while (@cnt <= @max)   
   		begin    
   			declare @hostID int;    
   			select @hostID = host_id from @tbl  where id = @cnt;        
   			
   			print convert(varchar(3), @cnt) + ' host id: ' + convert(varchar(30), @hostID); 
   			
   			-- Change Upgrade State
   			UPDATE dbo.hostmain 
   			SET upgrade_state=16 --Not requested
   			WHERE host_id = @hostID
   						
   			-- Expire host    
   			EXEC dbo.ExpireHostSession @hostID;     
   			
   			-- Update counter    
   			set @cnt = @cnt + 1;       
   		end

4. Start the Server & Reporter services.

Additional Notes

In some instances simply restarting the App Control services will allow some Agents to begin upgrading, however this may only be a temporary workaround.

Related Content

App Control: Intermittent Slow Console Performance
App Control : SQL Server Consumes High Memory in a Single-Tier Environment
App Control: Approvals Out of Date on Agents & High CPU on App Control Server