App Control: Web Console Unavailable Due to IIS 500/503 Errors

[#1]

Environment

• App Control Server: All Supported Versions
• Microsoft Windows Server: All Supported Versions
• Microsoft IIS: All Supported Versions

Symptoms

• Issues start after triggering large amount of agent upgrades, refreshing policies on multiple agents at once or after recent server certificate update
• Web console pages fail to load with errors:

  500 Internal server error
  503 Service is unavailable

• IIS logs show GET requests with 500/503 status:

  2022-11-15 03:43:12 10.11.12.13 GET /hostpkg/pkg.php pkg=configlistwithdeleteditems.xml.egk 443 - 10.22.33.44 Bit9+Parity - 500 0 64 81986
  2022-11-15 03:43:12 10.11.12.13 GET /hostpkg/pkg.php pkg=ParityHostAgent.msi 443 - 10.22.33.44 Bit9+Parity - 500 0 64 41986
  2022-11-15 03:43:12 10.11.12.13 GET /hostpkg/pkg.php pkg=TrustedCertList.pem 443 - 10.22.33.44 Bit9+Parity - 503 4 64 41986
  

• Windows event logs:

  Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures.

Cause

• IIS "DefaultAppPool" private memory is set too low and cannot handle the incoming download requests. This will cause the Default App Pool to crash and render the App Control console unavailable
• FastCGI Queue Length is not sufficient to serve all download requests

Resolution

1. Upgrade to server version 8.10.2+ where there are several notable changes to IIS configurations, including:
   • A new app pool (AppCDownloads) that is dedicated to agent downloads
   • Recycling > PrivateMemoryLimit is now set to 0
   • TrustedCertList.Pem and Keychain.json now take advantage of direct IIS downloads
2. If the issue still persist, please grab high debug server logs and IIS logs

Additional Notes

If unable to upgrade to 8.10.2, the following steps can be taken to manually apply the changes:

1. Open IIS manager (inetmgr) > Expand the web server name in the left menu options
2. From the left menu right-click on "Application Pools" > right-click the "DefaultAppPool" > Advanced Setting
3. In the Advanced Settings scroll down and update the following:
   • Process Model > Identity: Use the App Control Service Account
   • Rapid-Fail Protection > Maximum Failures: 10
   • Recycling > PrivateMemoryLimit: 0
4. Select the web server name in the left menu options
5. Select "FastCGI" Settings
6. Edit "C:\Program Files (x86)\Bit9\Parity Console\php\php-cgi.exe"
7. Verify and update the following:

• Instance MaxRequests: 10000
• Max Instances: 0
• Queue Length: 10000

8. Restart the IIS Web Server from CMD: iisreset