Software Update Causes Global IT Outage

Today’s global IT outage affecting Microsoft Windows users was caused by a bug in a CrowdStrike product (Falcon Strike) protection content update. The resulting widespread impact demonstrates an urgent importance for maintaining strong quality assurance (QA) processes before products or their updates are released into production. This is true for all software; however, with cybersecurity products that protect endpoints and require elevated privileges in operating systems (OSs) like Windows and others, even a minor mistake in the release of an update can knock servers offline, in this case signaled by a looping 'blue screen of death' (BSOD).

This is not the first time endpoint security software has crashed operating systems like Windows. In the past, signature or detection updates that are legitimate and critically important, have had bugs that caused them to misidentify a critical component in an operating system as malware, and then attempt to block or quarantine critical OS files. This is how a seemingly minor 'bug' within an update can crashes systems and can cause a domino effect across global systems at organizations who use that product.

CrowdStrike’s issue was more nuanced, but still had the same impact. One of the daily updates to their product’s behavioral protections had a bug that put the Windows OS into a BSOD, and that state would continue through reboots until the computer was booted in recover mode to follow CrowdStrike’s fix. Since many enterprises from global airlines, financial institutions, and health care use the affected product, this protection update bug took out those organizations’ computers.

Early media reports incorrectly stated or implied that the CrowdStrike issue caused a Microsoft Azure outage, but that was a separate and unrelated incident that has now been resolved.

We’ll continue to monitor this issue and will provide updates as needed. Additionally, we cover this in our new episode of The 443 Security Simplified podcast. Here’s a preview.

Implications for WatchGuard Partners

For WatchGuard partners, there is no impact unless they directly sell CrowdStrike products. Partners dealing with affected customers will need to assist in recovering their systems by installing the necessary CrowdStrike fix in safe mode. WatchGuard products remain unaffected by these incidents.