A faulty software update from CrowdStrike Holdings Inc., a major cybersecurity provider, has disrupted the operations of hospitals, financial institutions and numerous other organizations around the world.

The incident has been described as one of the largest-ever information technology outages. Experts are warning that some companies could take days to recover their systems fully.

CrowdStrike customers began reporting technical issues en masse early today. The problem affects the Windows version of Falcon, the company’s flagship endpoint protection platform for securing employee computers and other enterprise devices. CrowdStrike’s faulty update caused a malfunction in the Falcon Sensor, a component of the platform that runs locally on the user’s device and scans it for malware.

The issue manifested in the form of a boot loop on CrowdStrike customers’ Windows devices. The bug, which is also known as the blue screen of death, prevents users from booting their computers. CrowdStrike said that the malfunction doesn’t affect Macs, Linux machines and Windows devices booted after 1:27 a.m. EDT. Devices running certain legacy versions of Microsoft Corp.’s operating system, namely Windows 7 and Windows Server 2008 R2, avoided the boot loop as well.

CrowdStrike has published instructions for how to fix affected machines. First, administrators must boot malfunctioning devices in Safe Mode, which loads Windows such that only some of the operating system’s features are activated. From there, they must delete a file called “C-00000291*.sys” that can be found in CrowdStrike’s file system folder.

Removing the file in question is relatively simple. However, the process must be done manually, which means administrators can only fix affected devices one by one rather than in bulk via scripts. As a result, cybersecurity experts believe that some companies with a large number of Windows computers may take days to fully restore their IT operations.

There are also other complicating factors, notably Windows’ built-in BitLocker data encryption feature. “Windows Computers and Servers with BitLocker enabled, BitLocker Recovery keys that are stored in Active Directory and any Active Directory Domain Controller that was impacted and can’t be recovered, may be unrecoverable,” said Andrew Costis, the chapter lead of the adversary research team at AttackIQ Inc., a venture-backed cybersecurity provider.

The fallout from the faulty Falcon update is being felt worldwide. In the US, airlines canceled about 3,000 flights and delayed tens of thousands more. In the UK, some healthcare organizations are struggling to process prescriptions. The outage has also impacted the London Stock Exchange, TV channels, taxi services, supermarkets and numerous other organizations worldwide.

CrowdStrike says its software is used by more than 29,000 customers worldwide, including over half the Fortune 500. The company’s stock plunged more than 13% in trading today on the outage. However, some observers argue that the long-term impact on the company’s market position will be limited. 

“Even for outages lasting under 24 hours, loss of business at large organizations can be in the millions of dollars,” said Third Bridge analyst Jordan Berger. “Despite this and despite any associated impact to CrowdStrike’s brand, the most significant aspect of today’s CrowdStrike outage may be the fact that the outage was in fact not tied to any security incident or breach, and as such the company’s security track record remains untarnished for now.”

As for competitive concerns, Berger added, “The degree to which competitors like Palo Alto Networks and SentinelOne stand to benefit is unclear, especially considering the amount of effort required to replace a large security provider.”

But if CrowdStrike’s rivals may not benefit from the incident, hackers might. Cybersecurity experts have cautioned that cybercriminals could take advantage of the outage to target affected organizations with fake technical support emails containing malware.

“This disruption creates a fertile ground for exploitation, as attackers prey on the vulnerability of users seeking solutions,” explained SecurityScorecard Inc. Chief Executive Aleksandr Yampolskiy. “The timing of this event and how public it is happens to be exactly what attackers look for to craft targeted attacks. Threat actors may use social engineering tactics to disguise malware as legitimate restoration tools to gain unauthorized access to systems.”

The cybersecurity ramifications are particularly significant given that organizations could face similar outages, and the associated hacking risks, in the future. “Outages are not a problem we’re going to completely solve,” said Spencer Kimball, co-founder and CEO of database startup Cockroach Labs Inc. “Cloud environments are only growing more complex and interconnected. This complexity at scale will continue to increase risk, particularly for businesses that are still in the initial stages of cloud adoption. Continuous monitoring and alerting are essential to detect and address issues before they escalate.”

The release of CrowdStrike’s faulty update coincided with a smaller, but still widely felt, outage in Microsoft’s Azure cloud platform. It affected the company’s US Central cluster of cloud data centers. The issue began just before 6 p.m. EDT Thursday and took more than three hours to resolve. 

According to Microsoft, the outage was caused by a configuration error that blocked the movement of data between some storage systems and virtual machines. As a result, a subset of the services offered in Azure’s US Central data center cluster became unavailable. Separately, some organizations reported difficulty accessing the Microsoft 365 suite of productivity applications. 

Photo: CrowdStrike