NEWS: The day when a “Security Snafu” sent millions of computers to BSOD
So as you all must have been aware of the huge tech outage that occurred on July 19th,2024 sent millions of computers spiraling to their Blue Screen of DOOM!. All the systems running Microsoft had their screen turn into Blue Screen Of Death(BSOD) after the update and just like that many organization stopped or struggled to continue their work and their employees had their moment of fun. So in this article, I will discuss about who is responsible for it? How did this happen? what can be done in future to avoid these sort of issues?
Who is responsible for this massive outbreak?
An American cybersecurity company popularly known as CrowdStrike Holdings, Inc was the company that pushed an update to systems having their third party software installed ,crowdstrike falcon, to cause all the crashes. So that means only systems having this 3rd party software crashed and it didn't affect any other operating systems or systems that didn't have their software.
According to Wikipedia:
CrowdStrike Holdings, Inc. is an American cybersecurity technology company based in Austin, Texas. It provides cloud workload protection and endpoint security, threat intelligence, and cyberattack response services
Crowdstrike falcon is popularly known as “Endpoint detection and response”(EDR) software. So its like an antivirus that checks your system is compromised or not. They pushed a update to vulnerability scanner but had flaws in their updated code to cause the crashes.
NOW DIVING A LIL BIT MORE TECHNICAL!
Now they run their code at a kernel level on our systems. We gave them those privileges upon installing this software. They run their code as a Kernel mode driver (.sys files). But it makes sense for these antivirus software to be run at a kernel level so that they can monitor our system as a whole.
How can we fix this?
Now this issue can be fixed, you can refer to https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ on how can we fix this issue. Basically, you have to boot into safe mode from your BIOS where it doesn’t install drivers and then manually navigate to the “C-00000291*.sys” file and remove it.
If all these seems a bit too much then there is a second options. Which is to wait patiently until they push another patch update for your system :) .
What exactly happened?
Now that .sys file which got shipped to our system got corrupted somehow. It was all null!.
So all your system got as a update was my math mark (jk I didn’t score 0 in my math test, please don’t tell my mom). So your computer was like ¯\(ツ)/¯ and then BOOM! BSOD.
Now, you might wonder why don’t we just disable the driver . Well here is the thing, any antivirus software that you have installed on your system , while upon powering up your system, starts a boot-start driver. What it does is that , it will tell your system to load up the drivers to start your system with a safe environment. Logically thinking, an antivirus driver is deemed as an safe driver which is needed for a safe environment for our systems. So your .sys file starts at the very boot level along with your OS once you turn on your computer. And that .sys file is full of nulls which lead to your blue-screen at recovery.
What could have been done?
One of my close friend talks about this on his LinkedIn post, please do refer his post https://www.linkedin.com/posts/activity-7220664121506115584-ARNf?utm_source=share&utm_medium=member_desktop. He talks about how important it is for this high end tech companies to do testing of their product before pushing it to the world. It is quite important for people to understand what are these companies capable of doing to our system before blinding accepting all their terms and condition. Of-course an outage like this cant be blamed on the end users but on the company themself.
The irony is that we gave permission to sit at kernel level for a piece of software in our system to protect us only to end up in a ditch by the very thing that was supposed to protect us. This wasn’t a hack but was merely a fault in our code. It could have been worse!.
Interested in knowing more? Watch https://www.youtube.com/watch?v=pCxvyIx922A by Low Level Learning, to understand how this was a null pointer d reference problem.
My references:
