In this daily security byte with WatchGuard CSO, Corey Nachreiner, he explains the recent Global IT outage cause by a CrowdStrike update. We also follow-up on RockYou and the RockYou2024 data dump of 10 billion records.
View Transcript
Corey Nachreiner 0:00
Hey everyone. Welcome back to the 443 security simplified podcast. I'm your guest host this week, Corey needs a drink and a workout Nachreiner. And joining me today is
Trevor Collins 0:13
Trevor Collins. Thanks for having,
Corey Nachreiner 0:16
of course I'm happy you're here. With Marc out this week, I am both a guest host and Trevor is joining me. But with today's episode, we have more security news than we can handle. So we need to dive right in. And it includes late late breaking Friday emergencies that down planes for at least an hour. And on top of that we will demystify the 10 billion password RockYou 2024 elite talk about a shady spyware product that suffered their own breach, and also update you on the SEC case against SolarWinds. With that, let's fly on in assuming the planes are taking off again. I will start with a story about rock Q. Do you remember the rock you breach which I think or leak which literally happened I think in 22,009 Trevor wasn't that long ago?
Trevor Collins 1:11
Yep. A while ago, I actually do remember it a little bit.
Corey Nachreiner 1:15
Yeah, yeah. So if the listeners haven't heard, as you probably know, there's always various types of breaches that result in databases and records showing up on the underground. Usually they start being privately sold before they show to the public. But over time, they sometimes get leaked and back to really start the history of rock you back in 2019. Some threat actor or gray hat on a forum leaked something that was literally called Rock u dot txt. And this rock u dot txt was basically a file full of 14 million unique passwords. And over the past years, decade, really decade that has grown, there's been additional RockYou releases various files, and over time, it's aggregated to about 8.4 billion passwords. In any case, in the past few weeks, there has been a new update to RockYou called RockYou 2024, which has a new file. And of course, this release is grabbing a lot of headlines talking about the RockYou data leak, which it's grabbing headlines, because apparently it's 10 billion records this time. However, I don't think this one is as big a deal is something Do you have any thoughts so far? Trevor?
Trevor Collins 2:35
Yeah, I mean, so this touch reminded me of a leak in the past and every every couple of years. And well, let me let me give you a little bit of background,
Corey Nachreiner 2:46
are you referring to my lab, by the way, but I so the leak,
Trevor Collins 2:50
the leak I'm specifically referring to is actually the clear voice surveys, which was about three years ago, it was released as this clear text password list. And then after reviewing it, and after looking at it, we found out that it really was just a bunch of garbage. There was a few like passwords in it, but it was all the same passwords that are in RockYou. And, and so I actually have a little bit of experience in this because for sure
Corey Nachreiner 3:22
you run our you ran our dark web database for a long time
Trevor Collins 3:26
for a bit for a bit not not anymore. But I in the past, I was looking at a lot of breaches. And and every once a while you would see somebody releasing a breach that had a whole bunch of passwords in it, claiming that it was a legitimate password release. Well come to find out. A lot of times what they'll do is they'll download a download a breach, go through all the passwords, and then name it something else and just release it with just the passwords itself.
Corey Nachreiner 3:57
Or even act as the aggregator right. I mean, if folks listening haven't heard Moab stands for Mother of all breaches. And it was another huge headline that came out because 26 billion records have been leaked passwords and etc, etc. It turns out that was primarily an aggregate of a decade worth of other leaks over time just someone repackaging it kind of in the way you say,
Trevor Collins 4:25
yep, between collections and breaches. It was all really always difficult to make sure that they were legitimate breaches and not somebody trying to get you know, street cred on that, you know, Dark Web Forms. So, and
Corey Nachreiner 4:39
it turns out you're right by the way the people viewing on YouTube can see that we're showing a reference from my net watch man who is another organization in person that actually follows breaches. They even put a D hyping Reddit post on our cybersecurity, but like us like you did with our data breach database for a while my net watch man actually has done a lot of breach analysis. And it turns out that this RockYou 2024 release is not that consequential in my opinion, it does add about 1.5 billion records as in passwords to the previous 2021 file. But the vast majority of it is the same, you know, starting with the first 14 million up to 8.4 billion in 2014. It's just a continuation of the things that have happened. One other thing with this breach is in are with this leak, and a lot of these database leaks, they actually have what's called credential pairs, meaning they they will not only have a hash or a password, but they will have a username to go with it or an email address that acts as the username rocku is very different in that it only has the password side and not always the clear text password. So by only having the password, it's a lot less valuable than having the full credential pair knowing a potential email address or username that might be associated with a password is much better. But anyways, according to my net Watchmen, only 1.5 billion things that are new, it is a mix of clear text passwords along with hashed or encrypted passwords. But there's also some just, you know, strings and word list records that are pretty inconsequential in the data set as well. According to my net watchman, they kind of say 84% of it is older than three is three or more years old. So it's not new passwords. And rat this 10 billion grabs the headlines, you might think, Oh, wow, 10 billion new passwords being leaked is a big deal. But most of these seem to be quite a bit old. And by the way, 10% of them are still kind of encrypted or hashed. I suspect, you know, when a actual real password leak comes out, they are all typically hashed. But the short character ones tend to get cracked pretty quickly. And really, over the years, people have probably been working on hashes and cracked ones up to say 12 or 14 characters, but I bet you the 10% of encrypted ones might just be very long passwords, so they don't have much value.
Trevor Collins 7:24
He had one one thing to add to that. And just to start customers know, the RockYou wordless was very powerful list. And the reason it was so well known and well used, is because you could use that password list with a couple of other rules and different programs that you could call, like a program called Hashcat, that anybody could really run within, you know, a day of trying to search on the internet with and come up with about 70 90% of passwords from any breach. Now it's becoming a little bit harder nowadays, but those that that combination of the rocky WORD LIST allowed a lot of breaches to be taken from the hashes and actually produced passwords out of that to that, then compromise that you could then proceed to a compromised account for if you were a hacker.
Corey Nachreiner 8:22
It's like you're reading my mind, Trevor, because I was about to ask you my first question, which is what if anything, is the value of this rock you leak. And I agree with you, that's it. You know, other breaches that have credential pairs might have higher value, because you literally can go and try an existing credential pair and not even not only potentially gain access for that account directly and decide it was leaked for but also see if that email address is associated with other accounts using the same password. So credential pairs are still way more valuable. But to your point, just exactly what Trevor said, if you're brute forcing, true brute forcing is trying random things. And by the time you get to try and random things, when you have really long passwords, it starts to take a real long time. So most brute forcing, you know applications like hash cat and the many others really get value from having a word list that they can try these words before they actually truly start brute forcing character by character. And like you say they have cool rules where they won't just try a word alone, but they can concoct innate words together they can add leet speak to it. And if you already have a list of known passwords, it's it's just a it speeds up brute forcing and prevents you from literally having to do one at a time. So you know, I guess we both think this isn't the end of the world. The fact that there might be 1.5 billion new passwords might make a word list a little bit better, but is there anything else you think our organization or our customers or partners can do to protect themselves from this? Well,
Trevor Collins 10:02
I think there are a few things that we can certainly do. And keeping your passwords safe. There's certainly things like multi factor authentication that should absolutely be used for any corporate environment. And when you're creating a password, you should usually think of it as passphrase instead of just a password. So, you know, instead of trying to figure out how many exclamation points and the numbers at the end of your, you know, dog's name is the best way to do it. Think of it as
Corey Nachreiner 10:35
isn't the best way to do it. It's not sarcasm, do it.
Trevor Collins 10:40
Yeah. So that's not the best way to do it. But think, think about, you know, what, you know, maybe some random part of your history something that isn't, you know, your mother's maiden name or your dog's name, but could be
Corey Nachreiner 10:53
a sentence or a longer phrase that actually is a combination of things together? Absolutely.
Trevor Collins 10:58
That and don't, don't you reuse your passwords. And then when you can use Password Manager,
Corey Nachreiner 11:08
as well, I might even skip directly to MFA and a password manager. I think you're absolutely right. You always should have MFA. Because if any credential leaks, even if you have a super, super strong password, there are ways credentials can leak through phishing, or even kind of man in the middle or adversary in the middle attacks where, you know, they might actually fish it from from something else or get malware on your computer that can pull it from memory. So you definitely need MFA to protect against leaked passwords. I agree with you, when you have to write the one password that you enter in, you should use a passphrase. And for me, I just use the sentence. I know that seems long. But the sentence with spaces and punctuation is usually long enough brute forcing won't get it. And it's relatively easy to remember. But the key thing is the real advice is your password should be as random and as long as possible and different everywhere. And I don't believe a human can do that. So our main advice is besides your Master Password, you should be using a password manager with a master password. And I would just recommend using the generate password, you know, set it to 24 to 32 characters in length press generate, it will give you this crazy password that you're never going to remember. But you don't need to because the password manager remembers it for you. Cool. So in any case, interesting to see rocky folks continuing to update their password word list. But you shouldn't consider this as dangerous is like a credential pair leak. And it is really a lot of old information. Moving on to our next story, it actually involves SolarWinds. I'm not sure if you remember the SolarWinds breach. But let me give a quick update. You're not update summary for our audience, although I'm sure they remember when we covered it long ago. SolarWinds is of course a very popular company that makes it services one of their biggest products is called Orion, which is a network and it monitoring solution. So it helps you monitor network stuff. In any case quite a while ago that that company their company was breached, and attackers got so much access that they got access to build servers and source codes. And during that time, during a long period of time, they lived in solar winds network, they adjusted the source code and build process to actually add a Trojan to the installer package of that this very popular Orion package. Of course, this was before solar winds knew about it. So this legitimate Orion installer went out to many customers, including many Fortune five hundreds installed malware on their computer, a Trojan that allowed the attacker that then get into many other big networks, and it was quite a big deal a good example of a digital supply chain attack. Now more recently, in October 2023, the Securities and Exchange Commission or sec, actually charged solar winds and their seaso. Their their chief information security officer Tim Brown, for fraud and internal control failures relating to these known cybersecurity risk and vulnerability at a high level, you know, the complaint basically allege it alleged that between October 2018 through December 2020, which was during the time solar winds was doing their initial public offering to go on the stock market. That long story short, this sunburst this attack this breach happened and solar winds did not or failed to disclose this as a known risk as part of its filings with the SEC to go public. So this of course could mislead investors into investing in this particular company. because this is critical information that you really need to give to your investors to give them the information they need to know if they're going to invest in your company. However, the one question is during this entire public offering, did solar winds know? So the news story, which I'll pull up real quick right now, let me get it up for our audience is actually a judge has thrown out a large chunk of of these SEC charges against solar winds. So kind of good news for solar winds. Basically, on July 18, a judge dismissed a major portion of this saying basically, that solar winds and the seaso brown cannot be held liable for statements and filings made after the breach of the company. So one of the things that the SEC was using for evidence are both internal and external statements that basically allude to security wasn't perfect at the company. But these were all things after solar winds learned about the breach not before. However, on the other hand, the judge did say they can proceed with charges for any misrepresentation the company made about their cybersecurity posture leading up to the attack. So in short, if companies, I kind of think this is a good thing, Trevor, because when a company does learn about a breach, you know, it's one thing you would like to see them do is transparently disclose that to authorities to the public, and even to the cybersecurity community to kind of share details about what happened mostly to help train the, you know, help inform the community on how to avoid this. And if companies are held liable for information they share after a security incident, that would probably kind of disincentive or curtail customers from wanting to transparently disclose things. We'll talk about what the judge can says the SEC can continue to do though, but what do you think about this ruling so far? Drever?
Trevor Collins 17:11
Yeah, I I liked the ruling for for the most part, I do have some some caveats about that. And I think that it in when it comes to the security, in principle is that it is much better to be transparent on your security. And the only way to allow that to happen is by allowing people to make mistakes in security. So if everybody gets punished into an find into oblivion, then nobody is going to be transparent about it. But however, there was a little bit of shenanigans that seemed like to be going on with, with some of the higher ups in the SolarWinds business where they were selling shares and things like that were happening, and there's more like, you know, financial side of it. But it seems like maybe it wasn't really, either they didn't have enough information, or it wasn't big enough for them to really care about in the whole scheme of things. What the judge seemed to be really caring about is misrepresentations that were happening me for the incident that was causing problems.
Corey Nachreiner 18:25
Exactly. And that gets to the point of solar winds actually is not off the hook completely. The judge did also say that the SEC can proceed with any charges about the cyber about the actual cybersecurity claims the company made before the breach. So for instance, if the company kind of posed as having great security while they were doing the public offering, while there were internal things where they were saying differently, the SEC can still charge them for that. So I do think besides potential, you know, when people were selling off share shenanigans, I do know the CSO was known for kind of publicly talking about how the, you know, solar winds had good security. But internally, the seaso did know that, you know, they weren't meeting all of this requirements or made statements about being behind on patches. So it is an interesting case. Also, I think it's I'm a CISO myself, although I guess I'm called the CSO here at WatchGuard but I'm under the gun. In some cases, if WatchGuard ever did anything that negligent in this way. Do you think this puts helps CISOs or puts them more under the gun?
Trevor Collins 19:39
I think like I was mentioning earlier, I think it does help the CISOs because it allows us to be more transparent. And it's not an we're not trying to hide our security. You know, this isn't. I haven't worked for a publicly traded company in the security It sent as a security position. But you know, what seems to be what was happening in solar wind was that they were claiming that they were having a program that this is kind of what what a lot of what I have seen, people are thinking that they claim to have something, set up some type of completely organized security plan, security setting. And they were just like in the beginning planning stages.
Corey Nachreiner 20:30
So like they were doing a framework or a standard, like NIST, but they they weren't complete on the maturity model. They were just actually starting it.
Trevor Collins 20:38
Correct. Correct. It was it was on a piece of paper, but nothing was actually, you know, being done about it. So So I think but back to your question, I think that this does help us, I think that does help the CISOs. And, and helps to us to be a little bit more transparent. So we're not worried about, you know, somebody.
Corey Nachreiner 21:00
I do too. And frankly, I actually so far don't think CISOs are too far under the gun, as long as they're good CISOs. Like the I want to make sure there's no laws that put CISOs unfairly under the gun, but there have been negligent like I won't, maybe you were being kind because of who we are. And I am at WatchGuard we really do try our best to be good at for security, but the Uber See, so there have been CISOs that have literally hid breaches when they knew about the breach and paid hackers to not disclose it while they were actively hiding it. So I think the difference is between we want to protect CISOs that are trying to do their job well. And and in an ethical fashion. But we don't necessarily need to protect CISOs that are actually lying and trying to hide attacks unethically. So I think the good part of this is, one, I do want to be able to transparently share details without being damaged financially as a company about any sort of incident vulnerability things that happen to a product to my customers in the world in order to protect them. Like if I have a vulnerability and I'm investigating the incident, I think it's very worthwhile for the customers to have all the details on how to protect themselves, and to make sure they know if they're affected by any sort of incident that happens. So I really believe in, you know, as far as we can, sometimes authorities don't let you share everything. But I would like to share some of that detail to help the entire cybersecurity community and tech community learn from it. The other thing I'd like to protect is internally CISOs also have to be able to talk frankly, about security. Without thinking it's going to pop up in a court case, you know what I mean? Like if no matter how good you are at security, you're going to find things you want to fix nonconformities risks and part of our job as CISOs and our office, which are part of Trevor you help run our sock is to not only point those out and make sure the right leaders know about them, but to try to push towards those getting fixed, if certain things are brought up in court. And, you know, even though we're ISO 2002 2701 compliant as a company here at WatchGuard. That doesn't mean there's not perfect security everywhere we find things we want to fix all the time. So it's the fact that I'm talking about my good ISO certification. But internally, I found a risk and that shows up in court and somehow makes us negligent. I don't think that's fair. I think CISOs can't have their hands tied and they need to be able to talk about security risks. On the flip side, though, if there's a misrepresentation if there's negligence if there's a line about a state of security, when investors are actually really, you know, kind of banking on knowing you're doing well as a cybersecurity company, that is pure negligence. So I kind of feel like the judge properly thread the needle and made sure good CISOs in the future can disclose things while also making sure that negligence is still something that can be prosecuted. So good. Maybe I won't be going to jail next year Trevor. Oh, damn,
Trevor Collins 24:25
no
Corey Nachreiner 24:27
Elan marker disappointed he had become one of you would become CISO and the other one would become a director. So let's jump into the MSP spyware or I'm sorry, M spy spyware breach. I'm not sure if you heard about this, but but let me share around July 11. The world got news through not M spy the company I'm going to talk about in a second but from other people that there was a big data leak that seemed to show that a popular spyware application called M spy, that company had been hacked and some information had leaked on the underground. For our listeners, M spy is not like a ubiquitous software. So M spy is a spyware application, that markets itself is legitimate. It's like a legitimate tool for a parent, a spouse, or a partner or a workplace to keep track of someone's device and what people are doing on that device. So like, if you're a child, you can put it on a traditional computer. But em spy is mostly for mobile phones. And you can keep track of their geolocation. You can see who they're calling what they're texting, gather files and pictures from the phone. So it's technically, spyware. In my opinion, I'd love to know your opinion. But it's something that's actually marketing as a legitimate product for parents to put on children's phone, and for spouses to put on each other's phone. Now, before we dive into what you think, do you know making spyware and selling spyware is not illegal, but installing any snooping software on someone else's device without them knowing it is illegal. So first of all, what do you think about this legal spyware or this semi illegal spyware? Trevor,
Trevor Collins 26:17
it's there. There are programs that come out that are legitimate programs, like cobalt strike, and that will awesome also be used for malware. For malware breaches. There's also illegitimate programs that could be used in the proper in a proper way to be used legitimately. I think for the most part, um, Spy, is it legitimate? I think that their their intentions are to kind of release this gray ware software. And also, when you start looking at who they are, and their record, you can see that they this is now the third time they've been breached. And there don't seem to be very, yeah,
Corey Nachreiner 27:11
let's jump into that. I agree with you. While they kind of pose as a legitimate company. I think the secret thing is, they're assuming parents are installing this on children software, and more, particularly the stalkers and upset spouses worried about cheating. They're installing this without the person knowing this software is made to hide, it's not made to know for the victim that you've installed it on their device to know it's there. And if it were legitimate and legal, the person that had it would have to know so think about workplaces, workplaces, it's probably legitimate, that if someone gives you a work computer, they can monitor that you're doing work on it, that that probably is legal. But the key thing is you're going to know your workplace is monitoring you. In this case, it's meant to hide on the phone. So to me that makes it kind of at the very least unethical. But one of the things that came out, you mentioned two things I I don't I didn't know about all three breaches. But one, we now know the name of the company. It was you know, before the name of the company that makes this was not easy to find. But M spy is made by a Ukrainian company called Brain stack. And you're right that they've had breaches before. The first one was in 2018. But getting back to this new one, let's talk about this new one. As I mentioned, oops, it came to light mostly because data showed up on undergrounds. We're even organizations like crabs, and have I been poned started finding it and the data that showed up. The other thing to say is brain stack have not commented on this at all. They have not disclosed it themselves. They also have not publicly acknowledged any of the stories are the breaches. But in this particular breach, we don't know how the attacker got access to brain stack and M spies information. But they seem to have gained access to the Zendesk system, a very common support system that a lot of companies use to provide technical support. And in that particular system, the you know, brain stacker M spy has service records dating back from 2014. So over or right around a decade 24 million unique emails were in the data leak, according to Brian Krebs, others found 100 gigabytes of data. Now, this isn't all the company's data, but if you think about what would show up in a sort of support system, there is personally identifiable information like PII so names, email addresses, IP addresses associated with the devices that M spy is basically spying on and that sort of information, even images of credit cards when some of these customers were trying to pay somehow there were images of credit cards in the system. But there's also some other interesting information. Like, if you think of all the things this spies on, I don't know how they would show up in support cases, but there were personal documents from the devices being spied on. And most Concerningly, there were some nude or inappropriate pictures selfies. And that's one of the things that concerns me about this type of spyware is one partner might have access to a bunch of private pictures on someone else's phone. So this is the type of data that leaked. What do you think attackers can do with that type of data? By the way, what's the risk there? Well,
Trevor Collins 30:39
the immediate risk would be blackmail, you know, if they say, we've all gotten those emails that say, Oh, we installed a virus on your computer, and we've seen what you've done in your private time. And second, it's extortion, extortion, then they, and they never they never actually have anything, obviously. But in
Corey Nachreiner 31:01
most cases, they don't. But sometimes they have. Yeah,
Trevor Collins 31:04
no point oh, 1% of them. But but in this case, here, it they actually will have something and they could they could basically say we're going to release this and last year, extortion. And so that that is certainly one possibility. And I think there were like, there were, there might have been text messages in this speech. I to be honest, they're all there. So the 2015 breach of 2018 breach and this breach all kind of got pushed into one that gets confused by like the there might have been text messages on there. And and like you said, a screenshot I
Corey Nachreiner 31:42
think for this one, like the software is definitely capable of recording text messages and more. But I do you think this I do think in the past breach, it was actually the database data for the cloud app that holds some of this in this one, since it's the support case, data. It's not like every text message stored in the cloud. It you know what I mean, they're getting the data, but it's kind of on a case by case like I don't think they've I don't think the attackers, at least from what we can see from the leaked information have gotten the Cloud Source of all the data each individual's M spy user might have in the cloud that's being spied on. But that data sometimes is showing up in support cases. So it's kind of hit or miss. If you open a support case with n spy. And the person that got the the spyware installed on their partner might be asking the question about why the text message thing isn't working, there might be examples of some of the text messages in that support case. But unlike the 2018 breach, I don't know if they literally got every piece of information possible that this but But nonetheless, as you can tell from the 100 gigabytes, there's still plenty of of sensitive data in that, that support information, including those nude selfies. So the same point, I don't think every image that M spy has is given to the partner that spying on them is in this breach. But I think somehow some of them got into the support data. By the way, that's one thing that kind of concerns me about this company, right? Like, I would assume the data, the picture data that if you bought a company to spy on your partner, you want access to the picture data, but you don't want that floating around at the employees of the company that have it. So I'm curious, were these nude images, the specific question in the support case? Or could it just be a support? Like think about the support rev that has access to this information might just be looking at some of the data there while they're supporting it? So it is somewhat unclear how much of it's there. I'm glad you went straight to extortion because that was my worry. Obviously, some of the stories out there are talking about the PII leak, you know, things like email address and IP address can be used for in same way as other personally identifiable information to help with further social engineering and to potentially help with identity theft. Although I don't think there's enough data here to really do identity theft. But my worry was just like yours when there's nude images like i Why not go straight to that customer? The other way I think you can extort it is remember, the real victim here is the person whose phone is being spied on. Do they even know their workplace, their wife, some nation state agency, the apparent is spying on them. Someone can eat literally go to the spire, whether there's nude information or not and say, I know you're spying on your spouse if you don't do something I'm going to tell them. So there's a lot of privacy implications for sure. The one other thing I want to talk about Trevor is in the story, it was awesome. have found out that various government agencies and authorities seem to have played with em spy. And this is me putting on my spec, my speculation opinion hat. But I do not like these type of programs and think they should be illegal. But my one of my feelings is the authorities actually are looking for ways to sometimes spy on people under the guise of catching criminals. And they sometimes seem to support these applications. Do you have any thoughts about that?
Trevor Collins 35:29
I did. You know, when I was first reading about this and seeing how, okay, this is the third time that M spy was breech. I'm like, Why does the FBI have to make the anom phones, which were some phones that the FBI made a few years ago as a to where they didn't create them, but they still updated them, to allow criminal organizations to be to use them. And then the FBI could basically read all that information, while the criminals were thinking that they were hiding it. Well, why does the FBI have to do that? If the spy is already there doing it for them? Exactly. Yeah. Yeah. It's, it's something that you're not gonna want to get involved with. And usually when there's a little bit of shady business happening, you shady things will happen and breaches like this. I feel like you just, I
Corey Nachreiner 36:25
just say like you say, if it's shady, they're probably not making their own security a top priority either. So at the very least, can you even trust him with this data? So that was going to be the weekend news, Trevor. But unfortunately, the day we're recording this for the listeners out there. It's going to be Friday, the week before, but we had kind of big news today. That was an emergency that affected global businesses around the world. So long story short, there's many headlines talking about Microsoft and or CrowdStrike outages. Some of the headlines I saw even said Microsoft outages caused by CrowdStrike. Now, let me actually describe the stories. You know, by the time you listen to this, you've probably seen the news because this is making headlines everywhere. Airlines had to ground planes for a long time. And there's been a lot of delays because airlines were affected by one of these outages. But the truth is, there's actually a couple outages that really created the perfect storm of confusion. I will talk about the CrowdStrike one, which is the biggest one. But one of the things that happened the late October 18. And early I'm sorry, late July 18. And early today, July 19, was Microsoft Azure and M 365. Had some partial outages as your went down. For at least I think it was around three hours, at least in central USA regions. And while this was happening around the world at different locations, there were also many, many windows computers suffering a blue screen of death. I'm sure all the nerds around the world know that that means it's basically a Windows OS crash, that your your machine crashes. Hopefully if you restart it, it will get better. Those are two different incidents. So the blue screen of death, as it turns out was a product from a company called CrowdStrike. The product is called Falcon slight strike. It's essentially kind of enterprise endpoint protection and EDR endpoint detection and response software's similar to some of the software protection the endpoint protection products we make here at WatchGuard CrowdStrike. For companies that were using it only on Windows computers, it did not affect their Mac and Linux versions, there was an update. And as listeners know, endpoint software actually has very strong deep ties into operating systems. In order for it to do its job, it often has to have kernel level drivers in order to monitor legitimate Windows processes to because living off the land attacks nowadays target very legitimate Windows processes. It often has lots of hooks and in memory injection detection techniques. And if it makes a mistake, it can crash an OS. And in short, CrowdStrike released what they call a single content update, which sounds to me more like it's a detection and like a signature, but detections update, which may include some additional files but may not be a full product update, but a single content update that had a bug in it a mistake that had to do with one of their new memory detection techniques that caused windows to crash to blue screen of death. And unfortunately, it was a crash loop that unless you went into safe mode and did a lot of steps to fix this. You could not reboot the computer. So while joueur was having outages, other companies were having blue screens of death. And if you think about endpoint software, we not only install these on all our computers, we install them on all our servers. And if you're a flight, you know, an airline, you might install them on the kiosks on the airports might install them on the, you know, the computers running the monitors talking about schedule updates. So many, many companies had huge outages where they basically all their computers went out, and they had to figure out what's going on. And it turned out to be this CrowdStrike update. And I also think a lot of people run virtual machines in Azure as part of their infrastructure. And I think part of why the Microsoft Azure outages might be mixed with this is crowd if you're using CrowdStrike, to protect any of your organization's private issuer installs or virtual machines, they would be affected too. So let's pause there. What do you think about this? Or do you have any other additional information about this story?
Trevor Collins 41:02
Yeah, when as the news was coming out, it definitely seemed like they were related. But if you take a look at, there's less there, there's about half an hour just less than half an hour difference between where the end of this or was kind of what finally confirmed as recovered to the point where the cloud strike file, because of the file name that was involved, if you can kind of tell what time that started, it just a half an hour difference, last hour difference between that.
Corey Nachreiner 41:33
And the other thing to point out, by the way, Microsoft has confirmed they're not related for the actual issue or outage, they basically said, you know, they have cluster management, I'm sure, you know, clusters, you know, they're probably using containerized clusters of machines that they spin up and down to support their issue or infrastructure. And they said they made a configuration change to their cluster management that caused access to the central US region to be cut off. So So Microsoft has claimed responsibility in a configuration change to their back end, to cause the issuer thing, so it's very clear the issuer thing was was not related. I will say the CrowdStrike issue is a kind of a bigger deal. In that one, it affected a lot of big companies. But more importantly, as your outages are a big deal, everyone uses a juror and everyone uses em 365. But the good thing about a public cloud outage that say that cloud providers fault is when you go down, you go down, you can't access it, it goes down, you can't access it. So that's an issue. But they will bring it back up. And when it comes back up, you don't have to do anything things to start working again, right on their own. In this CrowdStrike issue, it was nothing to do with the cloud, it was the the sensor update on individual computers. And more importantly, it would put you in this Blue Screen of Death Loop, which to fix CrowdStrike couldn't fix it alone, they released of course, an update and instructions to fix it. But you literally have to go to each of these endpoint devices, put them into a safe mode or Windows Recovery mode, and do some steps before you can get the computer back into a state where you can get the update that fixes the problem. In fact, a mark I forgot to mention or in the intro, I mentioned how I'm guest hosting this week, Marc is literally traveling home. And I won't name his airline in question. But his airline was having blue screens of death all over their kiosk. And he even caught a picture of a service person going to individual, you know, kiosks and to recover them. And one other way to recover them to is use like I think they use BitLocker and other stuff. Anyways, go ahead and try that.
Trevor Collins 43:51
Was that Marcs picture or is that off the off the internet?
Corey Nachreiner 43:54
It might have been off the internet to I do know Mark sent pictures of his gait though a blue
Trevor Collins 44:01
gate. Yeah, that that was that was interesting to watch. But yeah, i i As far as the cloud strike issue that you could kind of read between the lines on on some of this here. And, and I'm I'm kind of making some some guesses educated guesses on this here. But you could tell they wanted you to connect to the Ethernet so that you could actually get to get to the internet faster, because it seem so you could summarize that the program had some type of loop that would eventually crash if it didn't connect and updated fast enough. So as opposed to going on Wi Fi would take longer to connect. Give it more time for that program to run and crash. And then yeah, the recommendation I'm going to save mode keeps the program from running, things like that. There was
Corey Nachreiner 44:55
obviously specifically there's a little bit of detail. I'm not sure if you've seen it but CrowdStrike finally updated their own that the CEO has been making news media podcasts. I've been doing news today, which is why I'm more dressed up. But they released a update on their site, which said specifically it was in a portion of it was an A new detection that had to do with memory detection techniques. But it would cause the CPU to go to 100%. So whatever the issue is, the way it was conflicting with the operating system, to your point, the issue mainly was it would quickly make the CPU go to 100%. So like you say, booting into safe mode, so that it couldn't load. Some of the CrowdStrike drivers, including the part of the update that caused the CPUs to go to 100% is how you would prevent it from happening long enough that you could remove a particular file and fix the issue. You can find all this information on CrowdStrike site on the thing was a big deal today. I will say by the way, you know, when this outage was happening, people were wondering, you know, is it as your isn't Microsoft is as a cyber attack is WatchGuard affected? We're not at all affected by this. This was you know, remember the Microsoft issue or outage, different situation. Microsoft's configuration change messed it up. But that's been fixed for quite a long time now and all is good. This CrowdStrike issue is only going to affect you if you're using CrowdStrike Falcon point. And then it will affect you. But WatchGuard products are obviously not an issue. In that case. Obviously, we have a lot of partners and customers out there that use lots of things. So if you were affected by either the Microsoft outage or the CrowdStrike issue or our heart goes out to you. Let's talk about what do you think CrowdStrike should do about this in the future.
Trevor Collins 46:49
For for CrowdStrike, what they would want to do is its quality control is obviously going to be an issue moving forward with this. There are some things they can do for further testing is I would have, I would think that they have a robust testing environment, you know, tested on different windows for the regression testing, somehow that failed. So you talk about doing some type of not a not a security incident response, but an incident response on the engineering level, to try and figure out exactly what process in the testing failed to identify the under percent CPU issue. And, and also on the on the customer facing side moving forward. You want to make sure that you are reaching out to the customers and telling them, Hey, we need to go through official channels and make sure you contact us directly. Don't let anybody and so far they've done a very good job at this. But one thing that I will add, as I was reading this was that people are going to try and take advantage of this and be like, Oh, hey, did you get this blue screen of death? While you're on using CrowdStrike? Falcon, you're going to need our assistance here. We're here to help. And well, it's actually they're not there to help. They're just there to kind
Corey Nachreiner 48:19
of like when the fake Microsoft support calls or Apple support calls where they're like there's malware on your computer. But really, it's social engineering based on a known incident that's happening
Trevor Collins 48:29
exactly, precisely. And and CrowdStrike has asked on their blog statement that you only contact their support and contact them through official channels. So I think they have done
Corey Nachreiner 48:41
well typically what they'll say is we don't call you you should call us. So like if someone if a general tip for that that type of fishing or fishing, if someone calls you saying their support from any company, you should say, okay, whatever, I'm going to call back your real number to see if this is true.
Trevor Collins 48:59
Right and including, including if there's a website that go to CrowdStrike, the real CrowdStrike we promise we won't, you know, don't don't go to make sure you're on the official website. Yeah.
Corey Nachreiner 49:13
So I agree with you with the quality assurance. I know that that CrowdStrike probably has QA. I do think over time, we're might be learning more specifically about what was updated. The this is diving more in the weeds than I did today and public interviews. But I'm not sure if the update was just a client that like there's two types of updates with endpoint software endpoint security software, there could be a complete sensor or client update like a revision to the client that basically is a reinstall of the software. But these clients also get protection updates regularly, like a full revision to the software probably only happens once a quarter you know point revisions maybe once a month, but things like that. detection, signatures, detection, new detection rules or patterns, even other things might be updated on a more regular basis. CrowdStrike keeps on calling this a single content update, which I don't really, I think that's their term for something that's not a common term. But the fact that single content isn't it makes me think it wasn't a sensor update. But it was maybe just a some sort of detection, update the kind of update that you get very regularly, rather than the kind of update you get with a regular install. And the point I'm getting at is I agree with you that you need mature QA or quality assurance processes for both. There should be a lot of focus on a big client updates, and one of the things I'm proud of at WatchGuard, for our endpoint software is we kind of have a unique way of rolling out endpoint updates. First of all, we do plenty of normal QA, regression testing, and internal testing before it ever touches anyone else outside our organization. But one thing with endpoints, which end up on millions and millions of devices is you can't perfectly recreate every weird environment your endpoints going to be in. So you can only recreate so many tests. But we also have a very phased beta process, we have a very deep phase beta process, which even includes a period of time, which we call friends and family. So before we're even rolling it out to real like customers, we roll it out internally to ourselves to our sales engineers, to friends of the family that we've shared software with, or special partners who've asked to be part of our friends and family beta. So even after our own internal QA test, that gives us the opportunity to get a more limited beta for the client updates. And then the other thing we do is if you think about like hardware updates, usually someone, usually companies will put up a firmware. And when it's ready, it's given to everyone, and you can go and download and get it. And that could be the case with endpoint software. But even when we start to roll this out to GA, we do it in a phased approach. Customers don't all get it at a time. You know, after the friend doesn't family beta, it goes out to portions of customers at a time. And this is to continue to watch it and make sure there's no things we've missed, minimize any impact of a unknown bug. So long story short, I mean, every software company will have bugs, you know, this is, no matter how good you are, I don't think we should fault a company for having bugs. But to your point, we need to continue to learn from the bugs we have and improve our QA processes to catch those bugs. The only other thing is the updates that the security software gets on a more regular basis to detections and signatures. That QA process is kind of a different one, and you need to make sure to mature it. So I assume CrowdStrike is going to go look at the specific bug that they had. They're going to figure out why their existing regression test didn't catch it. And they'll probably update their tests to find a way to catch that type of thing in the future. So was a bad day for CrowdStrike I'm sorry for any customers that were affected because I'm sure you had to touch all of your computers. But going forward CrowdStrike and other security companies can learn from this and get better. Well, it has been a busy week, Trevor. Absolutely. Okay, everyone, thanks again for listening. As always, if you enjoyed today's episode, don't forget to rate review and subscribe. If you have any questions on today's topics or suggestions for future recordings, be sure to reach out to us on Instagram, where WatchGuard underscore technologies. Thanks again for listening and you will most likely hear from us next week.
