This week on the podcast, we cover OpenSSH's recent critical vulnerability and what it means for systems administrators. Before that, we discuss the CDK Global ransomware attack impacting car dealerships across the us, a Korean internet service provider delivering malware to their customers, and a takeover of a popular JavaScript library gone hostile.
View Transcript
Marc Laliberte 0:00
Hey everyone, welcome back to the 443 security simplified, I'm your host Marc Laliberte and joining me today is Corey
Corey Nachreiner 0:07
Sibley do no cap Nachreiner. Why? Maybe we'll find out during the podcast on today's
Marc Laliberte 0:17
kindergarteners, Corey just got back from family vacation where I'm sure all he did was get to listen to tick tock videos. Anyways, on today's episode, we'll be discussing the CDK global ransomware attack is taken down car dealerships across the United States and internet service provider in South Korea, potentially delivering malware to all of their consumers a hostile takeover of a very popular JavaScript library, or at least the delivery vehicle for it. And then some quick breaking news on a critical vulnerability and open SSH. With that
Corey Nachreiner 0:55
it's much like light relaxed news. So we should have fun.
Marc Laliberte 1:00
Yeah, let's just roll on.
Corey Nachreiner 1:04
crawl on him, maybe.
Marc Laliberte 1:15
So let's start today with the first story where if you are a citizen of North America, you've probably heard this one from either a friend or family member or just if you don't live under a rock. You've probably heard this one where the North American automotive sales industry has been having a pretty rough couple of weeks now. I think we're going on week three now, after a ransomware attack starting on June 18, impacted a company called CDK global. Which if you're like me, up until a couple of weeks ago, you'd probably never heard of CDK global ever in your entire life.
Corey Nachreiner 1:50
But maybe it's a vacation or I was living under a rock because until today I had not heard this story Marc. Now do from friends. Maybe No one's buying cars for from my family. They're all too cheap.
Marc Laliberte 2:04
That's possible. Okay. Interesting. I'm surprised anyways. So CDK global, it's the software provider for around 15,000 dealerships in North America. Their software runs everything from managing sales, managing finance, insurance, the repairs and service department. And so if this software or its cloud infrastructure goes down, none of those things can happen like they normally would. And I'll have to revert to pen and paper which and the automotive industry takes a very long time to do anything.
Corey Nachreiner 2:36
Apparently it's not just the healthcare industry that doesn't like paper anymore. Honestly,
Marc Laliberte 2:42
what industry does like paper night? Yeah,
Corey Nachreiner 2:45
can you imagine them having to bring out the carbon copy you know, their credit cards with the junk junk carbon copy things? Like
Marc Laliberte 2:54
work on like modern credit cards these days? Like, literally, they
Corey Nachreiner 3:00
they literally can do I mean, they have to call it in and it's what do they call it when it's card? Not? Card? Not they're not present is? Yeah, which require a little extra validation of other things, but I think they're still allowed and believe it or not. Anyways,
Marc Laliberte 3:17
some folks trying to buy their their automobile may have been using those custom cathodic machines over the last couple of weeks. Because at least at the time of this recording, recording CDK global still has not fully recovered from the incident. There aren't any details on exactly how it started. But we do know that the ransomware as a service operators known as black suit have claimed credit for it. They initially demanded about $10 million, and Ransom extortions and upset two 50 million after CDK global missed the first deadline. And Bloomberg actually reported the company has actually paid the ransom in order to retrieve the keys and they're in the process of decrypting all of their systems now.
Corey Nachreiner 4:02
It's curious though I wonder if that means they paid the full 50 million, although it's actually what isn't talked about you and I will probably get into why we don't love people paying the ransom even though we know there sometimes extenuating circumstances. But one of the things that ensures at least do is hire negotiators. So even when there is a ransom they tend to on the same underground where they advertise this they often have a account based chat that the effected victim can join and people negotiate. I assume Bloomberg didn't go into this detail, but I'm curious if that means they paid the 50 million or if they had to negotiate or try to get closer to the original 10 Or even below. But that would be crazy. I mean, timing wise if they ended up having to pay 50 When it started at 10. That must just add insult to injury. Yeah,
Marc Laliberte 4:53
speaking to negotiators like a story we weren't going to cover today. Last week, I guess the week before wasn't last week whatever. Someone claimed to have stolen a bunch of data, I think it was locked bid a bunch of data from the was the Federal Reserve of the United States. And they even posted online saying the they need to fire their ransom negotiator because it's worth way more than $50,000. Turns out the data was not from the Federal Reserve is from a bank that uses the Federal Reserve like every other bank. But sometimes maybe they should. Exactly. But
Corey Nachreiner 5:31
I'm with you where you might have been going, I've always been skeptical of negotiation, because I'm not sure if like, what is their leverage? Like? It seems like the ransomware author has all the cards. So once you start negotiating, they could easily just say, Okay, it's doubled now. So I don't I don't know where you were going. But I always wonder whether negotiation is a good or bad strategy when you're really dealing with a criminal in the first place.
Marc Laliberte 5:59
Yeah, I know. And also, I think this story specifically highlights another side of it, where even if you do pay the ransom, and do get your keys back, it's not like that instantaneously turns the lights back on. I got CDK global, they actually just right now released an update saying they expect to be back online by July 4. So three days from now, and probably about a week or so since they pay that ransom demand. Because, yeah, it takes time to decrypt every single system and your organization.
Corey Nachreiner 6:31
And by the way, for the folks watching the video, I don't know if we officially said the timeline, but for them CDK learned of it, it seems like June 18. So June 18 to July 4, I mean, even when paying the ransomware what is that close to three weeks, at least two weeks of time, so not great.
Marc Laliberte 6:50
And I think this also, like we've discussed a couple episodes ago, I think about like the ascension ransomware attack, and then also the federal government's investment and the research around resiliency for healthcare. So basically, what do they do when they suffer an incident? Like how do they make sure they don't revert to paper. This is another example of no if your status quo is your systems are working, operate like perfectly and you've got your software running the show, it gets hit with ransomware. Now you're back to the Stone Age like we got to find some middle ground in there for companies to run. Maybe a little lighter.
Corey Nachreiner 7:26
I'm with you. I think we talked about it before when ransomware started, we kept on pushing how backup was important. And obviously backups important, but people didn't have it when ransomware started. So it was good to remind people. But then we quickly I think even two three years ago, I mean, way more than that Marc, we move from not just backup is report. Important. But time to restore is important. It's how you backup. And there are backup companies that not only have fast restores, but literally backup in a virtual images. So one option is if you have like, you know one of your three, if you you understand the idea that you actually probably need multiple backups, just in case to our attackers target backups to one of your cloud backups could include a virtual image that literally could be back online within an hour. It's not the actual restored server, but it's a virtual image running in the cloud until you finish the restore. So while I sometimes poopoo on the cyber resiliency because I just think it's the new buzzword for having a good business continuation disaster recovery plan. You know, cyber resiliency is nothing more than making sure you have ways to quickly restore business in the event of any sort of disaster.
Marc Laliberte 8:45
Also fewer words and fewer syllables, though, so
Corey Nachreiner 8:47
that's true. That's actually a good point. I hate talking about BCDR, which is already another acronym. So cyber resiliency makes sense principle
Marc Laliberte 8:55
of least privileged or zero trust. Yeah, good point.
Corey Nachreiner 8:59
So now we're starting new acronyms maybe Gartner are earning Yeah, Gartner is make a new, shorter words for the same things, at least it's less, less Cisely acronyms. But either way, I think we both agree. It's not just I mean, we've gotten the past past the part where you need to have a backup plan. Now we're to the part where it's your backup plan needs to happen at a time period that's acceptable to your business. If you're forget even automotive or healthcare, if you're an E commerce site where you're literally making a million dollars every hour, every hour, that site is down is a million dollar loss. So cyber resiliency being able to restore that site, regardless of what disaster affects it within minutes instead of weeks is critically important. And
Marc Laliberte 9:48
one other point I wanted to touch on on this is we've seen over the last few years, like a big push towards vendor consolidation and really like software vendors across the board. work across different sectors going up a buying competition to make arguably a better joint product that they can sell to their customers then. But this comes at a risk. Now we're seeing where if you can hit that one vendor and take down everything, like the impact is massive, like I
Corey Nachreiner 10:18
don't want to get in Yeah,
Marc Laliberte 10:19
I'm with you, I don't know the history of CDK. Global, I have to imagine that like previously, maybe they had like, separate finance versus separate repair versus separate, you know, billing, whatever services and they brought them all together. And like that's, that is one risk of that.
Corey Nachreiner 10:34
I agree. I'm going straight to monopolies to it's not just separate business units. I mean, it without getting into politics, you know, besides monopolies, removing consumer choice, like you say they're painting one big target at one place, you absolutely can absolutely see the business draw to this idea. Like, when you have a huge organization, let's say like the the CDK you imagined before, that was you know, had separate departments. You know, I can see operations and finance people wanting to remove do D boob location. Look, this one organization uses the same product as this other organization from a different vendor, and we're wasting all this money. And if we bring it all together, our business be more efficient, blah, blah, blah. So you can see what drives it to that but to your point, it also the more companies you buy up the in, the more streamline that business gets, the bigger the fallout if that business is is targeted in a successful attack, so I'm not sure what the fix is there. I'm not so I'm someone that the bigger a company gets, I get scared from a consumer standpoint, because it does remove. There's other non security aspects of that that I worry about. But I think security is a big aspect. But we also I mean, you see that with Apple's walled garden, you know, that walled garden, if you think about it is the most anti consumer thing out there. But wallet has put a if you can get into Apple's walled garden, you you have billions of victims immediately. But if you have the right walls around your walled garden, maybe there is something you can you know, being a bigger target just means you better have some pretty big walls. And if you're a big target without those walls while you might learn the lesson that CDK has
Marc Laliberte 12:29
and good thing there are totally walls out there that are completely impenetrable, right?
Corey Nachreiner 12:34
Oh yeah, those exist there's it's very easy to have impenetrable wall right Marc? No sarcasm is no cap. No cap as the kids say, by the way. audience that was 100% Full on cap. Lots of cap in that statement.
Marc Laliberte 12:50
What the hell are you saying? It's
Corey Nachreiner 12:52
like a 50 year old no cap means it's true cap means it's not true.
Marc Laliberte 12:57
How does that translate like
Corey Nachreiner 13:00
ask plays millennial I don't know. I just listened to the crap they say and have to look up somewhere to translate
Marc Laliberte 13:08
it. Like okay, time to get on my old band rocker. When I was a kid the slang was like was up which is literally like, what's up massage together? You could still reverse engineer it into like, what its actual meaning is what the hell does no Kathleen, I
Corey Nachreiner 13:23
don't know, skill these days. Sigma. There's all kinds of crazy words that I don't get skippity toilet. Ask a kindergarten teacher, they probably know, our
Marc Laliberte 13:35
future is looking dark. Moving on to the next story. So this one's for all of our 1000s of Korean listeners to this podcast. So file sharing services like bit torrents are still extremely popular in South Korea. They even have their own like kind of flavor for them because they're
Corey Nachreiner 13:53
definitely not popular in the United States. Marc Right I'm sure I know buddy here in the US uses file sharing anymore, but but continue.
Marc Laliberte 14:02
Proceed. So but in South Korea, they've got their own like, kind of like I don't know version of it. They called Web hard, which is short for web hard drive. It's basically bit torrents that have a dedicated seeding server for it to make sure that that file always stays online. So the way bit torrents work. Basically, in order to download it, you have to download it from someone that has it available. And sometimes with older bit torrent files in the US, at least a random world. If the last seed goes away, no one can download that file anymore. So this web hard infrastructure makes sure it stays up. That can also be a problem for internet service providers because they use a huge amount of bandwidth. And that bandwidth typically does not use peering agreements that Internet Service Providers maintain with other ISPs, which means it cost them even more money to this apparently has been ticking off one of South Korea's biggest internet service providers known as Katie, formerly Korea tele Calm to the point where last week torrentfreak reported that a Korean language news outlet JB TC found that Katie so formally Korean Telecom was distributing malware to 600,000 computer machines and an attempt to disrupt this BitTorrent traffic. What Wow.
Corey Nachreiner 15:22
Like I get not wanting you know, pirates or even torrent or is on your ISP, but you can't that's not an excuse to distribute Mauer, in my opinion,
Marc Laliberte 15:32
their local like regional law enforcement agency, which I'm not going to attempt to pronounce, launched an investigation and even raided some of Katie's data centers last month, they identified a dedicated team at k t, with the goals of eavesdropping on customers interfering with our file sharing by any means, including installing malware on subscriber computers. Now, they didn't give any like direct evidence of any of these malware samples files, not even like an MD five hash or anything.
Corey Nachreiner 16:04
But they know so I'm curious in hindsight, when she's mentioned the evidence, how would they like it's not like they have control of injecting the Tor the magnet files or the torrent files? Shut up by man I don't even know where that came from.
Marc Laliberte 16:20
I does not want you downloading torrents. Yeah, it's
Corey Nachreiner 16:23
proof that the FBI is listening to my Apple watch. Hey, FBI, I hope you're doing good. Thanks for helping us with Cyclops blink. Like how I how would other than maybe through HTML means I mean, they do man in the middle all the web pages their customers go through? It's not like they could infect the torrents themselves? Could they? I mean that would they need to control the actual torrents to do that. So
Marc Laliberte 16:52
it needs to control the torrent because it uses cryptography and like a cryptographic hash to validate the integrity of the tornado
Corey Nachreiner 17:00
file or which which torrent there there's their ISP customers are downloading. So it makes more sense to me that they if they did inject malware in some way, it was through just normal web traffic and man in the middle delene Whatever pages and but even then they would either have to know a vulnerability to exploit the force of download or just have some sort of pop up that entice people through man in the middle. I
Marc Laliberte 17:25
think if I were in their shoes, how I would do it is similar to what you said. So like you said, they've got man in the middle capabilities across all their customers. And if you're specifically trying to target torrent users, you could basically rewrite the magnet URL as the user goes to download that that kind of starter file that Magnolia to your own version of file to your own version of that includes the malware in it like that's how I would do it. I would do it directly
Corey Nachreiner 17:53
through torrents I guess you could also if you're not trying to to affect their their specific torrent downloads. You could even as you can tell they're going to some torrent provider page, you could have a pop up that says something like Cubitt torrent needs an update, click here to get it. And so there's, we're by the way, audience we're obviously just speculating, but this has to do with the no evidence, just the news that somehow they deployed malware to 600,000. Computers, allegedly I guess.
Marc Laliberte 18:23
Like yeah, if we take a bit of a generous assumption that this is true, this is pretty insane for an Internet service provider to do like in the US this would be immediately be a violation of that computer for sure sacked and a pretty dang big one. On
Corey Nachreiner 18:39
the flip side, I think the authorities would be more than happy to help ISPs track tormentors through normal means which is more about throttling and then sending letters. Like if you have evidence of torrenting activity, which you know, without VPN intent, which is something you should be doing. I think in general just because your ISP is watching you and can always man in the middle you no matter what you're doing. Without VPN, if they have any evidence of you torrenting you know, I would just work with the authorities to gather evidence and legal letters and scare your subscribers in the stopping or sending the authorities after them and cutting off their service seems a much smarter way to stop tormentors than breaking the law yourself.
Marc Laliberte 19:26
And Korean internet service providers aren't exactly afraid of trying to use the law to shape bandwidth. When I was looking into this story I stumbled across an older one where SK broadband and other South Korean Internet Service Provider actually sued Netflix back in 2021 to try and force them into paying for the network traffic that Netflix users generated on SK broadband networks. They've claimed that basically because of like squid games, Netflix is generating a massive amount of traffic and they should have to pay for some of it and Netflix was pretty quick to point out at that time that SK broadband had not requested the free open connect CDN system that Netflix provides ISPs basically letting you keep a copy of all of Netflix's data locally. So you're not paying a ton of money for peering agreements. But long story short, like I So my understanding my limited and ignorant and foreign understanding of South Korean internet is it's actually like speeds are pretty dang progressive there compared
Corey Nachreiner 20:28
to Oh, yeah, they're super fast new infrastructure,
Marc Laliberte 20:31
no bandwidth caps at all. Yet, they're still doing this case, allegedly, pretty archaic methods to try and control bandwidth usage user users. It's pretty
Corey Nachreiner 20:43
the way for the we already did the speculation. But I started playing a video for those that have YouTube a news video of this from Korea. And I'm just repeating the star. Yes, they're showing a bunch of popup boxes. So it does look like some sort of web based man in the middle that's trying to get people to install ABC file. So I mean, we're just guessing based on the newscast, but those pop up boxes seem to suggest something to the How
Marc Laliberte 21:13
I wish I read Korean so I could understand exactly what the heck they're showing on this news thing, because right now, it's just a bunch of cool looking letters for
Corey Nachreiner 21:20
the people watching at least have auto translation on the newscaster. But anyways, we should move on very interesting story. It's neat to see if we have any Korean listeners, if you have anything to add anything you've learned about this story in your country, reach out to us, I guess now on Instagram, because we don't love x. Because we'd love to hear more in detail about it. Yes,
Marc Laliberte 21:42
please. Anyways, moving on. So before we dive into this story, a little bit of JavaScript developer background. So JavaScript, the active scripting language we use for basically every single website these days, is governed by a standard called ie CMA script. It's actually the same standard that's used by J script, which runs directly on Windows, and ActionScript used by older web browsers. Over time, new features get added to this EICMA script standard. And new web browsers will add those features and new releases. Generally, new features are not backported to maintenance releases for web browsers. And sometimes web browsers, cough Internet Explorer will implement or not implement features differently from the ECFA script. Standard. So what this leads to is if you just rely on the JavaScript functions that are available in web browsers that users around the world are using, the reality is there's hundreds or potentially 1000s of different permutations of what may or may not work. When a user visits your website, depending on what client they're connecting from, it would be just about impossible for a web app developer to maintain dozens or hundreds of different unique versions of their web apps, just to support all these different browsers. So instead, developers can use what's called a polyfill library to implement these new standards and features manually using options that are available in older libraries. So for example, there's a relatively new, I think it was like around 2018 or so maybe 2008, maybe an older new feature in JavaScript called a promise, it's basically a way to asynchronously handle a function. So basically, instead of having to wait for a function, you call to return data before you can continue processing, that function will return a promise that at some point in the future, it will return that data for you. So that JavaScript application can continue chugging along, waiting for that promise to resolve ultimately, and then once it does resolve it either, you know, accepts it with whatever the data is or rejects it and begins like error handling. So long story short, this was only available in browser releases roughly around 2014. And it was never available in Internet Explorer. Microsoft never added it to IE before killing that finally, last year. So if a developer wanted to use this promise architecture in their app, which in my opinion, also is way cleaner than basic JavaScript function handling, they would have to use a polyfill library, which could then implement it on Internet Explorer 11 and other older web browsers that their users might be using. So these polyfill libraries, they are critical for web development, and just about every single website out there probably has a polyfill library to help. Some of these newer functions work on older websites. To give an example of one polyfill library core Jas was one we've talked about a couple of times on this podcast. It's one of the most popular polyfill libraries with like 32 million weekly downloads. It is owned and maintained by a single person Someday in a week,
Corey Nachreiner 25:01
oh, god, okay. Who should check with this.
Marc Laliberte 25:07
And this is the one the gentleman that in 2019 was sentenced to 18 months in prison after hitting and killing two pedestrians. This motorcycle. Yeah,
Corey Nachreiner 25:16
I'm not laughing at him killing pedestrians. I'm laughing at the fact that a particular situation who maintains a library that gets 32 million downloads a week was in jail that that's it's funny Marc my feeling on open like I'm rah, rah for open source, don't let like don't control information, let there be open source. But I mean less and less raw raw for the fact that open source means security, I'm starting to other than the way cryptography open source is maintained. Without more auditing, and kind of, into industry wide auditing of open source libraries, I actually think open source by definition can be less secure when there's so many incidents like this one. So back
Marc Laliberte 26:06
in 2019, it actually caused like GitHub and the Node Package index, and even the Python package index to start reviewing their policies for transferring ownership of popular repositories just in case, the single maintainer of one of the most popular libraries under the sun gets thrown in prison for a year and a half.
Corey Nachreiner 26:23
Yeah, we joke about getting hit by a bus. But that could actually happen to a lot of these Open Source Repositories.
Marc Laliberte 26:30
On fortune. Unfortunately, other things can happen to these repositories as well, too. So another popular polyfill library is called polyfill.io, which is hosted on the polyfill.io domain. With source code available in a public GitHub repo. This was actually originally developed by the application engineering team at the Financial Times to help deliver their news media to all sorts of different web browsers, the Financial Times, it spun it out into its own standalone project several years ago. But then back in February of this year, the last maintainer of polyfill sold the polyfill.io domain to a previously unknown Chinese company called Fun NL, which I'm going to directly translate to no fun. Sorry, the now the original author at the time, the one that originally created polyfill, while they worked at the Financial Times, forever go like started raising alarm bells, basically saying anyone that uses this library, remove it immediately if you're pointing directly to cdn.polyfill.io to download this JavaScript package Cloudflare and fastly both published mirrors of the library at that time as well to to keep it online. And unfortunately, last month, all of our biggest nightmares came true with this, where researchers found the library was injecting malware, onto mobile devices on sites that still use that cdn.polyfill.io domain to load the library. And one example, they use the fake Google Analytics domain called Googy. And it edited analytics and nytex. Yeah, Googy, Dash and itix to redirect mobile users to a sports betting website. So this article, of course, sharing on the YouTube video from a sand sec, when they actually first published the research, someone launched a denial of service attack against their website, and then again, against bleeping computer, which is the first news outlet to pick this up. But the cats out of the bag now. And there's actually been some pretty strong responses to this event. Google started blocking Google ads from loading on any website that uses that polyfill.io domain. Cloudflare implemented real time rewrites to change cdn.polyfill.io to the net
Corey Nachreiner 28:49
spritual. That is, while I worry about organizations like Cloudflare and Google basic, basically having a path to all our internet data like ISPs, this is the upside to that is automatic rewriting of malicious domains.
Marc Laliberte 29:05
It is but it is a that's pause there for a second that is a giant can of worms, though, so it can go both ways. Exactly. Cloudflare is a CDN effectively, like absolutely massive on orders of magnitude size CDN. But basically they can man in the middle and they do man in the middle everyone's connections to actual web applications in order to speed them up and make them load faster. But with that great power means they can do things like this and read I'm not gonna say it Spider Man.
Corey Nachreiner 29:34
So as you were pulling yourself back from Spider Man uncle references, I guess.
Marc Laliberte 29:41
So with that, though, they can tamper with the traffic that goes through their services. So in this case, it was for good. They're rewriting a objectively malicious takeover of this polyfill JavaScript library to one that's safe, but in other cases like So what happens when let's if we're going to play, you know, tinfoil hat? What happens when our friends at the FBI come knocking, saying, hey, we need your help at this one thing? Can you rewrite all the traffic on the internet for this? It's,
Corey Nachreiner 30:11
well, we are friends, we like authorities doing legal things, we think there should be checks and balances to be able to have evidence and reasons. For subpoena, we believe in privacy of legal citizens as much as we believe in, you know, authorities having some power so to only
Marc Laliberte 30:30
Americans. So to be clear, no, why
Corey Nachreiner 30:32
he that's another example of no Kathy's throw in there a little bit of sarcasm. Everyone should have privacy and freedom. But But I definitely hear it It all depends on on if the organization is a do no bad, like a Google famously started as this organization is we're going to do good, we're going to do no bad. But as soon as Prophet hits as a motive that seems to disappear into the ether. So I just don't, it's not that I think Cloudflare and Google are bad. I don't think you think that either. But your point about great power gives great power. And it's it's not always at a business owners change all the time leaders change all the time boards change all the time, you saw the case with polyfill, polyfill. got bought by a Chinese government, and or Chinese own company, the US government to and and immediately its mission changed overnight. So I think it's not as throwing shade at Cloudflare. In particular, it's just the amount of power you have, if you have this kind of man in the middle capability, through legitimate means is it can go both ways. So today, it seemed to save the day. But it's also something we really need to consider as a global community.
Marc Laliberte 31:53
Yep. So quick, like WatchGuard aside, we actually we use the polyfill library and WatchGuard cloud, but we use our own self hosted copy of it that gets downloaded as part of the flashcard cloud web app bundle from our own servers, not from this taken over cdn.polyfill.io domain. That is one important distinction. The polyfill library itself is safe. It is the original distribution for it that was taken over by foreign hostile. So either way, nuts story and a another bit of evidence for what a fun
Corey Nachreiner 32:31
things to worry about. And, you know, you let me know a few weeks ago that my favorite drone company is now banned in the US because there's this fun new thing that suddenly companies like tick tock can be bought by a foreign entity, and then they turn dangerous. Yay. As if we were already heading towards 1984.
Marc Laliberte 32:54
Our future is dark. So moving on to our bonus fourth news article that actually just popped up this morning on Monday as we're recording this right now. So researchers at Qualis published a security advisory for a critical vulnerability in open SSH, which is the widely almost ubiquitously used SSH server implementation. They published that this morning. Now the vulnerability could give a unauthenticated attacker remote code execution with system level permissions on affected systems. When I created this one through the CVSS scale, I gave it a nine Dotto. Ouch, ouch. Yeah, only because the the attack complexity is actually pretty difficult to pull this one off, but the impact of it is crazy across. Yeah, it's unauthenticated
Corey Nachreiner 33:41
and full RCE. I'm glad at some more complex vulnerability. We have an easy to read article too, but Qualis published a very deep and typical analysis or not analysis disclosure of this flaw.
Marc Laliberte 33:58
The vulnerabilities specifically impacts G lib C based Linux systems, which is basically all Linux systems. In comparison, it does not affect other open SSH implementations, like Unix based systems like FreeBSD are not affected by this one. The vulnerability it's a race condition that triggers when a client does not authenticate within the configured login Grace time, which by default is 120 seconds, 600 seconds on some older systems. And before we get into like high level details on it real quickly, we already mentioned earlier in that JavaScript section, I mentioned the word asynchronous. So basically, programming program execution can either be asynchronous or synchronous. Synchronous means it has to wait for a function to complete before it can continue moving on to the program. So it will literally sit there hanging until whatever processing was going on, and that function completes and then it moves on to the next one. Asynchronous allows a program to spit out a bunch of basically parallel processing that can happened not necessarily like literally at the same time, there's different scheduling that goes on into CPU. But different things can happen. And you can do
Corey Nachreiner 35:07
other things while it's waiting Exactly. And maybe eventually it will have to stop and wait. But it can do other things while it's waiting for a return of some information or parameter.
Marc Laliberte 35:17
Yep, so asynchronous programmings. Generally, it's more efficient, it allows a program to run quicker, or at least handle more things more efficiently. But it comes with risks if you don't implement it correctly. And that sometimes the memory allocations for your program can get a bit funky if you're not handling things safely. So this vulnerability stems from the handler function that executes when that authentication attempt times out being called a synchronously despite it containing other function calls inside it that are not safe for asynchronous use, because they don't properly tidy up their memory allocations when they're used asynchronously. This actually is a vulnerability from 2006, that was originally resolved, but reintroduced in 2020, when a developer accidentally removed a little bit of logic that made that timeout handler safe for asynchronous use. And it gets very technical very quickly. If you are a nerd, definitely go read calluses advisory it's very thorough about basically their entire investigation flow, and a few specific examples of how they could exploit this vulnerability on different systems. But like real quick or not,
Corey Nachreiner 36:33
there's very friendly news articles that will give you the gist.
Marc Laliberte 36:37
Yes. But at like a 10,000 foot level, a race condition works by telling you an application to do something, and then messing with it before it can finish doing that thing. So if you say add one plus one, and then while it's trying to do that math, you change that first one to a three, well, then instead of the two that expects to get, it'll spit out a four, for example. So in this attack, they trick open SSH into thinking it correctly cleaned up a memory allocation, when in reality, the attacker still has control over the contents of that memory. That memory then gets reallocated later in the program, while the attacker still has control over it, which basically means they can use that control to modify the program execution and gain control of that program execution. Later on. There are some mitigations to this, like address space layout, randomization, or ASLR. makes it more difficult not impossible, but more difficult to exploit this flaw. They found in general, it takes about 10,000 attempts on average, to win that race condition, which is actually a bit of a mitigating factor on older open SSH implementations, where they had very long timeouts of 600 seconds. And also a very small number of concurrent pending authentications that they would allow, basically, meaning if you look at open SSH on Debian three from 2005. So 19 years ago, it would actually take about a week on average, to obtain a remote shell by hitting that one in 10,000 chance with all the concurrent limitations that have, but on modern open SSH takes about three to four hours to win the race condition. And on systems that implement address space layout randomization takes about six to eight hours to obtain that shell. So it's not a instantaneous, boom, it's exposed, I've popped it and now I have full control. It takes 10,000 attempts to do it and getting a little bit lucky with your timing. But it is still entirely possible and proven not just a theoretical vulnerability at this point. One part about timing attacks, though, there's another bit of a mitigating factor and that they are reliant on timing things correctly. So like network jitter, and latency, and even just being far away geographically from the target, increasing that latency and jitter can be a bit of a help for mitigating it. But either way, this is a good way to put it simply,
Corey Nachreiner 39:04
it's timing and luck. I mean, anyone that's played with Metsploit with a race condition memory vulnerability knows that you if you're trying to like they're, they're the worst if you're trying to do a live demo with the demo gods to get show something on the first exploit because typically, you have to run them that maybe one out of five times successfully. So you you get a perfect system where you can kind of control timing and you don't have to worry about network because it's local, and you run the exploit five times and it fails until that fifth attempt. So I what I'm getting at is I'm with you on it drops to a nine because it's unreliable. And your point about some older systems that unreliable, this becomes even a bigger deal because they simply can't handle the amount of requests you might have to do. But I will say if a perfect exploit came out for this and by perfect I don't I mean, it runs every time, it's simply a matter of rerunning it over the internet 10 or 20 times, or, I'm not saying this out of experience for this exact one. But with similar flaws, they still end up being easily exploitable. And that you just have to repeat it over and over against the same system, and eventually it hits. So I don't know that amount of detail for this one. But I have seen those, everyone's probably seen those Metis played exploits that, you know, it's probably only going to have 20% or less hit time. But that doesn't prevent you from running it over and over again, as long as the service stays up, call
Marc Laliberte 40:37
us actually mentioned that in their advisory to they said that this is probably the most inefficient way to try and exploit this vulnerability, they just know it works. They even suggest it like because part of the problem is you have to gauge like, you have to measure the your your timing window, basically your window to gain that the the race condition victory. And then within that there's a number of smother things that potentially when I see and they said that measuring that window and knowing when to send that final packet to try and trigger the exploit. It's actually pretty difficult for them and they were using some different algorithms they created. They proposed like maybe a motivated attacker could use like LLM learning or like machine learning Oh, wow. Hey things LLM these days but use machine learning to make it a little bit
Corey Nachreiner 41:30
how large language models anyways, I get what your point using some sort of machine learning to to figure out the window better. That's all
Marc Laliberte 41:38
like even the exploit they suspect there could be significantly more improvements for this but even if we like like that's just one small bit like it is unreliable, it takes a bunch of attempts to do it. The impact is still substantial. If you've got a exposed open SSH server, someone can get huge
Corey Nachreiner 41:56
it's big ly like go back on vacation in Corey too soon.
Marc Laliberte 42:06
Anyway, anyways, a quick WatchGuard aside on this one as well. FireBox appliances are actually affected by this one. But if you are a FireBox administrator that has been following our guidance for the last decade and a half, you hopefully don't have your management access exposed to untrusted networks and especially not the internet. If you want to you can explicitly block the SSH port which is TCP 4118 to the FireBox itself if you don't use SSH management at all, but please just follow the guidance we've always given a don't expose management access to untrusted networks period. That's it.
Corey Nachreiner 42:43
And I presume by the way, even though this you even though we're technically vulnerable today, but you shouldn't you should have an easy way not to expose it it will be part of our P cert process. So we will fix this in some course of order with with everything we go through our security team project management and engineering to to figure out the impact which we think this is probably pretty low because you shouldn't have your your management exposed on the internet to decide how quickly we fix things.
Marc Laliberte 43:14
Now the good news is our engineering department is treating it as a critical issue and should update the FireBox relatively soon. check
Corey Nachreiner 43:23
Marcs PCERT page,
Marc Laliberte 43:24
we do have an advisory that will be the it is not my bucer page it is WatchGuard is piecework page, P cert, guide
Corey Nachreiner 43:32
it for initial design to who it should go to. That
Marc Laliberte 43:36
is the best place to keep up to date on the progress of this advisory. So man, what an exciting week full of things that give me great hope for both cybersecurity and humanity. We
Corey Nachreiner 43:50
haven't even talked about the the big healthcare ransomware I heard about a while back. And there's some Hemlock thing maybe we'll talk about cluster bomb malware. I wish the headlines would stop between global politics and information security and everything else. I just don't like headlines lately.
Marc Laliberte 44:08
Yeah, welcome back from vacation Corey
Corey Nachreiner 44:11
I'm going to leave I'm gonna go back to an island maybe maybe you can have the sea so overall, I
Marc Laliberte 44:19
you know, working in McDonald's is looking more and more appealing every day.
Corey Nachreiner 44:24
I bet you I'd be happy is a what do they call it the old Best Buy job? Maybe geek squad would be the same horrible
Marc Laliberte 44:36
living hell speaking from experience
Corey Nachreiner 44:43
with that lovely news credits.
Marc Laliberte 44:50
Hey, everyone, thanks again for listening. As always, if you enjoyed today's episode, don't forget to rate review and subscribe. If you have any questions on today's topics or suggestions for future episode topics or if you speak Rinne and you want to tell us what was going on in that newscast. Reach out to us on Instagram. We're at WatchGuard underscore technologies. Thanks again for listening and you will hear from us next week.
