The White House Tackles BGP Security

Marc Laliberte  0:00  
Hey everyone, welcome back to the 443 security simplified. I'm your host, Marc Laliberte, and joining me today is Corey,

Corey Nachreiner  0:09  
Nachreiner. I'm sorry about that, but if you hear any snores during our podcast, you're watching the video. This little dog next to me is the culprit. There's some are for you. I hope I didn't trigger anyone. At least it wasn't lips met, smacking all. I hate What's worse than eating on a podcast snoring.

Marc Laliberte  0:29  
We bore Coreys dog to death by discussing several topics, including the latest way to hijack Python packages on the Python package index, some research from a interesting company on a what sounded like a scary vulnerability in Microsoft's Wi Fi driver. And then ending with another publication from the White House to try and better security around the world,

Corey Nachreiner  0:57  
a government document just second. Let me get my eye mask and pillow. Stay awake. Man,

Marc Laliberte  1:03  
it's interesting.

Corey Nachreiner  1:04  
It's good stuff. It's only boring to dogs.

Marc Laliberte  1:07  
Let's go ahead and sleep. Walk our way in. So

let's start with the first news story today, which is, man, I think the millionth time we've talked about something from researchers at JFrog, but they seem to be doing it.

Corey Nachreiner  1:26  
What does JFrog stand for?

Marc Laliberte  1:29  
I don't know. Jump off. That's what I'm going to assume.

Corey Nachreiner  1:32  
It could be it. It's definitely amphibian set like J I don't know, but they do good research.

Marc Laliberte  1:37  
They do it's not their

Corey Nachreiner  1:39  
first time talking about this subject either? I don't think

Marc Laliberte  1:42  
so they they do a like proactive I guess they call it like community service, activities of monitoring all major package indexes for things like Python packages, which is the Pypi Python package index. They monitor NPM for JavaScript packages to look for basically funky stuff that's going on. And historically, we've talked about a few like package hijacking attacks that they've discovered typically revolving around things like typo squatting or other mechanisms. But they just published another blog post describing a new hijack method, or hijack technique that they've even seen exploited in the wild, which they're calling revival hijack.

Corey Nachreiner  2:28  
So revival,

Marc Laliberte  2:31  
yep, which basically allows attackers to hijack a Python package by basically taking over a package that once existed, that had been deleted by the maintainers. They even gave an example that we'll talk about a little later, of a hijack involving a package called Pingdom v3 which is a tool associated with SolarWinds Pingdom monitoring library. So basically, this all starts when a Python package that used to exist is deleted from the Python package index and Pypi actually warns developers there's like these seven check boxes that you have to go through and select, affirming that you understand all of the security risks and impacts of deleting that package off the index, basically warns that all the files are gone forever, that the name will be up for grabs for anyone and a whole bunch of other stuff. But basically, when you delete a package off of the index, it means that anyone else can go and claim that name and host their own package, and that even gets picked up by users when they go to reinstall the package or update it if the new one is at a higher version. Now, Python package index has some information about the author and like the metadata that's included if you go to the index itself, but that is not reflected in the PIP the Python, oh man, install package application that most developers use.

Corey Nachreiner  4:01  
A little command line thing you'll see when you you install something, it might show a version change with a revived package, but that key difference, which is a new author won't show up. Yep.

Marc Laliberte  4:14  
So JFrog showed a proof of concept where they created a package called just revival package with a version 1.0 it's what core is showing on the screen right now. They then deleted that package with the original author, and then, using a malicious author, went and recreated the package and set the version to Ford auto. So if you look on the Python package index website, you'll see the new author and the new package. But if you look through just Pip, you'll see that it just says there's an update available to version 4.0. Do you want to install it or upgrade it? And this seems pretty trivial, but it actually has a wide ranging impact. They found 120,000 deleted packages so far on the package index. They realized that a lot of these are like spam packages that. IPI will delete themselves. So they actually had a filter they applied, but they looked specifically for packages with over 100,000 downloads or had been active for more than six months, and not a spam package that was still 22,000 packages are open for taking they noted there's a few reasons that a package could be removed. For example, sometimes, like, the functionality from something will be included in a official library or the built in API, for example, like node has a lot of built in functionality. Sometimes a feature that was a secondary package will get sucked into the node ecosystem, and you won't need that secondary package anymore, I guess that's JavaScript, but similar for Python, sometimes maintainers just abandon their project, like they get bored of it, or they go to jail, like we've talked about previously, and that could cause a an incident where they have it deleted off Python package index, and sometimes package developers just choose to rewrite their package entirely, host it as a brand new one, and delete the old one off of the index. So in response to this, JFrog actually went and created an account called Security underscore holding and they hijacked quote, unquote, basically the 22,000 packages that are potentially vulnerable to this, or at least the most most downloaded ones, and they are hosting a version 0.0, dot 0.1, version of all these packages with really just a blank file. And there was some interesting stats they already posted as a follow up to that, where they found, in just a few days after doing that, they had over 200,000 total downloads from these packages that they were hosting, they found a few were big contributors to this. There was one package that seems to be the one that vs codes IntelliSense recommends, versus the more updated version of it. There was a package in there associated with homebrew, which was really interesting, the way for installing, like ad hoc software on Mac OS,

Corey Nachreiner  7:06  
which I use a lot. It's instead of figuring out lots, it's just the easy way to install stuff on Mac OS, rather than trying to figure out dependencies manually. There

Marc Laliberte  7:17  
were some, like discord related packages that are still hard coded, and a whole bunch of GitHub repositories as the package you should use to use these repos. And so all of these are cases where, if this was a malicious actor that hijacked these packages, they could put whatever the heck they want in there, and anyone that goes and installs these packages would then be installing the malicious code. They gave an example of that, Pingdom v3 which is a package a Python version of the Pingdom application for monitoring websites. On April 12, their automated scanner noticed that someone had been assigned as a new owner for this package, and that owner a few days before, on March 30 had released a new update for the package, and then quickly released another new update that included a suspicious looking base 64 encoded payload. Payload basically checked to see if it was if an environment variable was set. Specifically it was like a Jenkins URL. So presumably, is it on a build server, and if so, it goes and retrieves a package or a payload from a web URL and attempts to run it. And unfortunately, by the time they discovered this, that URL had been taken down. So they don't have any idea what that secondary payload was,

Corey Nachreiner  8:34  
but I don't know if they said for sure it was taken down. They said they could never retrieve the payload. So it was either taken down or it could have been a smart only paying attention to certain IPs or GEOS, yeah,

Marc Laliberte  8:49  
but you could imagine the damages of installing arbitrary software on a build server, a Jenkins server, what that could potentially cause to a supply chain. So this one, like, it doesn't, this isn't like a complex hack to take over things. It's not even relying on social engineering. It literally just like, all it takes is a package getting deleted, and the attacker can slurp it up and start going after anyone that uses that package and isn't paying attention.

Corey Nachreiner  9:17  
And it appears this one was the case of, like the second example of why a package gets taken down just a maintainer decides to stop maintaining it.

Marc Laliberte  9:25  
Yep, they noted like it doesn't rely on the victims making a mistake with typo squatting. It can prey on developers that believe that, like a once safe package, like version 1.0, dot one, you know, that is safe, and reinstalling it should continue to be safe if you believe that you could fall victim to this. And also many development pipelines just automatically install packages straight from the Python package index. So there's a lot of risk around this as a potential attack, and so far, not a lot of built in protections from. The Python package index itself. They've got some they've got a undisclosed blacklist of packages that cannot be reclaimed, probably like the really highly downloaded ones that have been deleted. But that isn't like ubiquitous to everything. Like, there's no current block preventing someone from RE registering a name that's been given up. And, in fact, deleting the package frees it up within like hours, it seems,

Corey Nachreiner  10:23  
other than JFrog. I don't know if we can rely on it forever, but right now, JFrog in the appendix lists a bunch of things they're reserving. So it appears JFrog is watching for packages being deleted and reserving names themselves as a like a stopgap mitigation. I don't know how long they'll do that.

Marc Laliberte  10:46  
I mean, it is effectively free, so there isn't really any cost to them doing that. So hopefully they continue it until some built in mechanism becomes available. I mean, the good, it's not all like doom and gloom. So in that example, they gave the Pingdom v3 as soon as they discovered it, they reported it to the Python package index organization, and they immediately took down the malicious content. So like there is a security team to respond to these but this feels like something where there needs to be some more built in controls to prevent something as easy as just taking over immediately a package that was deleted, like maybe a grace period where you can't reclaim the name, or just never be able to reclaim a package

Corey Nachreiner  11:27  
I might even go with never. I mean, I guess there could be like never by default, with the only possibility of reclaiming the name is if someone actually goes to pi, pi and shows how the original maintainer is turned over the project to this new person on purpose. But it's trivial to how to underserved change a name, and it's the same name that is the issue here. I mean, the big deal is with all the type of squatting issues a user has to make a mistake. And I think if your command line, in PIP or whatever, you can maybe tab complete something and get one of those typo squatting. But nowadays, how many people actually go manually and install I mean, they're part of install scripts, and the install script is going to, like, pick the exact name of the package every time. So this is a big deal in the sense that if you have other scripts that are going and grabbing things in an automated fashion from pi, pi, and one of those packages is removed and reclaimed, it's just going to happen to you. So I think it's worse than typo squid, and I think they should just not let those names be used again. If the maintainer goes away and removes the package with the small caveat is a new maintainer that, like officially takes over, might might have to contact by pi or something.

Marc Laliberte  12:50  
Yep. I guess one mitigation that, like software engineering outfits can and at least the larger ones do have, is you mirror your own package index, where you only host known safe versions of them, and presumably your own package index would never go and re download the same version. You know that the version on their safe, or if

Corey Nachreiner  13:09  
it is, it would be quarantine, like if you're doing known good, even if it grows and gets something that doesn't get put into your index officially until it's somehow vetted.

Marc Laliberte  13:18  
Yep, exactly. So that is one mitigation, but that's more work and difficult for smaller organizations to manage something like that at scale.

Corey Nachreiner  13:27  
I think only the hugest organizations actually do that, like the Amazons of the world, that use a lot of open source or something, but realize how big a target I feel like there's very few teams that are reading code of their open source packages, or even just doing basic diffs.

Marc Laliberte  13:45  
Hopefully, at some point, maybe the AI cyber challenge will help out with that, and open source software will

Corey Nachreiner  13:51  
have AI to do it for us,

Marc Laliberte  13:52  
exactly either way. Cool research from JFrog, and I'm looking forward to their next blog post in like two weeks that we're going to talk about

Corey Nachreiner  14:00  
again. As far as supply chain issues, I'm glad they're looking out at all the repositories.

Marc Laliberte  14:05  
Yep. So moving on. If you remember back June, back in June, on Microsoft Patch Tuesday, they disclosed and patched CVE 2024, 30078,

Corey Nachreiner  14:18  
which they described, those random stream numbers. I do remember the CV 2024, part. There's only, like, 1000 of those, so

Marc Laliberte  14:27  
you probably remember talking about, or, I guess, hearing us talk about the high severity remote code execution vulnerability and the windows Wi Fi driver. That's

Corey Nachreiner  14:37  
with the caveat, though, right? And I maybe that's what we'll learn today. This was a big unauthenticated Wi Fi that they gave a decent severity, but Microsoft did say it was hard to exploit, or something like that. I think we talked about it because they didn't share enough technical detail. And while I always would like to believe. Believe that they're correct. It's hard to exploit when you have something like an unauthenticated Wi Fi throw a packet or two and things blow up. Type of flaw, you really want to know if they're correct there?

Marc Laliberte  15:13  
Yep. Well, thanks to the researchers at cloudfence, crowdfence, crowdfence, crowdfence, we now have that technical detail and technical analysis of the vulnerability to confirm whether it is a really big deal or a not so big

Corey Nachreiner  15:27  
Yeah, and it's very much technical detail. I do not envy you. What you're about to do and describe this like machine language reversing in a way that our audience will understand, but go for it. Tell us about this detail. Marc,

Marc Laliberte  15:43  
so they just published their analysis of the vulnerability after they both statically and dynamically evaluated the patch to find the flaw and look for potential exploitability. I'm sure everyone

Corey Nachreiner  15:54  
knows the difference between static and dynamic, but static is basically using tools just to look at the file not actually like in the case, if it's a DLL, you're not actually watching it running. You're just looking at the file differences between maybe patched and unpatched, using something to pull out the machine language, whereas dynamic is it's a running process, and you're using different tools to actually pay attention to how it runs memory location. I mean, you can reverse it both ways, live and static, but, yeah, most of our audience probably knows that. But in case you didn't static, and

Marc Laliberte  16:30  
if you remember, the worst case scenario for this vulnerability would have been anyone within Wi Fi range or with a strong enough antenna could potentially exploit this vulnerability on any Windows machine that has Wi Fi turn on

Corey Nachreiner  16:41  
that would be the worst case. Windows 11, yep.

Marc Laliberte  16:44  
So they started by diffing the patch, basically taking the pre patch version and the patched version of the vulnerable driver, and just seeing what changed. The driver itself was the N Wi Fi dot sys driver. It's called a filter driver, so it sits between the protocol driver and the mini port driver. And if you don't know Windows architecture, mini port drivers, they're responsible for managing the network interfaces on Microsoft Windows. And basically this driver and Wi Fi dot sys is responsible for reading and monitoring packets received from the Wi Fi mini port driver before passing them to the next driver in the stack. So they focused on one function that changed that's basically responsible for translating the 802 11 wireless packets into Ethernet frames so that other protocols or systems on the endpoint can interact with them easier.

Corey Nachreiner  17:41  
That's a really long name. Dot 11 translate 802, 11 to Ethernet, NDIS packet. So a long name, but a very intuitive name, because it does exactly what you just said it does. Yep, the name. Quick

Marc Laliberte  17:56  
lesson for the folks that didn't get their network plus, or they got their network plus 20 years ago. What are Ethernet frames and Wi Fi frames? They're kind of like an onion where you peel off layers to get to lower, I guess, technically higher layers in the protocol. You start with the Ethernet frame, which contains an IP packet, which contains a transport segment. So Ethernet around, IP around, like TCP or UDP, Wi Fi is just a little bit different in that they don't use Ethernet frames. They use 802 dot 11 frames to encapsulate that IP packet with the TCP or UDP or whatever segment inside of it. So in 802 11, there's a header called the link or logical link controller or LLC header. This used to exist in Ethernet frames, but with the newer Ethernet frame standard, it does not anymore. It's basically, it's an eight byte header that gives some information about the 802 11 frame. If there's VLANs being used in that wireless packet, then that LLC header type is going to be set to 81,000 or 8100 rather. And there will also be a four byte VLAN header behind the 802, dot 1q header, because that would bring the total size of this header, the LLC header, to 12 bytes instead of eight. So all

Corey Nachreiner  19:21  
of these packets, you immediately expect eight bytes. If there's VLAN, you should expect only four more bytes correct.

Marc Laliberte  19:28  
So the vulnerable driver, in the vulnerable driver version, the vulnerable function, does not verify that the VLAN header bytes are actually present in the packet. When that LLC header is set to 8100 so basically, the underlying driver expects there to be 12 bytes. But if there aren't, I will, if there's only eight bytes, for whatever reason, it'll still attempt to read 12 bytes out of that packet. And this creates a out of bound read issue. Nine, which translates into an overwrite and the part of the code that's responsible for building the Ethernet header as part of this translation process. Basically, if the LLC header type is set to 8100, and the VLAN ID is higher than the hexadecimal 0x 600 which is like 1536, out of what is it? 25,000 or so potential VLAN IDs, then a portion of the Ethernet frame will get written outside of the allocated memory space into just adjacent memory. It's usually the last two bytes of the source MAC address from the Ethernet frame which is under the attacker's control. So in theory, they can overwrite and write an arbitrary two bytes. Some are in memory. Now there's not

Corey Nachreiner  20:47  
a lot like, I know, there's lots of little tricks with shellcode staging, where, if you have like, 12 to 21 bytes, you can do some funky things. But two bytes is nothing. And the other thing we got to remind everyone is where in memory is very important, besides how much memory you have to overwrite, where you're actually sitting in Windows, memory, you know, buffer, heap, all kinds of other boring, benign structures. Makes a big difference, which I'm sure you'll get into, yep.

Marc Laliberte  21:18  
Generally, you want the memory buffer to have like useful things, like pointers to other functions. So you can build a it's called a gadget chain to let you execute arbitrary code.

Corey Nachreiner  21:29  
But so this, the more you have to build that chain, the more you hopefully have buffer to have the code to do all the hops necessary to get wherever you need to be.

Marc Laliberte  21:38  
There are ways around like limitations of small overwrites like this, you can use, like spraying techniques to seed values throughout the entire buffer and then chain together what you need. But it becomes more and more complex the smaller the amount of data you can write is. And there's other limitations in this specific vulnerability too. So the function verifies that the 802 dot 1q so the VLAN headers type ID field is correct. So for example, that type ID field should be 08, 100 for IPv four encapsulated packets might be different. For IPv six or other packets. It also verifies that the VLAN ID inside that header matches the VLAN ID inside the LLC header. So there's these additional checks that happen where it makes it extremely difficult to overwrite the bytes in a way that doesn't screw up that calculation later on and just halt before it gets to the part of the function that would actually overwrite and potentially give them code execution. So basically, in practice, it means they need to get lucky and have the proper bites behind the packet buffer that would allow them to pass that PID and the VLAN ID checks, which is borders from extremely difficult to entirely impossible in a practical attack, I can't enjoy the dog snoring. It's amazing. Did you hear?

Corey Nachreiner  23:02  
Sorry, my follow is not good enough. Oh, cute. Calla Lily is right under me

Marc Laliberte  23:09  
and taking a nap. She's cute. Anyways, what were we saying? Buffer overflows?

Corey Nachreiner  23:15  
Yeah, we're getting to it's extremely hard, but they actually did some dynamic analysis, where they did some attempts. And

Marc Laliberte  23:22  
so the other piece for this that makes it difficult is they confirm Microsoft's announcement that the attacker has to be on the same Wi Fi network as the target. So you can't, just like arbitrarily, throw Wi Fi frames at the wireless driver. You have to be on the same network, which means either an evil twin attack. Or, if you're on a public network, you have to crack the access points, like key that it uses in order to send arbitrary packets to other hosts. That adds complexity too. And so, like, Finally they come down to it, like they were able to partially exploit it. And by exploit, I mean, like, overwrite memory. But they even in their debugger, had to, like, set up the best case scenario and, like, manually overwrite buffers just to see what would happen if they able to do that. And they basically found that it's almost impossible to exploit this in practice in a way that would give code execution, they could potentially overwrite another packet that arrives right after yours, the malicious one, specifically, only four bytes of it, and it would have to be placed straight after the malicious packet and memory, which is unlikely and basically useless, crazy, and so the practical impact of this is low. Thankfully, I'm everyone. I'm glad that this didn't turn out to be a insane, critical. He

Corey Nachreiner  24:42  
said it was very unlikely and useless, as far as no interesting data to be leveraged,

Marc Laliberte  24:48  
yep. So either way, like this could have been a potential scary vulnerability, and on the face of it, like it sounded like it was a potentially scary vulnerability, even without. Deep, especially with

Corey Nachreiner  25:00  
the rating. It does show you that with the right like in CVE or CVSS, I should say ratings, common vulnerability scoring system. They do have a weighted thing for difficulty exploit. But it is kind of interesting that the rating, at least from Microsoft, of this vulnerability was pretty high, despite the fact that I would almost feel like in my gut, the exploitable rating should cut this in half, like it's very unlikely to even be exploited, let alone do anything if it is. So, there she goes again.

Marc Laliberte  25:40  
Last bit before we move on from the story too. Yeah, crowdfence is not a good company. Wow,

Corey Nachreiner  25:48  
feels so that's a hot take Marc. I was not expecting this part.

Marc Laliberte  25:52  
I did some digging, and they're actually a broker for like, black market or underground zero day vulnerabilities.

Corey Nachreiner  26:01  
Oh, great. They've got why they were trying to test it.

Marc Laliberte  26:05  
They've got a bug bounty program where they offer like, up to $9 million for like, mobile zero click or one click.

Corey Nachreiner  26:14  
That's crazy, that mobiles are up to 9 million instead of just one or two, yep, but yes, that's horrible. I do not like exploit brokers that sit on vulnerabilities instead of help get them patched, especially don't like them when they sell them at that price. That said, I guess bug bounties in general are good, but when you're sitting bounty is horrible thing for a 9 million it feels more like extortion, in

Marc Laliberte  26:39  
the traditional sense, of like, a company like Watchguard paying researchers bounties for vulnerabilities they find. Like, that's great. Like, I love that program. We get a lot of value out of it. As a software engineering company ourselves, but a broker like this that presumably is buying vulnerabilities at such a premium to sell them to other organizations to use in like commercial exploitation tools or nation state governments. I'm way less excited about that type of bug bounty program. So yeah, hot take. I don't like crowdfence, even if I do like this research that they published anyways. Moving on to the last story, and I guess, a continuation of a saga that we've been talking about for years now, how it was like four years ago, five years ago, Corey, where we predicted that hackers would take down the internet by disrupting BGP, which is the backbone. Yeah,

Corey Nachreiner  27:37  
we've picked two BGP or DNS, but it was BGP was one of our big ones. I think we've done a number of stories after that, which we'll probably get into a little on, on ways to secure BGP, which are starting to filter in. But when we did those stories, our other complaint as well, BGP is a huge vulnerability, as we'll get into for the internet. There are ways to secure it, but it feels like it's the same as anti spoofing, like unless every ISP does it, if one I you know, if one registrar or one ISP is not doing it, you still have a weak link. So yes, definitely remember that BGP is dangerous. I wonder if anyone agrees with us. Mark, so

Marc Laliberte  28:21  
the White House Office of the National Cyber director just published a roadmap for addressing security weaknesses in the BGP protocol. So they start this whole like 20 page document with an intro that I want to read verbatim. Here they say, quote, a secure and open Internet is critical to the economic prosperity and national security of the United States. However, many aspects of the internet's architecture and ecosystem, including the principal technology used to route traffic across the 1000s of independent networks that comprise the internet, do not provide adequate security for the threats we face today. So they basically, they go through a note a lot of the security weaknesses we've been talking about, everyone in the industry has been talking about for what feels like

Corey Nachreiner  29:07  
decades. It's decent for non technical people, because while it's, it's high level, they they, they walk you through what BGP is, and later, what some of the protection mechanism, you know, the simple things like ASNs, as ISS, etc, so it's a good time or 210,

Marc Laliberte  29:24  
1000 foot level bgps Or BGP is comprised of autonomous system numbers or autonomous systems, which you can think of like a routing organization or group or something. It could be an ISP. It could be an individual company. Could be one piece of an individual company. Those autonomous systems are responsible for advertising the IP address space that prefixes is the name for that that they own. So for example, like Watchguard might advertise our private IP address space into the BGP ecosystem, saying, in order to reach this IP address, route it through these autonomous system numbers and all the magic. Happens on the back end, where BGP ass around the entire world, coalesce and build these routing tables to get people to the destination they're trying to go to in the shortest, preferably, number of hops possible. But so BGP has been around for forever, decades now, and it was not originally built with security in mind, and so there's a few, like weaknesses in just the baseline BGP, for example, it doesn't have an ability to validate the authority of remote networks to originate announcements for specific destinations, so there's no built in way of proving that, like my as my autonomous system actually has the authority to say if you want to get to 1.1 dot, 1.1 come to me, there's no way to verify the integrity or authenticity of messages that are exchanged between networks. These advertisements get, like bounced around and forwarded on to other autonomous systems to build that full path for routing to any given IP address. There's no capability to ensure the authenticity or integrity of information from remote networks that are being passed around and direct routing announcements that violate policies between neighbor and networks. There's no mechanisms to protect against that, and we actually saw an example of that in 2018 which really this story felt like yesterday, but crap, it was six years ago where BGP, where Google briefly had an outage because a tiny ISP in Nigeria advertised in their routing space the authority for Google's IP addresses that was picked up by a telco in China and then picked up by a telco in Russia. Basically caused all of Google to get routed through China and Russia in a way that was not designed to do, pretty

Corey Nachreiner  31:49  
dangerous Exactly. We're talking about the normal BGP. There are new mechanisms that kind of solve a lot of those

Marc Laliberte  31:58  
Exactly. So this document then goes on to describe there are actually like standards that are available right now and in use in some capacity around the world to fix a lot of these security concerns. So they talk about our PKI, which is resource, public key infrastructure. It's basically like a mirror of how we do website security right now where you've got a certificate authority, we've got a, you know, dozens, or I guess, probably 100 certificate authorities that we collectively, or at least our web browsers, have agreed, are trusted. Those certificate authorities. Simplify,

Corey Nachreiner  32:34  
yeah, it was public private keys, but the certificate authority is some trusted party that certifies that the public private key is being issued to both parties are correct. So it's all based on public private keys for the BGP servers to essentially trust each other, but someone has to be the trust group to make sure those certificates are or public private keys are. So for

Marc Laliberte  32:57  
like websites, it's a certificate authority verifying and validating that someone owns the domain name and RPKI, it's a certificate authority verifying that a autonomous system owns the IP address prefix base. And those certificate authorities are managed by the local regional Internet number routing organizations. So like in the North America, we've got Aaron the what American Association of internet numbers, or something like that,

Corey Nachreiner  33:26  
rears. If we want more acronyms for our discussion, we absolutely

Marc Laliberte  33:31  
do want more acronyms. This is a cybersecurity podcast. We have regional Internet registers, yep. So there's this mechanism already to build a public key infrastructure, the RPKI infrastructure, for BGP specifically. And then with that established, then there's two additional concepts. There's the route origin validation, which is a process for verifying BGP advertisements by using route origin authorization messages, or ROAs. So ROAs, they're like a piece of data that declares which networks are authorized to originate, quote, unquote, advertise specific address prefixes from those blocks in BGP, and they are digitally signed with a certificate that is backed by that RPKI infrastructure, ROV is the process that a BGP router can verify an advertisement and filter out invalid ones by looking at that ROA data. Now these mechanisms, they're not you new, and in fact, in Europe, 70% of BGP routes have published ROAs, and the adoption rate, though in the United States, is quite a bit lower, only 3939

Corey Nachreiner  34:48  
by the way, this is, I'm a United States person that likes our country and believes that we found it and it minus a few mistakes for good free. Intentions, but with all our exceptionalism, and we're the best 39% compared to 70 this is I'm glad the White House is talking about this, and maybe they'll be able to push this. But this is what we talked about in the past when we first kind of introduced this whole way of securing BGP that not many people are using it, and it's kind of disappointing that most of the, more than half of the United States, doesn't use it. They do get into, I don't know how much you talk about it, but they do get into some of the reasons people don't move to it. Old routers need, you know, costs associated with it. But I still think it's kind of crazy that we're, you know, I guess we're a little less than half of Europe, or a little more than half of Europe, but that still kind of stinks, in my opinion. David, we're supposed to be the ones with the internet. Yeah, well, we are. They

Marc Laliberte  35:55  
even specifically called out the US federal government themselves as a big part of the issue. The US federal government owns 21% of the entire IPv, IPv four space in the United States, or in, I guess, the Aran region, and they've got a significantly lower RPKI adoption rate relative to the private sector. They also, they didn't call out by name, but

Corey Nachreiner  36:16  
I'm glad you said, Aaron. You're talking about a R. I N for a second I was like, Aryan, region, Marc, what did this suddenly become a horrible podcast? No, Aaron. Aaron, not Aaron,

Marc Laliberte  36:27  
yep. And then they also called out, I guess, not by name, but there's a few large commercial network providers that have really complex internet address inventories and low ROA adoption levels. And like you said, they they talked about some of the reasons for that, like, administrative reasons, meaning the IP address allocation occurred 2030, years ago, and the person that was responsible for that's gone, and there's just no one knows how the heck to deal with it. There's fear of, like, disrupting customers. Like, what happens if I turn on this mechanism? And, like, all of my customers can't get to my website or my services anymore, but they then go on to give like, a like, basically a baseline for network providers, for service providers, like, step by step instructions on here's how you should, like, actually start adopting this in a way that manages risk for your organizations. The end of the report is all about like, step by step instructions for like, how to do risk based planning to figure out what is the IP space you even own. Like, who are the autonomous systems you communicate with? How are you using BGP routing? Just in general, they talk about like, how do you go about publishing your ROAs so the signed attestations for the IP addresses that you own. How should you go about getting explicit contractual requirements with your service providers that they will validate BGP enabled routes and how do you monitor the status of your ROA data. So like some high level but still like at least good guidance on how to start rolling this out as an organization.

Corey Nachreiner  38:08  
They've got another section for a lot of detail on the planning phase, at least, yes,

Marc Laliberte  38:13  
they've got guidance for service providers on how to do ROV deployments, how to help your customers with their ROAs, and also how to disclose your routing security practices, to be transparent about this, and then guidance for the federal government and everything. So at the end of the day, it's like it was all pretty high level. I think it does a really good job of explaining pretty technical concepts so that anyone can understand them. It outlines some steps the US government has already taken, and some steps that it wants to take, and also some call outs for the private industry on steps that they should take too, because at the end of the day, like deploying these standards will help. But like you said, Corey, like it really only matters if everyone starts to use again,

Corey Nachreiner  38:56  
yeah, otherwise, the weakest links become worse and worse, I think it's a good step. I hope it's followed by, like, my, my same feeling happened when they first released the the White House cyber security strategy with the five pillars. And I'm like, all great stuff, but I need to see some action that will give it some teeth. And we've seen that. We've actually, I guess, we haven't, literally, seen regulation yet, but we've seen follow up publication from CISA warnings, communities coming together to, you know, fine tune, things like secure design, and I feel like it's all in a strategy To first inform, encourage, but eventually regulate like I hope that's the case. They're going with the White House cyber security strategy, and this that this is the first step, and maybe we'll see some publications from CISA afterwards with more details with the community of what's expected for how you should be running you. Any public BGP servers, if you do blah, blah, blah, and eventually, if you know, we still don't get to that 95% or whatever, at least most servers doing it, they'll might make it requirements, at least for, if not for all companies, at least for companies that were work with governments. I guess they have to get their own, I guess they can't do that until the federal government is complying to this. And they're one of the worst offenders right now.

Marc Laliberte  40:26  
Exactly, I would call this like success if, like, a year from now, or maybe generous two years from now, like 100% of the IPv four space owned by the federal government is covered by ROA records like, that'd be good, get their house in order, and then, then you can, like, lead by example, with the private sector too, exactly.

Corey Nachreiner  40:47  
Yeah, no. But good stuff. It's definitely stuff the whole and we obviously weren't even the first this BGP problem has been kind of the best kept secret problem for a long time on the internet. So it's good to see people still pushing the new security mechanisms that will help.

Marc Laliberte  41:03  
Yeah, and I just am, in general, happy with how much like the US federal government is trying to focus on cybersecurity lately, like with that cybersecurity strategy you mentioned, with all of the push from CISA on currently voluntary initiatives, but really powerful initiatives, and now this as well too. It does seem like they are, like, genuinely taking some of the bigger risks seriously and trying to put some of the weight of the federal government behind, like, giving guidance and recommendations and at some point, probably requirements to improve security across our country and around the world. That's good America. America, exactly. Maybe at some point we'll catch up to the rest of Europe with security practice. One

Corey Nachreiner  41:48  
day we might have an internet infrastructure like South Korea,

Marc Laliberte  41:54  
or privacy protections like the entirety of the EU,

Corey Nachreiner  41:57  
yeah. What happened to our GDPR will come to the US as a federal law that that's one that I'm disappointed. It's been quite a while since we made that prediction, and it's definitely much needed. Yes,

Marc Laliberte  42:11  
100% but I'm not getting my hopes up until at least after November. We'll

Corey Nachreiner  42:16  
see. We will see. Hey

Marc Laliberte  42:21  
everyone, thanks again for listening, as always. If you enjoyed today's episode, don't forget to rate, review and subscribe. If you have any questions on today's topics or suggestions for future episode topics, you can reach out to us on Instagram. We're at Watchguard underscore technologies. Thanks again for listening, and you will hear from us next week. Easy,

Corey Nachreiner  42:39  
easy.