Secplicity Blog

Cybersecurity Headlines & Trends Explained

Go to 

Breaking Alert: MSP Targeted Ransomware Attack (Kaseya Supply Chain Attack)

02 July 2021 By The Editor

Managed Service Providers (MSPs), especially ones using Kaseya VSA, should read this and take action as soon as possible.

High-level Summary:

On Friday, July 2, some MSPs using the on-premises version of Kaseya VSA suffered ransomware attacks that trickled down to their customers. Kaseya says around 1500 companies (so far), many customers of MSPs, have been affected and the attackers (Revil gang) are asking $70 million in ransom. The attack exploited unpatched vulnerabilities in the Kaseya product that Kaseya is working on fixing ASAP. Anyone using an on-premises Kaseya VSA server (does not seem to affect the SaaS version) should turn it off or remove it from your network until Kaseya releases the fix. WatchGuard's endpoint products like WatchGuard EPDR, Panda AD360, and others, can catch the dropped ransomware. This post came out on Friday, but will continue to update as we learn more information, scroll to the bottom to see the first post with the latest updates at the top.

Update 7: TDR coverage and Kaseya related phishing (July 9, 2021, 8:30am PT):

While the Kaseya VSA SaaS service is not yet restored (expected Sunday) and patches are not out yet, we do have a few relevant updates we wanted to share including one warning. First the warning. Kaseya has alerted that they have seen threat actors sending spam and phishing emails that pose as fake Kaseya updates or advisories about this incident. These emails can contain malicious links or attachments. You should remain skeptical of any emails claiming to be from Kaseya, and be sure to try to validate them as much as possible (pay attention to the full sender domains and the contents). Kasaye has promised that going forward they will not include any links or attachments in the email updates they send. So you if you receive one with those, that is a potential malicious indicator. In better news, after sharing how our endpoint products detected the Kaseya ransomware, we have had many queries on whether or not our Firebox Threat Detection and Response (TDR) service and agent can also detect it. It does, and in multiple ways! As mentioned our APT Blocker service did detect the Kaseya ransomware the day it was released. If you configure TDR to use APT Blocker, that alone would have caught it. However, we have also tested TDR's Host Ransomware Prevention (HRP) capability, which is a behavioral analysis model HRP applies to all running processes. HRP also triggers on the Kaseya Ransomware, and prevents it from encrypting files. We have also since added all this threats signatures and IoCs to our threat intelligence too, but either of those features would have prevented this even before those signatures and threat intelligence. As previously mentioned, we will update you as soon as Kaseya starts releasings their patches. 

 
TDR's HRP detects Kaseya ransomware 

Update 6: Kaseya release delays; People plan, God laughs (July 7, 2021, 4pm PT):

There is a saying that, "people plan, God laughs," essentially meaning the best laid plans don't always execute exactly as desired in the real world. We have been sharing Kaseya's interim updates about both the restoration of their VSA SaaS service and the patches for their on-premises version. It appears they will not meet some of the times and dates they shared earlier today. They say they will email clients a video from their CEO with more detail, but it appears the on-premises VSA patch will not come out today (nor is the SaaS service restored). Rather than continuing to share all their updates (including some that haven't come to pass), we will wait to update this post until they have solidly released a patch and/or restored the SaaS service. If you want all of their interim updates, follow their main advisory post. However, one promise they did deliver today was a "runbook" that gives on-premises VSA administrators a set of things to do in preparation for the upcoming patch. If you are a customer, you should receive that in email at some point tonight. At a high-level, this runbook includes:

  • Steps to isolate the VSA server from the network and the internet 
  • How to run Kaseya's Detection Tool  
  • Steps to patch your operating system to ensure it is up to date 
  • A detailed review of the required changes to IIS  
  • How to download the FireEye agent on the VSA Server 
  • How to implement the FireEye agent on the VSA Server 
  • Final review of the checklist before the installation of the new VSA release 

If you use the on-premises product, we highly recommend you follow the steps in that runbook as soon as possible so you are positions to patch your server and restore its services as quickly as possible when the patch comes out.

Update 5: Kaseya VSA SaaS still down, but patches today (July 7, 2021, 11am PT):

Despite them starting the process, Kaseya ran into technical issues restoring the SaaS VSA service and have since delayed. They now plan on beginning the SaaS restore process "no later than the evening of Thursday, July 8th US time." The good news is they seem to be going forward with the on-premises VSA patching despite the delay in the SaaS service restore. The say they plan on publishing a runbook of the changes to make to your on-premises VSA by noon PT today (3pm ET) and claim the on-premises patches will follow by 2pm PT (5pm ET). Will will update this post as soon as we see the patch.

Update 4: Kaseya VSA cloud service going back online (July 6, 2021, 6pm PT):

Kaseya just updated their main advisory page to inform customers that they have started bringing the cloud VSA service back online, including many security additions that should help protect it. They believe that their entire VSA SaaS infrastructure should be completely online and restored by July 7, 6:00am ET. More importantly for the on-premises VSA users, for which this attack affects, Kaseya has promised to release patches for the on-premises VSA within 24 hours  (or less) of restoring their SaaS service. That means, we can expect the VSA patch sometime tomorrow at best, or 6am ET on Thursday at the worst. As mentioned before, you should keep your on-premise VSA servers offline until these patches are release. We will let you know here as soon as the patches come out.

Update 3: Vulnerability details emerge (July 6, 2021):

Details continue to emerge on this incident as the long U.S. holiday ends. We now have confirmation that attackers likely exploited some unpatched vulnerabilities in the on-premises version of Kaseya VSA to install this ransomware. As it turns out, a security research company, DIVD, had found and responsibly reported a number of vulnerabilities in the Kaseya VSA product before this ransomware attack. While they say Kaseya was diligently working on fixing them, it turns out the Revil ransomware gang found them first. While DIVD is not yet sharing technical detail (until patches are available), the community believes the vulnerabilities may include a pre-authenticated injection vulnerability (likely SQL injection) and an arbitrary file upload issue. The attack may include other issues beyond DIVD's findings too. Kaseya still recommends you keep your VSA server offline, however, we expect patches for these flaws to come out shortly. Kaseya says they will release it within 24 hours of the SaaS service returning, which seems like it will be sometime tomorrow (Wed. July 7th). For more details on what DIVD has shared, see the reference section at the bottom of this post. Besides the new DIVD information, Huntress Labs has an excellent detailed post with regular updates and other technical information. See that in the reference section below too. Finally, TrueSec have released an excellent tool to identify and mitigate affected Kaseya systems.

Update 2: Technical & Product details (July 3, 2021):

Yesterday afternoon, we warned you of a breaking ransomware attack affecting Managed Service Providers (MSPs) and the customers they serve. Today, we are updating this post with more technical detail and product defense information, as well as some analysis that comes from our Panda Research team, using the threat intelligence from our Adaptive Defense 360 (AD360) and Cytomic products. Before diving into more technical details, let's cover WatchGuard's high-level product protections. Both our network and endpoint solutions have protections that can identify the malicious ransomware encryptor (agent.exe) dropped during this attack. Even when we first learned of this attack, APT Blocker, our proactive behavioral-based sandbox, proved to identify this ransomware file as malicious. Our Gateway Antivirus (GAV) service detects it as well. From the endpoint side, AD360 (WatchGuard EPDR) and Cytomic can also detect this threat. Besides identifying the malicious agent.exe file, AD360's Contextual Engine detects many of the malicious actions the compromised Kaseya agents attempts. We'll share more detail in our research below. That said, the community does not yet know the ultimate root cause of this compromise. Many suspect it's a vulnerability in Kaseya VSA. Not knowing how the attack starts, it's hard to say when our network products scanning might help, but if you use our endpoint products on your Kaseya VSA server (assuming you are using the on-prem one), you have protections. Let's get into some technical details, discovered by our Panda (endpoint) threat hunting and research team.

Incident Overview:

Our telemetry first sees this attack starting on July 2, at 13:56 UTC. While that represents our customers, it largely matches when the rest of the community started seeing this attack. 

 
When the Kaseya attack was first seen in our telemetry. Image from WatchGuard/Panda Orion tools.

While the community hasn't identified the root cause of how the compromised Kaseya agent (agent.exe) gets onto a victim computer (potentially a Kaseya vulnerability), we do know that the attack starts with this malicious cmd.exe (Windows CLI) command that executes seven actions.

"C:\WINDOWS\system32\cmd.exe" /c ping 127.0.0.1 -n 5388 > nul & C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Set-MpPreference -DisableRealtimeMonitoring $true -DisableIntrusionPreventionSystem $true -DisableIOAVProtection $true -DisableScriptScanning $true -EnableControlledFolderAccess Disabled -EnableNetworkProtection AuditMode -Force -MAPSReporting Disabled -SubmitSamplesConsent NeverSend & copy /Y C:\Windows\System32\certutil.exe C:\Windows\cert.exe & echo %RANDOM% >> C:\Windows\cert.exe & C:\Windows\cert.exe -decode C:\*REDACTED_TEMP*\agent.crt C:\*REDACTED_TEMP*\agent.exe & del /q /f C:\*REDACTED_TEMP*\agent.crt C:\Windows\cert.exe & C:\*REDACTED_TEMP*\agent.exe

Here's our analysis of the seven actions in this one command:

  1. ping 127.0.0.1 -n 5388 > null
    • This command is used as an evasion technique against security software that does not support multiple commands, as well as to perform a sleep before the attack is performed.
  2. C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Set-MpPreference -DisableRealtimeMonitoring $true -DisableIntrusionPreventionSystem $true -DisableIOAVProtection $true -DisableScriptScanning $true -EnableControlledFolderAccess Disabled -EnableNetworkProtection AuditMode -Force -MAPSReporting Disabled -SubmitSamplesConsent NeverSend
    • This PowerShell command disables Windows Defender protections.
  3. copy /Y C:\Windows\System32\certutil.exe C:\Windows\cert.exe
    • This command copies the system certutil to an operating system path as that specific directory is often excluded from analysis by some endpoints security products.
  4. echo %RANDOM% >> C:\Windows\cert.exe
    • This is another evasion technique based on modifying the original binary to bypass detections that identify system binaries outside of their standard installation paths.
  5. C:\Windows\cert.exe -decode C:\*REDACTED_TEMP*\agent.crt C:\*REDACTED_TEMP*\agent.exe
    • This decodes the malicious binary from its original location and renames the malware (ransomware decryptor) on the destination path.
  6. del /q /f C:\*REDACTED_TEMP*\agent.crt C:\Windows\cert.exe
    • This deletes the not unnecessary files from step 4 and 5.
  7. C:\*REDACTED_TEMP*\agent.exe
    • Finally, the PowerShell command executed the malicious agent.exe file, which is the actual ransomware.

 
A visualization of the CMD.EXE PowerShell execution. Image from WatchGuard/Panda Orion tools.

The executed agent.exe, generates the files MsMpEng.exe and mpsvc.dll in your default Windows path (%Windir% or your system root).

  • mpsvc.dll is the malicious DLL that belongs to the Revil family. In short, it's the ransomware encryptor.
  • MsMpEng is an old and valid version of Microsoft Defender that is used in this attack to do a DLL hijacking of the ransomware DLL.

Finally, MsMpEng.exe is executed to encrypt the content of the computer.

Adaptive Defense Contextual Engine Protections:

If you use our Adaptive Defense 360, Cytromic, or WatchGuard EPDR products, our solutions' Contextual Engine provides various protections against this threat. For instance, the Contextual Engine detects the initial CMD.EXE command launched by the compromised Kaseya agent as suspicious, blocking the initial step and protecting the computer from this attack. 

 
Contextual Engine detects malicious CMD.EXE execution. Image from WatchGuard/Panda Orion tools.

The Contextual Engine also detects the execution of the malicious PowerShell command. 

 
Contextual Engine detecting malicious PowerShell. Image from WatchGuard/Panda Orion tools.

Furthermore, if you have configured AD360 in Lock Mode, it prevents many associated files used in the various steps of the attack since they are categorized as "unknown." Specifically,

  • executing the modified cert.exe (step 5),
  • the execution of the decoded agent.exe (step 7),
  • loading the Revil DLL (mpsvc.dll) when agent.exe executes.

To summarize,  Adaptive Defense can detect and prevent this attack both by its Contextual Engine and Lock Mode protections. Our EPP and Panda Dome software also share these Contextual Engine protections too.

Summary:

If you use any of WatchGuard (Panda) endpoint products, you have protection against the malware used in this Kaseya attack. WatchGuard's Network (Firebox) products can also detect this malware when it transmits via any network services our proxies handle. That said, without the root cause of this Kaseya supply chain attack, it is hard to say how this threat first enters your network. We will share more on that as soon as it's discovered. Meanwhile, if you use Kaseya's on-premises VSA server, you should turn it off or remove it from the network until Kaseya tell's you more. They have already disabled their cloud version. Note that Kaseya has updated their own advisory since yesterday, so we recommend you check it regularly. We will continue to update you here as we learn more. Thanks to our Panda Research team and WatchGuard Threat Labs for the additional info in this post.

Original Post (July 2, 2021):

Midday Friday, right ahead of a long U.S. Holiday weekend, we became aware of an ongoing ransomware attack campaign targeting MSPs. The attack has allegedly affected eight MSPs (now up to 30) so far, and may leverage vulnerabilities or misconfigurations in Kaseya's VSA solution. If you use Kaseya VSA, Kaseya recommends you immediately shutdown your Kaseya VSA server until you get further notice from them! The Cybersecurity & Infrastructure Security agency has also posted an alert about this. This is an ongoing attack, and we are still getting up to speed on the details. Huntress Labs has created an excellent Reddit post on the r/MSP sub that is aggregating all the details the community shares as the issue breaks. I highly recommend MSPs frequent that post until we can share more detail. The post shares some indicators of compromise (IoCs) and other information that can help you protect yourself. The attack seems to come from the same group responsible for the REvil ransomware, which we've written about before. Internally, we will track these IoCs and make sure our security controls can identify them. We hope to share more detail next week, as we learn more. In the meantime, I've shared some references for MSPs to explore below.

REFERENCES:

—Corey Nachreiner, CISSP (@SecAdept)

Share this:

Filed under: Research
Blog Home

Related Posts

Blog Home