The recent Microsoft announcement on “Volt Typhoon” activity brings the reality of persistent threat actors back into the spotlight. It is not always the destruction of data seen in ransomware attacks that are a threat to organizations but rather the actors that maintain continued access are often even more dangerous.
The objectives of these threat groups are well planned and well-resourced by the agencies that support them. Furthermore, since maintaining access is a key goal for these groups the activity tends to follow more stealthy techniques to stay hidden and blend as well as having periods of dormancy. In the past as an incident response consultant, I have even observed state actors maintain access in an environment for more than a decade.
Microsoft's Revelation on Volt Typhoon's Critical Infrastructure Attacks
On Wednesday May 24th, Microsoft released details of attacks targeting US Critical National Infrastructure perpetrated by a Chinese state sponsored attacker tracked as Volt Typhoon. This campaign, also saw release of a separate flash from the Joint Cybersecurity Advisory board, which includes the NSA, CISA, and NCSC as members.
Stealthy APT Attacks and 'Living off the Land' Techniques
From the first APT1 attack, the Piriform CCleaner compromise, Solarflare attacks against SolarWinds builds leading to mass compromise, and many more besides, one thing is clear: highly motivated and competent attackers will always find a way to compromise a network, and when they do, they will do everything in their power to remain hidden.
In all these examples, attackers “lived off the land” and used built in tools to achieve a long lasting and hard to detect presence on networks.
Challenges in Detecting Stealthy and Strategic Threats
The unseen threats operating under the radar can often be more damaging, as their objectives are typically more strategic and carry greater consequences. One key takeaway is that using legitimate credentials and living off the land techniques makes detection of the activity even more difficult. The actors are often not installing malware or other commonly used hacking tools but instead relying on what is already present in the target environment to conduct espionage and other nefarious activity like data exfiltration. Even if new tools are used these are often opensource or legitimate which is unlikely to trigger any alarms by Endpoint Detection software. This puts a greater weight on detecting this activity by observed behavior, which is challenging since the activity is likely to occur within an organization’s environment legitimately. This all is to say that detection is going to rely more heavily on learned normal behaviors and methods for bringing those deviations, where they relate to attacker techniques, to the spotlight for investigation.
Persistence and Subtlety in Volt Typhoon's Tactics
The methods used to gain and maintain access in the Volt Typhoon reports are not new, with similar techniques being used by other state sponsored espionage campaigns over many years. With that said, APT threat actors are still a threat to the security of an organization and will continue to be, so bolstering detections around their activities is critical.
Breaking Down Volt Typhoon's Espionage Techniques
In the Volt Typhoon campaign, we can see a similar breakdown of “expected” espionage threat actor behaviour:
Initial compromise > Live off the land reconnaissance and lateral movement > Stage information for exfiltration through C2 channels
More specifically, from the reports written by Microsoft and the JCA advisories we see the following behaviour from Volt Typhoon
- The attacks originated from Small Office Home Office (SOHO) devices which had been compromised and then used to proxy traffic to the targets. This masks incoming attacks and C2 infrastructure behind residential IP address spaces.
- The attackers used compromised credentials to log in to systems at the targets. This masks attacker activity as there is no exploitation of external infrastructure or exposed web systems in order to gain a foothold in the environment.
- Once the attackers had access to the target environment, they did not use malware in order to gain persistence and a command-and-control channel. Instead, they relied on built in tools to create proxy tunnels, and only in very rare circumstances fell back on using tools such as Fast Reverse Proxy (frp), or EarthWorm.
- Attackers used built in Windows tools in order to reconnaissance their environments, and rarely used any attack tools, or known suspicious binaries (such as PSExec) to execute commands and collect data.
Even without knowing the origin of this attack and the state sponsored nature of this attack, these Tactics and Procedures already point to a highly motivated and skilled operation.
While the goals of the attacker are currently unclear, this follows the pattern of previous state sponsored attacks. Highly competent threat actors may be motivated by monetary gain, or notoriety in the form of compromising high value targets, Volt Typhoon, appear to follow traditional espionage patterns.
Understanding and Responding to LOLBins and Volt Typhoon IOCs
In the face of such intimidating attackers, what can we as defenders do? The sheer nature of LOLBins (Live Off the Land Binaries) and their use in espionage style attacks makes writing a comprehensive list of IOCs difficult. While the CSA and Microsoft blog share SHA256 hashes and strings to look for in your environment, most of these files are either existing tools which are marked as malicious in tools such as VirusTotal or are only useful when performing incident response and forensic activity.
The Challenges of SOHO Device Compromises
The best defenders can look to accomplish is to understand the motives and the basic practices of these types of attackers to hunt for threats in their own environments. To this end let’s look at the above bullet points and discuss some threat hunting basics we could implement in order illuminate malicious activity.
The attacks originated from Small Office Home Office (SOHO) devices which had been compromised and then used to proxy traffic to the targets. This masks incoming attacks and C2 infrastructure behind residential IP address spaces.
This point comes with a few challenges. Firstly, it’s worth noting that the attack originates from public IP address space, it doesn’t mean that everyone who has a broadband connection needs to run a full APT incident response process, but you should ensure that the remote administration tools available on most commercial routers are not available to the internet. As a potential target for APT moving through these SOHO devices, defenders should be looking at methods for detection sign on activity for trusted users but from untrusted or unknown locations. This can take the form of looking through logs, or investing in detection capabilities which can help identify these potentially malicious behaviours which are unusual for your environment.
The Role of Compromised Credentials in Volt Typhoon's Strategy
If there is a login from a device, or IP address which is unknown for a user, then services such as Spur.us can help identify the usage of the IP address and point defenders towards making a judgement about a specific event. There may also be several users logging in from the same device in quick succession which could also indicate malicious behaviour.
The attackers used compromised credentials to log in to systems at the targets. This masks attacker activity as there is no exploitation of external infrastructure or exposed web systems in order to gain a foothold in the environment.
As with the previous bullet point, this one pertains to compromised credentials being used to gain a foothold in the target environment. Whilst several of the previous pieces of advice still stand, there are also some other more account specific points to consider.
Once an attacker is inside an environment, they need to access hosts in the environment, and will touch services (Kerberos SPN services) in the environment which will potentially cause alarms in the environment. Defenders should look for unusual account activity across their network including but not limited to
- Low privilege accounts logging in to unknown high privilege hosts such as Domain Controllers or Exchange server
- High privilege accounts logging into low privilege hosts and accessing high privilege services e.g: A domain admin, logged into a HR laptop, connecting to Active Directory to query policy.
- User accounts logging into hosts where there is no clear link e.g: A finance officer logging in to a host belonging to IT support.
Analyzing Rare Tool Use and Proxy Tunnels in Attack Strategies
These types of detection can be automated with the correct tools but may serve as a good thought framework when determining if a host or account was used in a potential breach.
Once the attackers had access to the target environment, they did not use malware in order to gain persistence and a command-and-control channel. Instead, they relied on built in tools to create proxy tunnels, and only in very rare circumstances fell back on using tools such as Fast Reverse Proxy (frp), or EarthWorm.
It’s easy to look at this behaviour and believe that blocking access to Earthworm or FRP tools would be good enough to stop the attackers. However, this was only observed in rare cases. In most instances, the attacker used built in Windows tools. One example of this is the netsh.exe portproxy command. This command is limited to Windows Server products and allows a user to create a proxy on a host to forward communications to another host in the environment. The cases where this happened as opposed to just using valid credentials was minimal however, so account and credential validation should be done as an order of priority.
Defenders should here be looking for a few artefacts, either on the network or in memory on the hosts.
Scrutinizing the Use of Built-in Windows Tools for Reconnaissance
On the network, defenders should look for incoming connections to rare ports. Inbound traffic to the environment should be reasonably static and well understood, so incoming connections to hosts on unknown ports ay serve as a starting point for investigating a potential compromise. In the rare cases where off the shelf proxy software was used, the attackers hard coded the following ports to connect back to their C2 infrastructure 8080, 8443, 8043, 8000, and 10443. These, however, are well known ports for commercial internet proxies and may cause a defender to chase down too many false positives for this to be the beginning point of an investigation.
Attackers used built in Windows tools in order to reconnaissance their environments, and rarely used any attack tools, or known suspicious binaries (such as PSExec) to execute commands and collect data.
This last point is one of the most difficult for defenders to detect. While it is possible to find activity related to the use of LOLbins, distinguishing this traffic as separate from normal day to day work is difficult Therefore, the recommendation is to search for this activity when there is a suspicious host or account to focus in on and assess.
Volt Typhoon was known to use tools such as ping.exe to recon the network environment, therefore hosts which have performed ping sweeps of the network may trip internal detection mechanisms, be it network detection and response products, or honeypots which should not be touched, ping sweeps tend to be noisy. They also used wmic to perform remote operations on target hosts using the WMI (Windows Mangement Instrumentation) service. As a result, the above recommendations regarding looking for suspicious login activity would apply here as well. For example, wmic use may cause a Windows Event Log ID 4624 to be created, with the logon type for the account being 3 (Network).
If you are using a monitoring solution which can expose named pipes on the network, the following may expose WMIC being used to start processes remotely. However, these named pipes and associated methods are quite generic and will be observed across the network with normal activity. However, the ones to really pay attention to would be those which come from high privilege accounts, towards high privilege hosts. Specifically, IWbemServices::ExecMethod should be paid close attention to as this function is most typically seen to enable remote commands to be issued via WMI.
Staying Vigilant Against State-Sponsored Threats like Volt Typhoon
Attacks against critical infrastructure is always a scary prospect, particularly when the threat actors involved are state sponsored and highly capable espionage operators. However, as defenders we must remember to fall back on our principles of being on the blue team. Compromises happen, we must be prepared to deal with them when they happen. We must have a good understanding of our network and operations within the network, be this through tooling which can expose unusual activity or just by knowing the user base and their job roles intimately.
Just because someone is logging on from a home IP address, it doesn’t mean that everything is ok. Home and hybrid working means we see increasingly remote workforces, but this is no reason to let our guard down. Consider the locality of the user, and potential impossible travel scenarios as well as any established login patterns for that user.
Living off the land is a valuable and stealthy technique, one which is not going to go away any time soon, especially when the goal of the threat actor is espionage. As defenders and administrators, we need to understand how those tools we use daily can be turned to against us to compromise out networks and we need to know how to spot those activities which may, at first glance, look normal but do not stand up under scrutiny.
Attacks like this, at the highest level are not going to stop any time soon, we must be ready and warmed up before they hit us next.
For information on how these threats manifest in Vectra’s threat detection platform, read our support article.