Chasing Ghosts & The Small Business Gap

I've worked probably 50+ incident investigations in my time and there is a particular psychological phenomenon that I've seen occur multiple times, and even experienced myself early on, which I call Chasing Ghosts.

Chasing Ghosts is what happens when individuals and organizations experience a compromise, work around the clock, and start to see every event as a part of the compromise. They will inflate the sophistication of the attacker and see "ghosts" everywhere. I've seen this happen with red team assessments as well.

The Investigation

Recently, I got a call from a local company indicating that they believed they had been compromised based on the fact that their IT services provider had experienced a phish and ransomware against one of their other customers. As a precaution they disconnected their corporate network and shut their business down so as to not put customer data at risk or allow the potential compromise to spread. They called me in to perform a Breach Assessment (BA) in order to determine if they had actually been hacked.

Normally, I provide IR and BA to very large corporate customers in other states which cost between 50-100k depending on the complexity, my schedule, and other factors. I have seen other IR firms charge between 350 - 700$ per hour depending how busy they are. In my state, almost no one could possibly afford that. I had some downtime, and I care about my local community, so I decided to take it on for about 1/10th of my normal rate.

The customer decided that this was not going to be a law enforcement related forensics activity and so none of what I describe here will follow that level of forensic principals such as chain of custody, preservation of state, hard drive cloning, etc.

I went onsite and analyzed the customers network, servers, laptops, and desktops. I did not review cell phones or other BYODs in this case. I collected a ton of data including:

• A questionnaire to the customer regarding their environment, the incident, and concerns.
• Full packet capture at both the network and host level. I have a suite of tools that simulate an internet connection which I apply to each host during collection.
• Registry, File System, Process Table, and Process to Network events for a period of time on each host.
• Full OS memory dumps
• Server & host logs
• I also interviewed staff and recorded what they had observed.

We used out of band communications for everything incase their email or other systems were being monitored.

I looked for:

• Beacons
• Malware / C2
• Scans
• Data Exfiltration
• ARP spoofing & other network level anomalies
• Exploits
• Unusual user behavior
• Suspicious logs (services going up and down, unexpected logins, LSASS issues)
• Disablement of security software

And much more.

Any suspicious files were statically and dynamically analyzed in depth, including assembly level reverse engineering:

We have a number of in-house written tools that we use for analyzing PCAPs (example: https://github.com/redcrowlab/rcPCAPscan), process table "maliciousness" classification, file format parsers, etc. which we ran on all of the collected data as well as manual analysis.

Nothing was found.

Going to the Source

I talked with the IT services provider to find out what they were seeing and gather some indicators of compromise I could use to find the attackers. They had been working 20 hour days, were losing customers fast, and experiencing extreme stress. The evidence that a compromise had occurred was that one of their customers had received and clicked on a phishing email, had some of their files encrypted, and received a ransomware pop up. However, by the time I was involved, none of that evidence was available as it had not been retained.

The IT provider was positive they had also been compromised. When I asked them what indicators they had, these included:

• Screen flicker
• Phone acting strange
• Laptop crash / malfunction

No screenshots, logs, or other evidence had been retained for these suspected compromised devices. They told me stories of how they believed that the malware hopped from a compromised windows computer, to an ubuntu laptop, on to an external exfat USB drive, and then from there to an iPhone, all disconnected from each other and on separate networks or stand alone.

I asked for the compromised drive and one of the positively compromised laptops and ran my Breach Assessment process on them. If there was a cross platform attack tool that could hop airgaps, cause laptops to transmit to cell phones, and evade standard detection tools, I wanted a copy of it. Unfortunately, I found no evidence of any malicious activities, although I did find a corrupted partition.

I believe that the stress, lack of sleep, fear over the actual compromise, lead to a "chasing ghosts" scenario. Very unfortunate for all involved.

The Antidote

I have seen this sort of thing happen before in other organizations, and with myself when I first started. There is a simple formula for preventing it:

1. Sleep. If a suspected incident occurs, set up the organization into shifts, work 8 hours, and hand off to the next shift so everyone gets adequate rest.
2. Empiricism. Meticulously collect and retain all evidence. Make no assumptions or conclusions without hard evidence to back them up.
3. Skepticism. Be skeptical of everything, question each step, have other eyes double-check your work, and document extensively.

Resuming Operations

We had our customer temporarily sever connections with their IT provider and got them back up and running. We closely monitored everything for about a week to see if any signs showed up, and none did. I gave them an 80% confidence assessment, based on the evidence, that they had not been compromised. We recommended several changes to the environment and that they change all passwords.

The Small Business Gap

The week of downtime they suffered impacted all of their customers, probably 40 or so other small local businesses. The IT service provider had probably 100 other customers who were also affected, as well as their customers. You can see how this could quickly spiral out of control and cause significant damage to the local economy.

This got me thinking about the fact that attackers have significantly broadened their target scope, and its not just fortune 100 companies and governments at risk anymore. Small business are now a target, but we have a major shortfall in security companies that can support them.

There are very few security companies in my state and most of them work for government or out of state customers. If I can make 50k for a 1-2 week engagement in California, why would I work for 5k in New Mexico? However, if our local CPAs, Lawyers, Construction Companies, Credit Unions, get hacked, what happens to our local economy? What if I use one of these services for my business, how does this affect me? This problem exists in many states and cities around the country.

These small business that make a few hundred thousand dollars a year cannot afford the market cost of breach assessments, incident response, effective penetration tests, etc.

This is a serious problem.

Possible Solutions

I've been brainstorming how to approach this problem and I don't really like what I've come up with but I decided to put it out here in case anyone can add to it. Here is what I have thought of:

1. Security companies allocating a % of pro-bono (or massively discounted) hours for local businesses in distress.
2. The creation of a non-profit that solicits donations and whose mission is to help small businesses with prevention and response.
3. A sort of co-op design where every business pays a certain amount per month which forms a pool that can be drawn on for emergencies.
4. Education of local individuals willing to take on this level of work.

I can contribute immediately to #4 as I have a good amount of training materials created which I can share with my local community. I can contribute occasionally to #1, but only occasionally.

Does anyone have other ideas on how to address this problem?

Thanks for reading,

A.