Ransomware Incident Response for a Global Manufacturer

An international manufacturer, referred to here as ANYCORP.IO to protect its identity, lost nine of its ten Active Directory domain controllers to ransomware in July 2019. Its managed service provider’s incident response team and ADAMnetworks replaced the gateways at every factory with adam:ONE® in Zero Trust Connectivity (ZTc) configuration overnight. Mission-critical operations were back by Monday morning. The ransom was never paid.

At a glance

Timeline

January 2019: the attack begins

The initial attack arrived as a phishing email attachment that launched a PowerShell script. The attackers used TrickBot to establish a persistent connection. Over the next six months the advanced persistent threat spread to systems at every one of the company’s global locations, undetected by the McAfee Enterprise software ANYCORP.IO used and by its managed Cisco and McAfee services. By the time it was discovered, 400 of the company’s more than 3,000 internet-connected devices were infected.

July 2019: discovery

The malware made ANYCORP.IO’s systems increasingly unstable. That instability was the first indicator of compromise, and it led the IT team to ask its managed service provider for help with mounting network infrastructure problems. The MSP identified potential indicators of compromise and brought in the ADAMnetworks team.

The MSP’s incident response team warned the client that it was probably in the final stages of a ransomware attack. The C-suite was not convinced.

At 6 p.m. on a Friday, as the teams were packing up, they watched file extensions change on the hypervisor infrastructure: ransomware encryption had started. The IT team ran to the data center and found it locked, with no key available. Just before they broke down the door, the janitor arrived with a key. The team pulled the Ethernet and power cables from every piece of equipment in the room.

Back at a non-critical workstation that had not been shut down, the ANYCORP.IO, MSP and ADAMnetworks teams watched for attacker activity and found a ransom note. Nine of the ten global Active Directory domain controllers were already encrypted.

Containment

The team immediately took a snapshot image of the one Active Directory controller that had not been encrypted. It then removed the existing Cisco ASA gateways and site-to-site VPN infrastructure at all five factories and replaced them with adam:ONE.

Each adam:ONE deployment ran in ZTc configuration, which produced an automatic inventory of every device on the ANYCORP.IO network. A “Holding Tank” policy quarantined every device by default, so no device could communicate outside the network:

• Internal systems could reach only Windows Update and Webroot EDR.
• The attackers lost access to Active Directory. Every device went radio silent to every command-and-control channel, so they could do no further damage.

Don’t Talk To Strangers (DTTS)® severed every connection that did not begin with a policy-allowed DNS resolution, which defeated circumvention attempts by default. ZTc Adaptive AI™ built the dynamic allowlists needed to bring critical and ancillary services back.

The teams worked through the night and had the situation under control by morning. Tape backups went to a third party to recover as much data as possible.

The ransom decision

The ransomware group demanded US$600,000, which would grow by US$120,000 for every day it went unpaid after a one-week deadline. Under that pressure, the IT director wanted to pay in the hope of getting systems back faster.

The MSP and ADAMnetworks advised against paying because:

• There was no guarantee the decryption keys would work.
• More than one threat actor might be in the network, making re-infection likely.
• Restoring data without addressing the technology debt and vulnerabilities would leave the company open to the next attack.
• Paying would perpetuate the ransomware problem for the industry.

The C-suite agreed not to pay, approved a per diem rate for remediation, and the MSP and ADAMnetworks recruited a 20-person cleanup crew.

Recovery

The incident response team prioritized the systems to restore first and coordinated the cleanup with teams at every site:

• Every endpoint was inspected and scanned in safe mode.
• Clean devices moved out of the Holding Tank quarantine into the ZTc environment. Radio silence stayed in effect, so returning devices could not be re-infected or attacked anew.
• The adam:ONE live log showed which systems were still trying to reach the command-and-control servers.
• Shadow IT discovered during recovery was scheduled for later technology debt remediation.
• The on-premises Exchange server turned out to be infected, so mail delivery moved to the cloud to receive new messages. Lost email was restored from backups later.

Mission-critical operations were running again by Monday morning.

Tools used

• adam:ONE® in ZTc configuration:
  • Don’t Talk To Strangers (DTTS)®
  • ZTc Adaptive AI™ dynamic allowlisting
  • Automatic device inventory
  • Automatic device quarantine
  • Custom rules
  • adam1.tools live log
• Webroot endpoint protection

Results

• Total downtime was contained to a single weekend, and only one weekend production shift was lost.
• Over the following three months the critical technology debt was addressed, and the ZTc model substantially improved the company’s security posture.
• Remediation and the technology debt work together cost less than the original ransom demand, without the risks of paying it.
• OT system controllers still running Windows XP were too expensive to upgrade. Network isolation by adam:ONE ZTc mitigated their risk out of band, and the same devices remain in operation without re-infection.

Where things stand

adam:ONE remains in place at each of the major factories across three continents. The same command-and-control servers are still used by threat actors trying to re-infect the network, mostly through phishing. In the three and a half years after the attack, more than 5,000 re-infection attempts were tracked. Because both known and unknown malicious egress traffic is blocked, every one has been neutralized.

Some OT control systems at ANYCORP.IO still run software that is inherently vulnerable, because upgrading OT is often cost-prohibitive. With adam:ONE in place, the risk of another attack of this kind has been reduced to near zero.