Full IT Compromise, Contained by Segmentation

The Challenge

A US municipal water and wastewater utility engaged their managed security services provider for external and internal penetration testing. The MSSP brought us in as the technical delivery partner. A water utility is critical infrastructure in the most literal sense. It runs a corporate IT environment like any other organization, payroll, email, customer billing, a public website, and it runs an operational technology (OT) environment that controls the physical process of treating and delivering drinking water. The two worlds are very different, and the line between them is the thing that matters most.

The question we were really there to answer was not “can the IT network be breached.” Most networks can. The question was: if an attacker takes the corporate side, do they reach the plant? For a water utility, that distinction is the difference between a bad week and a public safety event. A ransomware crew in the billing system is expensive. The same crew on the SCADA network that controls chemical dosing and pumping is a different category of problem.

The utility had made real investments, and the MSSP needed a partner who would test the environment the way an actual adversary would: chaining findings into a single attack path rather than handing back a list of unrelated issues, and reporting honestly on what held up as well as what fell. The MSSP’s name was the one the client saw. Our job was to make the testing under that name as rigorous as anything a larger firm would deliver, while the MSSP kept the relationship they had built.

The Approach

We tested the external perimeter first, then moved to an internal assumed-breach scenario, following the PTES methodology we use on every pentest: intelligence gathering, threat modeling, vulnerability analysis, exploitation, and post-exploitation.

The external phase was the good news. Open-source intelligence and DNS analysis surfaced the usual housekeeping items: a DMARC record set to a policy of none rather than quarantine or reject, a handful of database and mail services exposed to the internet that did not need to be, an IIS shortname disclosure, and web servers sitting directly on the internet with no web application firewall in front of them. None of these gave us a way in. We documented them as the perimeter cleanup they were, and assessed the external posture as sound. The public-facing attack surface was not the story; the internal network was.

From an assumed-breach foothold, we started with the techniques an attacker runs in the first hour. We deployed Responder to poison name-resolution traffic and capture authentication, and it came back empty. That is rare, and it told us the client had hardened their workstations against a very common attack. So we kept going. Anonymous SMB and LDAP queries against the domain controllers returned the full user list, group membership, and the password policy without any credentials at all. We intercepted and downgraded Kerberos authentication on the local segment to capture hashes for offline cracking.

The first real break came from a tired habit. Testing the recovered user list with each account’s own username as its password returned one hit: a low-privilege service account whose password matched its name. That single weak credential was the hinge. With authenticated access, we requested service tickets for accounts configured with a Service Principal Name and cracked them offline (Kerberoasting). We crawled the file shares that account could read and found what file shares in long-lived environments almost always contain: configuration files with database connection strings, scripts with credentials hardcoded into them, and exported network device configs sitting in cleartext.

From there the chain built quickly. Database credentials pulled from a config file gave us system administrator rights on several Microsoft SQL servers, and from one of them we used the xp_cmdshell stored procedure to run operating system commands on the host. Exported switch and firewall configs held passwords stored in Cisco’s type 7 and “secret 5” formats, both of which are weak by design and trivially reversible; we recovered the plaintext for all of them. We found Cisco Smart Install listening across the switching fabric, a service CISA has flagged for years because it lets an unauthenticated attacker pull device configurations over the network. One unpatched Windows host was still vulnerable to EternalBlue (MS17-010), which gave us SYSTEM on the box and a place to dump credentials from memory.

Each captured credential opened the next door. We reached a Domain Admin account, used its hash to authenticate to a domain controller, and extracted the domain credential database (NTDS.dit), the hashed passwords for every account in the IT domain. At that point the corporate Active Directory environment was fully compromised.

For a lot of pentest firms, that is the finish line. Domain Admin is the trophy, the report gets written, everyone goes home. We kept pressing, because Domain Admin on the corporate network was never the real objective for a water utility. The plant was. So we used that access the way an attacker who wanted to disrupt water treatment would: hunting for more systems to move into, mapping the network for paths between segments, and pushing for any route that crossed from the IT side into the OT environment. There was plenty of low-hanging fruit left inside the corporate network. The boundary to the OT side was a different story. The segmentation stopped us in our tracks. No trust path, no reachable jump host, no forgotten dual-homed server. It was an architecture and an implementation that proved its value under exactly the pressure it was built to withstand.

The Outcomes

We documented the full path as a single kill chain, not a pile of findings. Every link was written up with the misconfiguration that created it, the technique we used, the control that should have caught it, and the specific fix that breaks that link, all mapped to the CIS Top 18 and the CISA Cross-Sector Cybersecurity Performance Goals so the MSSP and the utility could track remediation against frameworks they already used.

The most important finding was what we could not do. With full control of the corporate domain and Domain Admin credentials in hand, we could not reach the OT network. The SCADA environment lived in a separate, segregated domain with no trust path from the IT side, and that separation held under a full corporate compromise. For a water utility, that is the control that matters more than any other, and it worked exactly as it should have. We said so plainly in the report. A pentest that only lists failures gives a defender no way to know which of their investments are paying off. This one was.

Other controls held too, and we called them out. Workstations were hardened against the name-resolution poisoning that usually yields the first credentials. Active Directory Certificate Services, a frequent source of privilege-escalation paths, was configured without the common flaws. Group Policy blocked the session-hijacking technique we tried against an active administrator session. These were signs of a team that had been doing real work, and the report reflected that.

The remediation list was concrete and ordered by what broke the chain fastest: kill the cleartext credentials in file shares and config files, retire the weak Cisco type 7 and secret 5 passwords, disable Smart Install across the switching fabric, patch the EternalBlue host, put the web servers behind a WAF and close the unneeded internet-facing ports, and tighten the service account whose username-as-password started the whole thing. None of these is exotic. Chained together is exactly how a real intrusion would have unfolded, and breaking any early link stops it.

We did not disappear after delivery. We worked the priority order with the MSSP and the utility’s team, answered the technical questions on specific fixes, and stayed available for the conversations where IT leadership had to explain to non-technical decision-makers why each item mattered. The MSSP kept their client relationship having delivered a demonstrable security improvement, and the utility closed a real attack path while learning, with evidence, that the one boundary they most needed to hold did hold.

This is the kind of work automated scanning cannot do. A scanner flags a weak password and a missing patch as separate medium-severity lines. Only a human with access, time, and motivation turns them into a path to the domain controller, and only a human can tell you which of your defenses actually stopped the attack. If you run an MSP or MSSP and want to deliver pentests like this for your own clients, our partner program supports referral and resell models; the partner program post for MSPs, MSSPs, and VARs walks through how it works.