8.8 — What Actually Went Wrong

Almost none of these began with a novel technique. They began with an unpatched library, a credential in a repository, a supplier's account, or an endpoint nobody remembered was reachable.

Read each one for the same two things: the mechanism, and the control that would have stopped it — because most of those controls are in the previous seven chapters, and most are unglamorous.

1. Equifax, 2017 — a patch, and a certificate

What happened. Attackers exploited a known vulnerability in Apache Struts (CVE-2017-5638) on a public-facing complaint portal. A patch had been available for over two months. From that foothold they moved through an internal network that had little segmentation, found unencrypted credentials, and queried dozens of databases for about ten weeks. Roughly 147 million people's records were taken.

The detail that makes it worth studying. The organisation ran a device that inspected outbound encrypted traffic. Its certificate had expired months earlier, so the traffic had gone uninspected the whole time. When the certificate was replaced, the exfiltration was visible immediately.

What would have stopped it. Automated patching of a known vulnerability (Chapter 8.1's priority list). Segmentation, so a portal could not reach dozens of databases (Chapter 8.6.2). Certificate expiry monitoring (Chapter 8.3.1). Egress inspection that was actually working.

2. Capital One, 2019 — SSRF into the metadata service

What happened. A misconfigured web application firewall was exploited via server-side request forgery to reach the cloud instance metadata service, which returned the temporary credentials of the role attached to the instance. Those credentials had permission to list and read storage buckets. About 100 million records were taken.

Read this one against Chapter 8.5.2, because it is the exact shape of the diagram on that page.

What would have stopped it. IMDSv2, which requires a PUT to obtain a token and cannot be satisfied by a simple SSRF GET — a one-line configuration. A role scoped to the buckets that component actually needed, rather than broad read access (least privilege, Chapter 8.1). Egress rules blocking the link-local address. Alerting on unusual bulk reads.

3. SolarWinds, 2020 — the build system

What happened. Attackers compromised the build environment for a widely used network management product and inserted a backdoor into the source before compilation. The resulting update was signed with the vendor's legitimate key and distributed through normal update channels to thousands of organisations, including government agencies. The malicious code lay dormant, checked its environment, and communicated over channels designed to look like the product's own traffic.

Why it changed practice. Every downstream control worked: the signature was valid, the vendor was legitimate, the update channel was correct. The compromise was upstream of everything that could have detected it, which is what made supply-chain provenance a first-class concern.

What it produced. SLSA provenance, artefact signing with Sigstore, reproducible builds, and treating the build pipeline as a high-value target (Chapter 8.6.2).

4. Log4Shell, 2021 — a logger that evaluated its input

What happened. A widely used Java logging library evaluated ${jndi:ldap://…} expressions inside strings it logged, fetching and executing a remote object. Any application that logged attacker-controlled text — a user agent, a username, a chat message — could be made to run arbitrary code. The library is embedded in thousands of products, often indirectly.

The lesson generalises beyond Java. Chapter 8.5.1 states it: any component that evaluates its input is an injection sink, and a logger is the last place most people look.

What separated the organisations that recovered quickly. Not patching speed — inventory. Teams that could answer "where does this library exist in our estate" acted in hours; teams that could not spent days searching. That is the argument for an SBOM (Chapter 8.6.2).

5. Target, 2013 — a supplier's credentials

What happened. Credentials belonging to a heating and ventilation contractor were used to reach an externally facing vendor system, and from there the attackers moved into the internal network and eventually onto point-of-sale terminals, where malware captured card data in memory. Around 40 million card records were taken.

The recurring theme it established. Third-party access is your attack surface. A vendor account with network reach is equivalent to an employee account, and it usually has weaker controls, no MFA, and nobody reviewing it.

What would have stopped it. MFA on vendor access. Segmentation between a vendor portal and the payment network. Reviewing third-party accounts as rigorously as employee accounts.

6. Uber, 2016 and 2022 — the same lesson twice

2016. Credentials in a private source repository gave access to cloud storage containing data on roughly 57 million users. The company paid the attackers and characterised it as a bug bounty; the concealment led to a criminal conviction of the security executive involved.

2022. An attacker obtained a contractor's password, then used MFA fatigue — repeated push prompts plus a message posing as internal support — until it was approved. Once inside, they found a script on an internal share containing hardcoded administrative credentials for a privileged access management system.

The pattern across both. A stored credential turned a limited foothold into broad access. Chapter 8.6.1 is the direct response, and Chapter 8.4.7's number matching is the direct response to the second.

And a governance lesson: the 2016 concealment did more damage than the breach. Disclosure obligations (Chapter 8.7) are not optional, and covering up is a personal legal risk.

7. Okta and Lapsus$, 2022 — the support desk as a path

What happened. Attackers gained access to a laptop belonging to an engineer at a third-party customer support provider, and used that support tooling's session to act against a limited number of customer tenants.

Why it matters structurally. Identity providers are the highest-value target by definition (Chapter 8.4.6): compromising one reaches every application behind it. And the support path is often the weakest one into a strong system — an idea Chapter 8.4.1 raises about account recovery, here applied at the vendor level.

What it produced. Harder scrutiny of sub-processors, restrictions on support tooling access, session recording, and customers asking who at a vendor can act inside their tenant.

8. LastPass, 2022 — the second breach used the first

What happened. An initial compromise obtained source code and technical information. That knowledge was then used to target a small number of engineers with production access, in one case through a vulnerability in third-party media software on a personal home computer. The attackers reached development and cloud storage credentials, and exfiltrated backups of customer vaults — encrypted, but taken.

Two engineering consequences.

The offline attack becomes the whole defence. Once encrypted data is exfiltrated, only the key derivation function stands between the attacker and the contents (Chapter 8.4.1). Accounts with old, low iteration counts were substantially weaker than newer ones — which is exactly why rehash-on-login matters, and why it is worth doing before you need it.

The boundary is the person, not the office. A home machine with production access is production. That is a genuinely hard problem and the reason for managed devices, conditional access requiring device compliance (Chapter 8.4.9), and just-in-time rather than standing production access (Chapter 8.4.10).

9. Colonial Pipeline, 2021 — one password, no second factor

What happened. A legacy VPN account that was no longer in active use, protected by a password with no multi-factor authentication, was accessed — the password appeared in a credential dump, consistent with reuse. Ransomware followed, and the operator halted fuel distribution across a large part of the eastern United States.

The simplest lesson on this page. An unused account with a reused password and no second factor. Account inventory and MFA everywhere (Chapter 8.1's first item) would have prevented one of the most disruptive attacks in recent history.

10. Heartbleed, 2014 — the bug with no evidence

What happened. A missing bounds check in OpenSSL's TLS heartbeat handling let a client request more data than it had sent, and the server returned adjacent memory — potentially containing private keys, session tokens and passwords.

What made it unusually bad. Exploitation left no trace in logs. Nobody could determine whether their keys had been taken, so the only safe response was to assume the worst and rotate every certificate and credential globally.

The lasting lessons. Memory-safety bugs in C libraries that are everywhere have enormous blast radius (an argument that recurs in Chapter 3.4's discussion of Rust). And a design that leaves no evidence forces you to assume the worst — detectability is a design property.

11. Marriott / Starwood, 2018 — inherited, and undetected

What happened. Attackers were present in Starwood's reservation systems from around 2014. Marriott acquired Starwood in 2016 and the intrusion was discovered in 2018 — around four years of undetected access affecting hundreds of millions of guest records, including passport numbers.

The lesson. An acquisition acquires the acquired company's compromises. Security due diligence is part of due diligence, and integrating an estate before assessing it inherits everything in it. It is also the clearest illustration of the dwell-time problem: prevention was long past, and detection was the only remaining control for four years.

12. MOVEit, 2023 — one vendor, thousands of victims

What happened. A SQL injection vulnerability in a widely used managed file transfer product was exploited as a zero-day at scale, leading to data theft and extortion across thousands of organisations — governments, universities, banks — many of whom did not know they used it, because a supplier did.

Two things it demonstrates. A vulnerability in a widely deployed product is a mass event, not a single incident. And you inherit your suppliers' vendor choices, which is why the inventory in Chapter 8.7 must include third parties and their sub-processors.

And it is a reminder that injection (Chapter 8.5.1) is not a solved historical problem — it was the mechanism in one of the largest incidents of the decade.

13. Optus, 2022 — an endpoint with no authentication

What happened. An externally reachable API endpoint served customer records without requiring authentication, and identifiers were sequential and therefore enumerable. Data on millions of customers was retrieved.

Why it is on this list. It is the simplest failure possible: broken access control, which is A01 in the OWASP list and the subject of Chapter 8.4.10. No exploit, no malware, no cleverness — an endpoint that was reachable and did not check.

14. The pattern

Recurring cause	Chapter
Known vulnerability, unpatched	8.1 — automated updates
Credential in code or on a share	8.6.1
No MFA, or a phishable second factor	8.4.1, 8.4.7
Flat network, no segmentation	8.6.2
Excessive permissions on a role or account	8.1, 8.4.10
Third-party or supplier access	8.6.2, 8.7
Missing or unchecked authorisation	8.4.10
Undetected for months	8.1, 8.7 audit logging

Almost nothing here required an original technique, and that is the finding worth carrying out of this Part.

The useful exercise is to read the list as a set of questions about your own system, honestly:

• Would a known vulnerability in a dependency be patched this week, without anyone remembering to?
• If the web tier were fully compromised, what would it reach?
• Is there a credential in source control right now? How would you know?
• Does every administrator have a phishing-resistant second factor?
• Which supplier accounts can reach your systems, and when were they last reviewed?
• Could you enumerate every place a given library is used, in an hour?
• If an attacker had been reading your database for three months, what would show it?

The last one is the one most systems fail, and it is the difference between the incidents above being a bad week and being a bad year.

What the interviewer will push on

"Pick a breach and explain what went wrong." Choose one you can walk mechanically end to end. Capital One is the strongest choice for a backend role: SSRF to the metadata service, temporary credentials, bulk storage read — then the controls, with IMDSv2 and a narrowly scoped role named specifically. Vague recall of a headline is the weak answer.

"What did SolarWinds change?" It moved the concern upstream of the artefact. Every downstream check passed because the signature was genuine — the build system was compromised. It is why provenance (SLSA), artefact signing and treating CI as a high-value target became standard rather than aspirational.

"Why was Log4Shell so hard to respond to?" Not patching — inventory. The library was embedded transitively in thousands of products, so the hard question was where it existed at all. That is the concrete argument for an SBOM, and it is what separated fast responders from slow ones.

"What does the LastPass incident tell you about password hashing?" That once encrypted data is exfiltrated, the key derivation function is the only remaining defence — so iteration counts and rehash-on-login are not theoretical hygiene. It also shows the boundary is the person: an engineer's home machine with production access is production.

"What is the common thread across these?" Known vulnerabilities, stored credentials, missing MFA, flat networks, excessive permissions, supplier access, and long dwell times. Almost none required a novel technique, which reorders your priorities away from exotic defences and toward the unglamorous ones.

"How would you know if this were happening to you now?" The honest answer names detection: central authentication and authorisation-denial logs the compromised host cannot edit, alerting on bulk data reads and unusual KMS decrypt volume, egress monitoring, and a tested incident plan. Most organisations cannot answer this, and saying so plainly while naming what you would build is stronger than claiming otherwise.

One thing to volunteer: point out the Equifax certificate detail — outbound inspection had been blind for months because a certificate expired, and the exfiltration became visible the moment it was replaced. It is the clearest example of a security control that existed, was believed to be working, and was not. Controls need their own monitoring, and that observation applies to every control in this Part.

Recall

• Equifax (2017) — unpatched Struts, a flat internal network, and outbound inspection blind for months because a certificate had expired. Controls need monitoring of their own.
• Capital One (2019) — SSRF → metadata service → role credentials → bulk storage read. IMDSv2 and a narrowly scoped role would have stopped it.
• SolarWinds (2020) — the build system was compromised, so the malicious update was genuinely signed. Every downstream check passed. It produced SLSA, artefact signing, and CI as a protected target.
• Log4Shell (2021) — a logger that evaluated its input. The response bottleneck was inventory, not patching — the argument for an SBOM.
• Target (2013) and MOVEit (2023) — third-party access and third-party software are your attack surface, including suppliers you did not know you had.
• Uber (2016, 2022) — a stored credential turned a foothold into full access, twice; MFA fatigue was the way in the second time. The concealment in 2016 caused more damage than the breach.
• LastPass (2022) — exfiltrated encrypted vaults meant the key derivation function was the only remaining defence, so old low iteration counts mattered. An engineer's home machine with production access is production.
• Colonial Pipeline (2021) — one unused VPN account, a reused password, no MFA. Optus (2022) — an unauthenticated, enumerable endpoint. Marriott (2018) — four years undetected, inherited through an acquisition.
• The pattern: known vulnerabilities, stored credentials, missing MFA, flat networks, excessive permissions, supplier access, long dwell time. Almost none required a novel technique.

Self-test: Which control existed at Equifax but was silently not working? · What single configuration would have stopped Capital One? · Why did every downstream check pass for SolarWinds? · What was actually slow about the Log4Shell response? · What did the LastPass incident prove about iteration counts? · Of the seven questions in section 14, which can you answer about your own system?

Next: Part 12 turns to the other half of modern engineering — how machine learning actually works, from a single neuron to the transformer, and what building on top of a language model really involves.