8.6.2 — Boundaries, Service Identity and the Supply Chain

An attacker exploits an outdated library in a public-facing web server. That server can reach the database, the internal admin tool, the message broker, the build server and every other application — because everything is in one network and the firewall only guards the perimeter.

One vulnerability became total compromise, and nothing about the second step required skill. This chapter is about making the second step hard: boundaries between workloads, identity between services, and control over the code that arrives inside your images.

1. Boundaries

Firewalls and cloud security groups are stateful — allow a connection outbound and the return traffic is permitted automatically (Chapter 5.10 derives this properly). Default deny, then allow the specific flows you need, and prefer rules expressed in terms of which service rather than which IP address, because addresses change and services do not. Cloud security groups can reference each other for exactly this reason.

Egress filtering is the underused half. Most estates control what comes in and let anything out. Restricting outbound traffic:

• Blocks command-and-control, so a compromised workload cannot reach the attacker's server.
• Blocks exfiltration by the direct route.
• Neutralises SSRF (Chapter 8.5.2) — including in dependencies you did not write.

The objection is that it breaks things, and it will: a workload that talks to three known endpoints is easy, one that fetches arbitrary URLs is not. Start with the highest-value workloads, log first in report-only mode, then enforce.

Segmentation limits the blast radius of the first compromise. The classic tiering — public, application, data, with each layer only reachable from the one above — is the minimum. Microsegmentation goes per workload, which is what Kubernetes network policies express:

kind: NetworkPolicy
spec:
  podSelector: { matchLabels: { app: orders } }
  policyTypes: [Ingress, Egress]
  ingress:
    - from: [{ podSelector: { matchLabels: { app: gateway } } }]   # (1)
      ports: [{ port: 8080 }]
  egress:
    - to: [{ podSelector: { matchLabels: { app: postgres } } }]     # (2)

(1) Only the gateway may reach orders. (2) Orders may reach only the database. Note the crucial default: with no policy at all, Kubernetes allows everything — a cluster without network policies is one flat network, which is the opening story with extra steps. Apply a default-deny policy first, then allow.

Bastion hosts are being replaced. A jump host is a standing target with an open port. Cloud session managers and identity-aware proxies (Chapter 8.4.10) give shell access with no inbound port at all — the agent on the instance opens an outbound connection, access is authorised against your identity provider, and every session is logged and recordable. If you are still running an SSH bastion with a public IP, that is the thing to replace.

VPN versus zero trust, briefly: a VPN grants network position, and network position historically implied trust — which is exactly the assumption that makes the opening story fatal. Zero trust (Chapter 8.1) authenticates and authorises every request regardless of where it came from. In practice most estates run both, and the useful move is to stop treating "inside the VPN" as an authorisation decision.

2. Service-to-service authentication

Three mechanisms, and choosing between them is a common design question.

	API keys	OAuth client credentials	mTLS
What is presented	A shared string	A signed token	A certificate + proof of key
Stolen credential usable elsewhere	Yes	Yes (bearer)	No
Expiry	Rarely	Minutes	Certificate lifetime
Rotation	Manual	Automatic	Automated PKI
Needs	Nothing	An identity provider	A certificate authority
Common failure	Logged, shared, never rotated	Token theft, aud unchecked	Expiry outage

API keys are the weakest, and the reasons are specific rather than aesthetic: the same string is held by both parties, it usually never expires, it is frequently logged (in a URL, in a header dump, in an error report), and rotation requires a coordinated change. They are acceptable for low-value integrations and should not authenticate anything important.

OAuth client credentials (Chapter 8.4.3) give short-lived tokens with automatic rotation, centralised revocation and scopes. The remaining weakness is that a token is a bearer credential: whoever holds it can use it, from anywhere.

mTLS is the strongest because the credential is a private key that never moves. Possession of the certificate is not enough — you must prove possession of the key during the handshake, so an intercepted certificate is useless. Chapter 5.7 covers the handshake and Chapter 8.3.1 the certificate machinery.

The honest cost of mTLS is operational: you run a certificate authority, distribute certificates to every workload, and rotate them before expiry. An expired internal certificate is a total outage, and it is a common self-inflicted incident. So mTLS is right when it is automated — a service mesh, or a system issuing short-lived certificates from workload identity — and a poor idea when it is hand-managed.

SPIFFE gives workloads a platform-verified identity (spiffe://example.com/ns/prod/sa/orders) and SPIRE issues short-lived certificates for it, which is the general form of Chapter 8.6.1's answer to secret zero. A service mesh does the same inside a cluster and gives mTLS between all services with no application code changes — which is its strongest single argument.

And the general principle behind the table: prefer proof of possession over bearer credentials. A bearer token works for whoever holds it; a proof-of-possession credential requires demonstrating you hold a key you never transmit. mTLS is the mature form; DPoP is the emerging HTTP-level one.

3. The supply chain

Your application is mostly other people's code. A typical Node or Python project pulls in hundreds of transitive packages, and each one runs with your application's privileges.

The attack shapes:

Typosquatting — a package named one character away from a popular one.

Dependency confusion — 2021 research showed that if a package manager is configured with both a private registry and the public one, publishing a higher version number of an internal package name to the public registry can cause build systems to fetch the attacker's copy. It worked against many large companies. The fix is scoped names and registry configuration that never falls back to public for internal scopes.

Maintainer compromise — a popular package's account is phished and a malicious version published. This has happened repeatedly to widely used packages.

Protestware and abandonment — a maintainer deliberately breaks or sabotages their package, or simply stops maintaining a package that a million projects depend on.

Build-time execution — install scripts run arbitrary code during npm install, before any of your tests. Consider --ignore-scripts with an allow-list, which is more practical than it sounds because few packages genuinely need one.

The defences that work:

Lockfiles with integrity hashes, committed and enforced with npm ci or its equivalent (Chapter 3.10). A republished version with different content then fails to install — this is content addressing (Chapter 8.2.2) doing security work.

Pin what you can. Exact versions for applications; ranges are for libraries. In CI, pin third-party actions by commit hash, not by tag — tags are mutable and can be repointed at malicious code.

Automated updates. Chapter 8.1's priority list puts this near the top, because known vulnerabilities in outdated dependencies remain the most exploited category in the real world. A bot that opens update pull requests beats a quarterly review.

Reduce the surface. Every dependency is a decision. A one-line utility package is not worth a supply-chain risk; a minimal or distroless base image removes hundreds of packages from a container that never needed a shell.

An SBOM — a software bill of materials, in SPDX or CycloneDX format — lists everything in an artefact. Its value is answering "are we affected" in minutes rather than days, which is the question that dominates the first hours of an event like Log4Shell.

Signing and provenance. Sigstore and cosign sign artefacts using short-lived certificates tied to an identity, removing the long-lived signing key that used to make signing impractical. SLSA is the framework for provenance: a graded set of requirements about how an artefact was built — scripted build, authenticated provenance, hardened build platform — so a consumer can verify that a binary came from the source it claims.

4. The build pipeline is a target

CI has credentials to everything: source, registries, cloud accounts, production. It is a more valuable target than any single application, and it is often the least hardened system in an estate.

The controls that matter:

Use OIDC federation instead of long-lived cloud keys. The pipeline presents a signed token proving which repository and branch it is running for, and exchanges it for short-lived cloud credentials. There is then no cloud key stored in CI at all — the same workload-identity idea as Chapter 8.6.1.

Never expose secrets to pull requests from forks. A contributor's pull request can otherwise run arbitrary code with your credentials. This is a documented and repeatedly exploited path.

Separate build and deploy privileges. A build job needs no production access.

Pin actions and images by digest, review who can modify pipeline definitions, and require approval for changes to deployment workflows.

Protect the artefact between build and deploy — sign it, and verify the signature at admission, so a registry compromise does not become a production compromise.

5. Runtime hardening

Run as a non-root user, with a read-only root filesystem and writable volumes only where genuinely needed. This turns "attacker achieved code execution" into "attacker cannot install anything or persist".

Drop Linux capabilities to the minimum, and apply a seccomp profile restricting available system calls (Chapter 2.8 covers the mechanism).

Scan images, and enforce at admission — a policy that refuses to run an image with critical vulnerabilities or without a valid signature is the control; a scan report nobody acts on is not.

Then be honest about triage. Most images report dozens of vulnerabilities, most of which are unreachable from your code. Prioritise by reachability and exposure, not by count, or the team learns to ignore the scanner — which is the actual failure mode of image scanning programmes.

Detect at runtime. Alert on a shell spawning inside a container, an unexpected outbound connection, or a write to a path that should be read-only. These are high-signal events precisely because they should never happen in a well-defined workload, which is a benefit of the constraints above rather than of the detector.

What the interviewer will push on

"How do you limit the damage of one compromised service?" Segmentation with default deny, per-workload rules rather than tiers where possible, and egress filtering — which is the half most estates skip and which neutralises command-and-control, exfiltration and SSRF in code you did not write. Then note that Kubernetes allows all traffic by default, so a cluster with no network policies is a flat network.

"API key, OAuth client credentials or mTLS between services?" mTLS when it is automated by a mesh or a workload-identity system, because the credential never moves and a stolen certificate is useless without the key. Client credentials when you already have an identity provider and want central revocation. API keys only for low-value integrations — shared, static, logged, and rotated by hand.

"What is the difference between a bearer credential and proof of possession?" A bearer token works for whoever holds it, so theft is enough. Proof of possession requires demonstrating control of a key that is never transmitted, so interception gains nothing. mTLS is the mature form; that framing is what the question is testing.

"What is dependency confusion?" A build that resolves against both a private and the public registry can be made to fetch an attacker's package by publishing a higher version of an internal name publicly. The fix is scoped names and registry configuration that never falls back to public for internal scopes — not a scanner.

"How would you secure a CI pipeline?" OIDC federation for cloud access so no long-lived key exists in CI, no secrets exposed to fork pull requests, separated build and deploy privileges, actions pinned by digest, and artefacts signed at build and verified at admission. CI is more valuable than any single application and is usually the least hardened system.

"Your scanner reports 300 vulnerabilities. What do you do?" Triage by reachability and exposure rather than by count and severity score. An unreachable critical in a package your code never calls is not urgent; a medium on the internet-facing path is. A programme that treats every finding as urgent trains the team to ignore the scanner, which is worse than not scanning.

One thing to volunteer: point out that an SBOM's real value is answering "are we affected" in minutes rather than days. During Log4Shell, the organisations that recovered fastest were the ones that could enumerate where the library existed — the delay was inventory, not patching.

Recall

• Firewalls are stateful, default deny, and should reference services rather than addresses. Egress filtering is the neglected half — it blocks command-and-control, exfiltration and SSRF in dependencies you did not write.
• Kubernetes allows all pod traffic by default — apply a default-deny NetworkPolicy first, then allow. Replace SSH bastions with session managers or identity-aware proxies that need no inbound port.
• Service identity: API keys are weakest (shared, static, logged, hand-rotated); client credentials are short-lived bearer tokens; mTLS is strongest because the key never moves — and is only sensible when automated, because an expired internal certificate is a total outage. SPIFFE and a service mesh are how it gets automated.
• Prefer proof of possession over bearer credentials: a bearer token works for whoever holds it, a key you never transmit does not.
• Supply chain attacks: typosquatting, dependency confusion (a higher public version of an internal name — fix with scoped names and no public fallback), maintainer compromise, and install scripts running before your tests.
• Defences: lockfiles with integrity hashes enforced by npm ci, exact pins for applications, CI actions pinned by digest not tag, automated updates (outdated dependencies remain the most exploited category), fewer dependencies and minimal base images, plus SBOM, signing (Sigstore) and SLSA provenance.
• CI holds credentials to everything. Use OIDC federation instead of stored cloud keys, never expose secrets to fork pull requests, separate build from deploy, and verify artefact signatures at admission.
• Runtime: non-root, read-only filesystem, dropped capabilities, seccomp, and enforcement at admission rather than a scan report. Triage by reachability, not count, or the team stops reading the scanner.

Self-test: What does egress filtering defend that no application change can? · What is the Kubernetes networking default, and why does it matter? · Why is a stolen mTLS certificate useless? · How does dependency confusion pick the attacker's package? · What replaces a long-lived cloud key in CI? · What made Log4Shell response slow for most organisations?

Next: 8.7 covers the obligations that arrive with the data itself — classification, consent, deletion rights, audit logs, and what an engineer actually has to build to satisfy them.