10.25 — Security in Distributed Systems

Distributed systems multiply the attack surface. In a monolith, security is a hard outer shell; once inside, everything is trusted. In a distributed system, components communicate over untrusted networks, handle data for multiple tenants, and execute across hundreds of machines. We must adopt a "zero-trust" architecture: every request, internally or externally, must prove who it is and what it is allowed to do.

1. Encryption in Transit: TLS and mTLS

When data moves across a network, it is vulnerable to interception (eavesdropping) and modification (man-in-the-middle attacks). Transport Layer Security (TLS) prevents this.

How TLS Works

TLS combines asymmetric encryption (slow, highly secure key exchange) with symmetric encryption (fast, bulk data encryption).

TLS sequence diagram illustrating client-server handshake

Mutual TLS (mTLS)

Standard TLS only authenticates the server to the client. In a distributed system (like microservices), the server also needs to know exactly which client is calling it. mTLS requires the client to present a certificate as well.

This is the foundation of service meshes (like Istio or Linkerd). Every service gets a short-lived certificate, and all service-to-service communication happens over mTLS.

2. Encryption at Rest: Envelope Encryption

Storing plain text user data is negligent. However, encrypting a massive database with a single key means if that key is compromised, everything is compromised. If the key is lost, all data is gone.

Modern systems use Envelope Encryption, managed by a Key Management Service (KMS).

1. Master Key (KEK - Key Encryption Key): Stored in the KMS, never leaves the KMS.
2. Data Key (DEK - Data Encryption Key): Generated by the KMS to encrypt actual data.

Envelope Encryption workflow with Key Management Service

When reading data: The app fetches the Encrypted DEK from the database, sends it to the KMS, gets back the Plaintext DEK, decrypts the data, and discards the Plaintext DEK.

3. Secret Management

Hardcoding database passwords or API keys in code or configuration files leads to leaks. We use Secret Managers (like HashiCorp Vault or AWS Secrets Manager).

The gold standard is Dynamic Secrets. Instead of a static database password that lives for a year, the application requests access, and the secret manager creates a temporary database user with a 15-minute lifespan.

// Conceptual example of requesting a dynamic database credential
async function getDbCredentials(vaultClient: VaultClient, roleName: string): Promise<DbCreds> {
  // 1. App authenticates to Vault using its AWS IAM role or Kubernetes Service Account
  // 2. Vault generates a short-lived Postgres credential
  const response = await vaultClient.read(`database/creds/${roleName}`);
  
  return {
    username: response.data.username,
    password: response.data.password,
    leaseDurationSeconds: response.lease_duration // e.g., 900 seconds
  };
}

4. Identity: OAuth 2.0 and OIDC

OAuth 2.0 is an authorization framework (delegated access). OpenID Connect (OIDC) adds an identity layer on top of OAuth 2.0.

When a user logs in, the Auth Provider issues tokens:

1. Access Token: Proves the bearer has permission to access an API.
2. ID Token (OIDC): Contains identity information about the user (email, name) in JWT format.
3. Refresh Token: Used to get a new Access Token when the current one expires.

JSON Web Tokens (JWT) are widely used for stateless authentication.

import jwt from "jsonwebtoken";

interface AccessClaims { sub: string; roles: string[]; }

export function verifyAccessToken(authHeader: string | undefined, publicKey: string): AccessClaims {
  if (!authHeader?.startsWith("Bearer ")) throw new Error("401: missing bearer token");   // (1)
  const token = authHeader.slice("Bearer ".length);

  const claims = jwt.verify(token, publicKey, {
    algorithms: ["RS256"],                        // (2)
    issuer: "https://login.shop.example",         // (3)
    audience: "orders-api",                       // (4)
    clockTolerance: 30,                           // (5)
  });
  return claims as AccessClaims;
}

1. Reject a missing or malformed header before doing any work, with a clear 401.
2. Pin the algorithm. Without this list, an attacker can send a token signed with HS256, a shared-secret algorithm, using your public key as the secret, and an old library would accept it. Pinning RS256 closes that door.
3. Check who issued the token. A valid signature only proves that some holder of the private key signed it. If your identity provider serves several environments, a token from staging must not open production.
4. Check who the token is for. This is the line most often missing. Your login server also issues tokens for the marketing site and the mobile analytics service. Without an audience check, a token stolen from the least important of those apps is accepted by your orders API. The signature is valid; the token was simply never meant for you.
5. Allow 30 seconds of clock difference between servers when checking expiry, because clocks drift (10.3).

Tested with the real jsonwebtoken library: a token for orders-api passes, while a token issued for another audience, a token from the wrong issuer, an HS256 forgery and a request without a header are all rejected. Sessions and JWTs are covered in depth in 8.4.2.

5. Authorization: RBAC

Once we know who the user is (Authentication), we need to know what they can do (Authorization).

Role-Based Access Control (RBAC) groups permissions into roles, and assigns roles to users.

// RBAC implementation for a B2B SaaS platform
enum Permission {
  CREATE_ORDER = "create:order",
  READ_ORDER = "read:order",
  DELETE_ORDER = "delete:order"
}

const RolePermissions: Record<string, Permission[]> = {
  "admin": [Permission.CREATE_ORDER, Permission.READ_ORDER, Permission.DELETE_ORDER],
  "viewer": [Permission.READ_ORDER]
};

function hasPermission(userRole: string, requiredPermission: Permission): boolean {
  const permissions = RolePermissions[userRole] || [];
  return permissions.includes(requiredPermission);
}

// In an API handler:
if (!hasPermission(user.role, Permission.DELETE_ORDER)) {
  throw new Error("HTTP 403: Forbidden");
}

6. Tenant Isolation

In multi-tenant SaaS, you must guarantee that Tenant A cannot see Tenant B's data.

Logical Isolation (Pool Model): All tenants share the same database tables. Every table must have a tenant_id column.

• Pros: Cheap, easy to scale out.
• Cons: High risk of data bleed if a developer forgets a WHERE tenant_id = ? clause. "Noisy neighbor" problems.

Physical Isolation (Silo Model): Each tenant gets their own database instance or database schema.

• Pros: Zero risk of accidental cross-tenant data leaks. Easy to restore a single tenant's backup.
• Cons: Expensive. Hard to manage migrations across 1,000 separate databases.

Mitigating Logical Isolation Risks: Use Row-Level Security (RLS) in the database.

-- Postgres Row-Level Security (RLS) enforces isolation at the database engine level
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation_policy ON orders
    USING (tenant_id = current_setting('app.current_tenant')::uuid);

Two traps make RLS weaker than it looks. The table owner bypasses it. By default, PostgreSQL does not apply policies to the role that owns the table, and many applications connect as exactly that role. Add ALTER TABLE orders FORCE ROW LEVEL SECURITY; and have the application connect as a separate, non-owner role. Pooled connections remember settings. app.current_tenant is a setting on the database connection, and a connection pool hands the same connection to the next request. If the setting is made with a plain SET, a request for tenant B can run with tenant A's setting still in place. Set it with SET LOCAL (or set_config(..., true)) inside each transaction, so it disappears when the transaction ends.

7. Audit Logs

Audit logs answer: "Who did what, when, and from where?" They are critical for compliance (SOC2, HIPAA) and forensic analysis.

Audit logs must be:

1. Append-only: Never updated or deleted.
2. Comprehensive: Include actor, action, resource, timestamp, and IP.
3. Externalized: Shipped immediately to a separate, secure system (e.g., AWS S3 with Object Lock or a SIEM like Splunk) so an attacker who compromises the application server cannot erase their tracks.

What the interviewer will push on

1. "How do you revoke a JWT?"
   • Answer: JWTs are stateless, meaning you can't revoke them just by deleting a database row. You must either wait for them to expire (keep lifespans short, e.g., 5-15 minutes), or maintain a distributed blacklist (like Redis) of revoked tokens that the API gateway checks on every request, which defeats the stateless benefit.
2. "How do you handle a KMS outage if every DB read requires a decryption key?"
   • Answer: Data Key Caching. The application caches the plaintext DEK for a short period (e.g., 5 minutes) in memory. This reduces KMS API calls and provides resilience, though it slightly increases the risk if memory is dumped.
3. "A developer accidentally committed a static AWS access key to GitHub. What happens next?"
   • Answer: Automated bots scan GitHub for keys constantly. An attacker will spin up max-capacity crypto-mining EC2 instances in your account within seconds. To fix it: instantly rotate/delete the key in AWS, not just revert the commit. This is why we use IAM roles and dynamic secrets.

Recall

• mTLS: Both client and server authenticate each other using certificates. Critical for zero-trust microservices.
• Envelope Encryption: Data is encrypted by a Data Key (DEK). The DEK is encrypted by a Master Key (KEK) via a KMS.
• Dynamic Secrets: Short-lived, generated-on-demand credentials. Limits exposure windows.
• RBAC: Maps Users -> Roles -> Permissions.
• Tenant Isolation: Logical (shared DB + tenant_id + Row Level Security) vs Physical (separate DB per tenant).

Quiz

1. What is the primary advantage of Envelope Encryption over encrypting all data with a single master key? A) It uses asymmetric cryptography for the data, which is faster. B) You can rotate the master key without having to re-encrypt terabytes of data; you only re-encrypt the data keys. C) It prevents Man-in-the-Middle attacks. D) It allows the database to perform mathematical operations on encrypted data.

2. In an OAuth 2.0 / OIDC flow, what is the purpose of the Refresh Token? A) To authenticate the user directly to the database. B) To obtain a new Access Token when the current short-lived Access Token expires, without forcing the user to log in again. C) To encrypt the payload of the JWT. D) To sign out the user across all active sessions.

3. Why is Row-Level Security (RLS) recommended in a logically isolated multi-tenant database? A) It makes queries run faster by indexing rows per tenant. B) It enforces tenant isolation at the database engine level, preventing data leaks even if an application developer forgets to include a WHERE tenant_id = ? clause. C) It encrypts each row with a different tenant key. D) It converts logical isolation into physical isolation automatically.

Answers: 1: B, 2: B, 3: B

Next: 10.26 leaves the request path entirely: batch and stream processing for data too large for one machine.