11.27 Audit Log & Compliance Platform

System behavior leaves a trail, but without cryptographic certainty, that trail is legally and technically worthless in a regulated environment. An Audit Log & Compliance Platform is not just a logging system; it is an append-only, tamper-evident ledger designed to prove who did what, when, and where, and to make tampering detectable even by someone with administrator access to the database. As this page shows, that last promise needs one ingredient people usually leave out.

The Problem with Traditional Logging

Standard logs (like stdout scraped into Elasticsearch) fail compliance audits for three reasons:

1. Mutability: Any admin with database access can UPDATE logs SET user = 'not_me'.
2. Schema Rot: Unstructured JSON means fields like user_id might be strings in one app and numbers in another, making cross-system reporting impossible.
3. No Chain of Custody: There is no mathematical proof that logs haven't been deleted or altered since they were written.

A true compliance platform solves this through cryptography, strict schemas, and isolated storage tiers.

Requirements

The platform here serves a regulated fintech with about 200 internal services. Every one of them must record who did what to which customer record.

Functional. Accept audit events from every service and reject malformed ones. Make tampering detectable, including by insiders. Retain events for 7 years so that they cannot be deleted early. Search by actor, by resource and by time. Export to auditors in a standard format. Honour a "right to be forgotten" request without breaking the record that an action happened. Audit every read of the audit log itself.

Non-functional, with numbers.

• No acknowledged event is ever lost. Once a service has been told "accepted", the event must reach long-term storage.
• Accepting an event adds under 50 ms at p99 to the service that sends it, so teams do not skip auditing to save latency.
• Queries over the last 90 days answer within 5 seconds; older queries may take minutes.
• Integrity is verified end to end every day, automatically.

Out of scope today: the auditors' reporting UI, and security-alerting rules built on top of the stream.

The clarifying questions, and what each answer changes

"Which regulation applies, and for how long?" It sets the retention period and whether storage must be truly write-once (some financial rules require it). Seven years of write-once storage is the design driver for everything below the ingest tier.

"Can a missing audit event block the business action?" If yes, the action must fail when the audit write fails, which puts the audit service on the critical path of every service. If no, events go through a local, durable buffer and may arrive late but never vanish. This answer changes the availability target of the whole platform.

"Does personal data appear in events?" Almost always. Then deletion requests and write-once storage are in direct conflict, and crypto-shredding (below) is required from day one, because it cannot be added to events already written.

"Who is allowed to read the audit log?" If the answer includes support staff, the query API needs its own fine-grained permissions, and every query becomes an audit event too.

Estimation

Event rate. About 20 million customer and staff actions a day, each producing about 5 audit events across the services it touches: 100 million events a day, about 1,000 a second on average and 5,000 a second at peak. What that forces: a partitioned log (Kafka) in front of everything, so a slow downstream never pushes latency back onto the 200 producing services.

Seven years of storage. At about 1 KB of JSON per event that is 100 GB a day. Stored as compressed Parquet, roughly 5 times smaller, it is about 20 GB a day, so 20 GB × 365 × 7 ≈ 51 TB over 7 years. What that forces: object storage with write-once locks is the only sensible home; 51 TB in a database for seven years would be expensive and pointless.

The searchable window. Keeping 90 days searchable means 100 GB × 90 = 9 TB of raw events, about 13.5 TB in a search index with typical overhead. What that forces: the index holds 90 days and is rebuilt from object storage if ever needed. Anything older is queried directly from the Parquet files with a SQL engine over object storage, which takes minutes, which auditors accept.

How many hash chains. In the code below, each append waits for the write to be acknowledged before the next event can be chained. At about 5 ms per acknowledged write, one chain manages about 200 events a second. The peak is 5,000. What that forces: at least 25 independent chains; with headroom, 64 chains, each handling about 80 events a second at peak. This is the "partition the hash chains" answer in the interview section, now with a number behind it.

Checkpoint signatures. One signed checkpoint per chain per minute is 64 × 1,440 = about 92,000 signatures a day, roughly one a second. What that forces: nothing at the key service; it handles this easily. The checkpoint interval is a security setting, not a cost one: it bounds how many events are protected only by the chain at any moment.

API

POST /v1/events                              (batch of up to 500)
[ { "eventId": "0192a8f4-7c1e-7b3a-9d2e-5f8c1a2b3c4d",
    "timestamp": "2026-10-18T09:41:07.312Z",
    "actor": { "userId": "u_77", "role": "support", "ipAddress": "10.4.2.19", "userAgent": "admin-console/4.2" },
    "action": "customer.address.update",
    "resource": { "type": "customer", "id": "c_9182" },
    "context": { "fieldsChanged": ["line1", "postcode"] } } ]

→ 202 Accepted  { "accepted": 499, "rejected": [ { "index": 37, "code": "MISSING_ACTOR_IP" } ] }

GET /v1/events?resource=customer:c_9182&from=2026-10-01&to=2026-10-18&limit=100&cursor=…
→ 200 OK  { "items": [ … ], "nextCursor": "…" }       (this query is itself audited)

POST /v1/exports   { "filter": { "actor": "u_77" }, "from": "2025-01-01", "format": "parquet" }
→ 202 Accepted     { "exportId": "exp_31" }

GET /v1/chains/svc-payments-03/checkpoints?from=2026-10-18T00:00:00Z
→ 200 OK  [ { "seq": 51840, "hash": "9c1e…", "signature": "MEUCIQ…", "keyId": "audit-2026-q4" }, … ]

POST /v1/subjects/c_9182/shred               (requires two approvers)
→ 202 Accepted

The producer's eventId makes retries safe. A service that times out and resends the batch sends the same UUIDv7 ids, and ingest drops the duplicates. That is what allows "no acknowledged event lost" without "some events counted twice".

Partial acceptance is explicit. One malformed event should not reject 499 good ones, and the producer needs to know exactly which one failed.

Checkpoints are published through the API. An external auditor can download them and verify the chain with the public key, without trusting this system at all. That is the point of signing them.

Data model

audit event (the AuditEvent type below, plus)
  chain_id       TEXT                     -- one of 64, chosen by hashing the producing service
  seq            BIGINT                   -- position within the chain
  prev_hash      CHAR(64)
  hash           CHAR(64)

checkpoints
  chain_id, seq  -- PRIMARY KEY (chain_id, seq)
  hash           CHAR(64)
  signature      BYTEA
  key_id         TEXT                     -- which KMS key signed it
  created_at     TIMESTAMPTZ

subject_keys                              -- for crypto-shredding
  subject_id     TEXT PRIMARY KEY
  kms_key_ref    TEXT NOT NULL
  shredded_at    TIMESTAMPTZ NULL

object storage layout (write-once)
  s3://audit/date=2026-10-18/chain=svc-payments-03/part-00041.parquet
  s3://audit-checkpoints/chain=svc-payments-03/2026-10-18T09-42.json

Access patterns:

Query	Frequency	Returns
Append to one chain	80/s per chain at peak	one event
Events for one resource, last 90 days	operational, frequent	tens to hundreds
Events by one actor, any period	investigations	possibly thousands
A chain's events in order, one day	daily verification	~1.5 million per chain
Checkpoints for a chain and day	daily, and auditors	1,440 rows

Chains are keyed by producing service, so every event from one service lands in one chain in order, and a verification job can check one chain without reading the others. Storage is partitioned by date first, because both retention and auditors' requests are expressed in dates.

subject_keys lives outside the write-once store, on purpose. It is the one thing that must be deletable: removing a subject's key makes their personal fields unreadable in every archived event, while the record that the actions happened remains intact.

Core Architecture

The system is divided into an ingest plane (high write throughput, strict validation) and a query plane (isolated access, long-term retention).

Audit Log and Compliance Platform Architecture

1. Append-Only Events and Strict Schemas

Every event must be strongly typed. If an event lacks critical context (like the IP address of the actor), it is rejected at the gate, not stored silently.

The Standardized Event Schema

// types/audit.ts
export enum EventSeverity {
  INFO = 'INFO',
  WARN = 'WARN',
  CRITICAL = 'CRITICAL'
}

export interface AuditEvent {
  eventId: string;          // UUIDv7 (time-sortable)
  timestamp: string;        // ISO-8601 UTC
  actor: {
    userId: string;
    role: string;
    ipAddress: string;
    userAgent: string;
  };
  action: string;           // e.g., 'user.permissions.update'
  resource: {
    type: string;           // e.g., 'user_account'
    id: string;
  };
  context: Record<string, any>; // Flexible but size-limited payload
  hash?: string;            // SHA-256 chain link (added by ingest); a hash, not a signature
}

2. Tamper Evidence: The Hash Chain

To prove that no events were modified or deleted, we use a Hash Chain. Every new log entry contains a cryptographic hash of the previous log entry's hash, plus its own data.

If an attacker deletes event #4, the hash for event #5 becomes invalid because it was computed using the original hash of #4.

The hole in a bare hash chain, and how signed checkpoints close it

That sentence is true and it is also the trap. An attacker who can delete event #4 can also recompute the hashes of #5, #6 and every later event, because SHA-256 needs no secret: anyone can calculate it. The rebuilt chain is internally perfect and the verifier says "intact". A hash chain only catches people who edit a record without rewriting everything after it.

The fix is to tie the chain to a secret the attacker does not have. Every few minutes the ingest service takes the latest entry's position and hash, signs them with a private key held in a KMS or hardware security module (the application can ask for signatures but can never read the key), and writes that signed checkpoint somewhere the attacker cannot change, such as the S3 Object Lock bucket below. To forge history now, the attacker must rewrite the chain and produce a new signature, which needs the key.

import { createHash, generateKeyPairSync, sign, verify, type KeyObject } from "node:crypto";

interface Entry { seq: number; body: string; prevHash: string; hash: string; }
interface Checkpoint { seq: number; hash: string; sig: Buffer; }

const GENESIS = "0".repeat(64);
const hashOf = (prevHash: string, body: string) =>
  createHash("sha256").update(prevHash + body).digest("hex");

function append(log: Entry[], body: string): Entry {
  const prevHash = log.length ? log[log.length - 1].hash : GENESIS;
  const entry = { seq: log.length, body, prevHash, hash: hashOf(prevHash, body) };   // (1)
  log.push(entry);
  return entry;
}

function chainIsIntact(log: Entry[]): boolean {                                     // (2)
  let prev = GENESIS;
  for (const e of log) {
    if (e.prevHash !== prev || e.hash !== hashOf(prev, e.body)) return false;
    prev = e.hash;
  }
  return true;
}

function checkpoint(log: Entry[], key: KeyObject): Checkpoint {                     // (3)
  const head = log[log.length - 1];
  return { seq: head.seq, hash: head.hash, sig: sign(null, Buffer.from(`${head.seq}:${head.hash}`), key) };
}

function matchesCheckpoint(log: Entry[], cp: Checkpoint, pub: KeyObject): boolean {  // (4)
  const signed = verify(null, Buffer.from(`${cp.seq}:${cp.hash}`), pub, cp.sig);
  return signed && log[cp.seq]?.hash === cp.hash;
}

1. Each entry stores the previous entry's hash and its own, so the log forms a chain.
2. Walking the chain recomputes every hash from the start. This catches an edit made in place.
3. A checkpoint signs "entry number seq has hash hash". In production the sign call is a KMS API call, and the private key never leaves the KMS.
4. Verification checks the signature with the public key, then checks that the log still has that exact hash at that position.

Run with five events (login, view salary, export payroll, delete report, logout) and one checkpoint after the fifth:

Log	Chain check	Checkpoint check
Honest	intact	matches
One event edited, nothing else changed	broken	not needed
"export payroll" deleted and every later hash rebuilt	intact	fails

The third row is the attack that matters, because it is what a competent insider does, and only the signed checkpoint catches it. Two practical notes. First, events written after the last checkpoint are protected only by the chain until the next checkpoint arrives. That is why checkpoints are frequent. Second, the exact bytes that get hashed must never change. Use one fixed JSON format, such as RFC 8785. If an event is written out again later, for example when it is converted to Parquet, its fields can come out in a different order, and every hash would break.

Implementation: The Ingest Pipeline

import * as crypto from 'crypto';

class AuditLedger {
  private previousHash: string;

  constructor(initialHash: string) {
    this.previousHash = initialHash;
  }

  /**
   * Processes a raw event, adds cryptographic proof, and commits it.
   */
  public async commitEvent(rawEvent: Omit<AuditEvent, 'hash' | 'eventId'>): Promise<AuditEvent> {
    // 1. Generate time-sortable ID
    const eventId = this.generateUUIDv7();

    // 2. Serialize event deterministically
    const payloadString = JSON.stringify({
      ...rawEvent,
      eventId
    });

    // 3. Compute hash including the previous hash (The Chain)
    const currentHash = crypto
      .createHash('sha256')
      .update(this.previousHash + payloadString)
      .digest('hex');

    // 4. Construct final event
    const finalEvent: AuditEvent = {
      ...rawEvent,
      eventId,
      hash: currentHash
    };

    // 5. Write to the append-only log FIRST, and only then advance the chain.
    //    If the write fails, previousHash is unchanged and the event can be retried.
    await this.writeToKafka(finalEvent);
    this.previousHash = currentHash;

    return finalEvent;
  }

  private writeToKafka(event: AuditEvent) {
    // Write logic here
    return Promise.resolve();
  }

  private generateUUIDv7(): string {
    // Implementation omitted for brevity
    return crypto.randomUUID();
  }
}

3. Retention: WORM Storage

Compliance frameworks (like SEC 17a-4, HIPAA, SOC2) require data to be retained for specific periods (e.g., 7 years) and mathematically prevent premature deletion.

We use WORM (Write Once, Read Many) storage, such as Amazon S3 with Object Lock enabled in Compliance Mode.

• Compliance Mode: Even the AWS Root User cannot delete or overwrite the object until the retention period expires.
• Events are batched from Kafka into Parquet files every 5 minutes and pushed to S3.

4. Search and Indexing

While S3 holds the immutable truth, searching S3 directly is slow. We dual-write to a search index (like Elasticsearch) for operational queries.

However, the index is ephemeral state. If the index is compromised, it can be entirely rebuilt from the S3 WORM storage.

5. Access Control

Audit logs contain highly sensitive data (e.g., "HR viewed employee X's salary").

• No raw DB access: No human gets direct access to the search index or S3.
• RBAC on Queries: The Query API enforces Row-Level Security. A support agent can query action: "login" but cannot query action: "salary.view".

Scale and failure

What breaks	Blast radius	How you find out	What keeps it running	Recovery
An event lost between log and storage	a gap in one chain	daily verifier: broken link or checkpoint mismatch	replicated log; chain makes the gap visible	replay from the log within its retention
Key service unavailable	checkpoints pause	checkpoint age alarm	events are still chained	sign the latest head when it returns
Search index compromised or corrupted	operational queries	verifier disagrees with index	object storage is the truth	rebuild the index from object storage
A service stops sending events	that service's actions unaudited	expected-volume alarm per service	local durable buffer in each service	backfill from the buffer
Insider deletes and rebuilds a chain	history rewritten	signed checkpoint mismatch	keys held in the KMS, out of reach	restore from write-once copy; investigate
Personal data deletion request	one person's fields	n/a	crypto-shredding	delete the subject key; events remain

Alarm on silence, not only on errors. The most damaging failure in an audit system is a service that quietly stops sending events, because nothing fails anywhere. A per-service expected-volume alarm ("payments normally sends 300 events a minute and has sent none for ten minutes") is what catches it.

What the interviewer will push on

1. "How do you handle high throughput without bottlenecking the hash chain?"Answer: A single global hash chain limits you to single-threaded throughput. You partition the hash chains. For example, create one hash chain per tenant or per service. This allows horizontal scaling of the ingest nodes while maintaining tamper evidence within each partition.

2. "What if PII gets into the audit log and we get a GDPR Right to Be Forgotten request? S3 is in WORM mode!"Answer: Crypto-shredding. Do not store raw PII in the log. Store an encrypted payload in the log, and store the decryption key in a separate, mutable KMS (Key Management Service). To "delete" the PII from the immutable log, you delete the decryption key. The log entry remains to prove the action happened, but the PII is permanently unreadable.

3. "How do you detect if the Kafka stream drops an event before it reaches S3?"Answer: The Continuous Auditor. A background job reads the S3 files, re-computes the hash chain block by block, and raises a P0 alert if hashing hash(n-1) with data(n) does not reproduce hash(n), or if the chain no longer matches the latest signed checkpoint. A missing event shows up as either a broken link or a checkpoint mismatch.

---

Recall

Quick Review

• Append-only ledger: Standard logs are mutable; audit logs require mathematical proof of immutability.
• Hash Chains: Linking hash(n) to hash(n-1) creates tamper evidence. Deleting/altering one row invalidates the rest of the chain.
• WORM Storage: Cloud storage locks (e.g., S3 Object Lock) enforce legal retention periods at the infrastructure level.
• Crypto-shredding: Solves the paradox of immutable logs vs. privacy laws (GDPR) by destroying decryption keys rather than the immutable data.

Quiz

1. Why is a single, global hash chain a poor design for a system processing 100,000 events per second?
2. How does crypto-shredding satisfy both SEC retention laws and GDPR deletion requirements simultaneously?
3. If a malicious admin gains full access to the Elasticsearch cluster holding the audit logs, how does the system recover its integrity?