8.2.2 — Hashing, MACs and Integrity

A webhook receiver checks the sender's signature like this:

if (req.headers['x-signature'] === expectedSignature) { … }   // two bugs

The first bug is that expectedSignature was computed as sha256(secret + body), which — for SHA-256 — an attacker can extend without knowing the secret. The second is that === compares byte by byte and returns as soon as two bytes differ, so the time it takes to reject a forgery tells the attacker how many leading bytes were correct, and a few thousand requests recover the whole signature.

Both bugs come from the same place: treating a hash function as if it were a general-purpose security tool. It is not. It is a specific tool with specific properties, and the gaps between what it does and what people assume are where these failures live.

1. What a hash function guarantees

A cryptographic hash function maps input of any length to a fixed-size output — SHA-256 gives 256 bits — with four properties:

Deterministic. The same input always gives the same output.

Avalanche effect. Change one bit of input and roughly half the output bits change, unpredictably. There is no partial similarity: two nearly identical inputs give completely unrelated hashes.

Preimage resistance. Given a hash, you cannot find an input that produces it. This is the "one-way" property.

Collision resistance. You cannot find any two different inputs with the same hash.

The last two are different, and the difference decides which attacks are practical. Preimage resistance protects a stored value: given the hash, find the original. Collision resistance protects an agreement: find two documents with the same hash, get one signed, and swap in the other.

The birthday bound is why collision resistance is weaker. Finding a collision among n-bit hashes takes roughly 2^{n/2} attempts, not 2^n — because you are looking for any pair, not a specific match, and the number of pairs grows quadratically. SHA-256 gives 256-bit preimage resistance and only 128-bit collision resistance. Both are far out of reach; the point is that the two numbers differ by half.

What a hash does not give you: it is not encryption (there is nothing to reverse), it is not authentication (anyone can compute it), and it is not slow, which is the property password storage needs and the reason Chapter 8.4.1 uses a different tool entirely.

2. Which hash functions to use, and which are dead

MD5 — broken. Collisions were demonstrated in 2004 and can now be produced in seconds. Chosen-prefix collisions followed in 2009, and were used in the Flame malware (2012) to forge a Microsoft code-signing certificate. Never use it for anything security-related.

SHA-1 — broken. The SHAttered attack (2017) produced two different PDFs with the same SHA-1. Chosen-prefix collisions arrived in 2020 for around $45,000 of compute, which is now much less. Browsers stopped trusting SHA-1 certificates in 2017.

SHA-256 / SHA-512 — the current default, and fine.

SHA-3 (Keccak) — a structurally different design, standardised as insurance in case a weakness is found in the SHA-2 family. Not faster; it is a hedge.

BLAKE3 — modern, very fast, parallelisable. A good choice where hashing throughput matters.

A nuance worth keeping: these are broken for collision resistance, not for preimages. Git still uses SHA-1 (with collision detection added, and a migration to SHA-256 in progress) and is not immediately unsafe. But "broken for collisions" means an attacker who controls both inputs can cheat — which is exactly the situation in code signing, certificates and document signatures. Do not reason about whether your specific use survives; move off it.

MD5 is still fine for non-security uses — a cache key, a shard selector, a checksum against accidental corruption. Being precise about that is more useful than a blanket ban, and using a non-cryptographic hash such as xxHash for those jobs is faster anyway.

3. Length extension, and why HMAC exists

Here is the first bug from the opening, properly.

SHA-256 and MD5 use the Merkle-Damgård construction: process the message in blocks, carrying an internal state forward, and output the final internal state as the hash.

That last part is the flaw. The hash is the internal state after the message, so an attacker who knows H(secret || message) and the length of the secret can resume from that state and compute H(secret || message || padding || extra) — a valid signature for a longer message — without ever knowing the secret.

Concretely: an API signs secret + "user=ana&role=user" and an attacker appends &role=admin with a valid signature. This has broken real systems, most famously Flickr's API in 2009.

HMAC is the fix, and it works by hashing twice with two derived keys:

\text{HMAC}(K, m) = H\big((K \oplus \text{opad}) \,\|\, H((K \oplus \text{ipad}) \,\|\, m)\big)

The outer hash means the output is not the internal state of a message-processing pass, so extension is impossible. You never need to know that formula, and you do need the rule: to authenticate with a hash, use HMAC — never hash(secret + message).

SHA-3 and BLAKE are not vulnerable to length extension by construction, so BLAKE3(key, message) in keyed mode is safe. HMAC-SHA-256 remains the interoperable default because it is what everyone else implements.

4. MACs: proving a message came from someone with the key

A MAC (message authentication code) proves two things at once: the message was not modified, and it came from someone holding the shared key.

The difference from a hash is the key. Anyone can compute SHA-256(message), so a bare hash proves nothing about origin. Only a key-holder can compute HMAC(key, message).

The difference from a signature (Chapter 8.2.3) is that a MAC uses one shared key, so both parties can produce it — which means a MAC cannot prove to a third party who sent the message. If you need that, you need a signature.

Where MACs actually appear in your work:

• Webhook signatures — Stripe, GitHub and others sign the raw body with a shared secret. Chapter 5.8 covers the receiver's checklist.
• Signed cookies and session tokens — the server signs the value so the client cannot alter it.
• JWTs with HS256 — the signature is an HMAC. Chapter 8.4.2 explains when that is the wrong choice.
• Signed URLs — a time-limited link to a private file, where the signature covers the path and the expiry.
• Blind indexes — storing HMAC(key, lower(email)) next to an encrypted email so exact-match lookup still works. Use HMAC and not a plain hash, or anyone with the database can test guesses offline.

Three rules for signing anything:

Sign the raw bytes, not the parsed object. Parsing then re-serialising can reorder keys or change whitespace, and the signature will not match — or worse, will match something different from what you validated. This is why webhook frameworks insist on the raw body.

Include everything that matters in what is signed. A signature over the body only, with the URL and timestamp outside it, lets an attacker replay the same body to a different endpoint.

Include a timestamp and reject old messages. Otherwise a captured valid request is valid forever.

5. Timing attacks and constant-time comparison

The second bug in the opening.

a === b            // returns at the first differing byte

A comparison that exits early takes measurably longer when more leading bytes match. Over a network the difference is nanoseconds buried in noise — and statistics defeats noise. Send enough requests, average the timings, and the correct first byte stands out. Then fix it and attack the second. A 32-byte signature falls in roughly 32 × 256 averaged attempts instead of 2^{256}.

The fix is a comparison that always examines every byte:

import { timingSafeEqual } from 'node:crypto';

function verify(received: string, expected: string): boolean {
  const a = Buffer.from(received, 'hex');
  const b = Buffer.from(expected, 'hex');
  if (a.length !== b.length) return false;      // (1)
  return timingSafeEqual(a, b);                 // (2)
}

(1) timingSafeEqual throws on unequal lengths, so check first. The length itself is not secret. (2) XORs every byte pair and accumulates, so the running time depends only on the length. Python's equivalent is hmac.compare_digest.

Use it for anything an attacker can submit repeatedly and vary: signatures, API keys, session tokens, password reset tokens, one-time codes. Not for passwords, because a password comparison is a hash comparison inside a slow KDF whose verify function already handles it.

A useful defensive habit: hash both sides first and compare the hashes. Comparing two 32-byte HMACs of the candidate and the truth leaks nothing about the underlying value even under a naive comparison, because the attacker cannot steer the hash.

6. Merkle trees: verifying a piece without the whole

Hashing a list of items gives one value that changes if anything changes — but to check one item you need every item. A Merkle tree fixes that: hash the items, hash each pair of hashes, repeat up to a single root hash.

The root commits to every leaf. Proving one leaf costs only the sibling hashes along its path, which is logarithmic in the number of leaves.

The property that makes it valuable: proving membership costs \log n hashes instead of n items. To prove block 2 is part of the set, you send block 2 plus its sibling A and its uncle CD; the verifier recomputes the root and compares.

Where you already meet it:

• Git — every commit hashes its tree, which hashes its subtrees and files. That is why a commit id fixes the entire history: changing anything anywhere changes every hash above it. Chapter 14.1 covers the object model.
• Certificate transparency — every issued certificate is added to an append-only log whose Merkle root is published, so a mis-issued certificate for your domain is detectable. This is the log Chapter 8.1 mentioned as a reconnaissance source.
• Blockchains — the block header commits to all transactions via a Merkle root, so a light client can verify one transaction without the whole block (Chapter 10.13).
• Backup and sync — comparing two directory trees by hash finds what differs without reading everything.

Content addressing is the related idea: name a piece of data by its own hash. The name then proves the content — you cannot serve something else under it — which is why container image digests, package lock file integrity fields, and Subresource Integrity all work this way. Chapter 6.10 covers SRI from the browser side.

7. Practical integrity in your work

Verify downloads. A published SHA-256 checksum protects against corruption and against a compromised mirror — but only if the checksum comes over a channel the attacker does not also control. A checksum on the same page as the download stops corruption, not a compromised server. A signature (Chapter 8.2.3) does.

Pin dependencies by hash. Lockfiles record an integrity hash per package, so a republished version with different content fails to install. Chapter 3.10 covers the supply-chain reasoning.

Do not use a hash as an identifier for something with few possible values. Hashing an email address does not anonymise it: an attacker hashes every address in a leaked list and matches. The same applies to phone numbers, national identity numbers and postcodes. A hash anonymises nothing whose input space can be enumerated — that needs a keyed HMAC where the key is secret, or a random identifier.

Do not deduplicate encrypted user data by hash of the plaintext unless you have thought it through: it tells you when two users hold the same file, which is information.

What the interviewer will push on

"What is the difference between hashing and encryption?" Encryption is reversible with a key; hashing is one-way with no key and a fixed-size output. Then push further to the useful distinction — a hash proves nothing about origin, because anyone can compute it, which is why you need a MAC.

"Why can't you sign a message with sha256(secret + message)?" Length extension: SHA-256 outputs its internal state, so an attacker who has the hash and knows the secret's length can append data and produce a valid hash without the secret. Use HMAC, whose outer hash removes the property. Naming Flickr's 2009 API break makes it concrete.

"What is a timing attack and where does it apply?" An early-exit comparison leaks how many leading bytes matched, so averaging over many requests recovers a secret byte by byte — 32 × 256 attempts instead of 2^{256}. Use timingSafeEqual or compare_digest for signatures, API keys and tokens. Volunteering that hashing both sides first also defeats it shows real understanding.

"MD5 and SHA-1 are broken — broken how?" For collision resistance, not preimages. That distinction matters because collisions only help an attacker who controls both inputs — code signing, certificates, documents. Then say the practical thing: MD5 is still fine for a cache key, and you should still move off both for anything security-related rather than reasoning about your specific case.

"What is a Merkle tree for?" Proving one item belongs to a set with \log n hashes instead of the whole set. Give real instances — Git commit ids, certificate transparency, blockchain light clients — because that is what shows you have met it rather than read about it.

"Is hashing an email address enough to anonymise it?" No. The input space is enumerable, so an attacker hashes a leaked address list and matches every one. You need a keyed HMAC with a secret key, or a random identifier with a lookup table you control. This is one of the most common privacy mistakes in analytics pipelines.

One thing to volunteer: point out that a signature must cover the raw bytes and include the URL and a timestamp, not just the parsed body — otherwise a valid request can be replayed against a different endpoint forever. It is the failure mode that survives even after someone has correctly implemented HMAC.

Recall

• A cryptographic hash is deterministic, has avalanche, and gives preimage and collision resistance. The birthday bound halves collision strength: SHA-256 is 128-bit collision resistant.
• MD5 and SHA-1 are broken for collisions, not preimages — which matters when the attacker controls both inputs (certificates, code signing). Use SHA-256, SHA-3 or BLAKE3; MD5 is still fine as a cache key.
• Never hash(secret + message). Merkle-Damgård hashes output their internal state, so length extension lets an attacker append data and produce a valid hash without the key. Use HMAC.
• A MAC proves integrity and origin using a shared key — so both parties can produce it, and it cannot prove to a third party who sent it. That needs a signature.
• Signing rules: sign the raw bytes, include the URL and a timestamp so replays expire, and reject old messages.
• === on a secret leaks how many leading bytes matched. Use timingSafeEqual / compare_digest for signatures, keys and tokens — or hash both sides and compare the hashes.
• A Merkle tree proves membership in \log n hashes. It is why a Git commit id fixes all history, how certificate transparency detects mis-issuance, and how a blockchain light client verifies one transaction. Content addressing names data by its own hash, so the name proves the content.
• A hash anonymises nothing enumerable. Hashed emails, phone numbers and postcodes are recoverable by brute force; you need a keyed HMAC with a secret key or a random identifier.

Self-test: Why is collision resistance half the bit strength of preimage resistance? · What exactly does length extension let an attacker do? · When does a MAC fail to prove who sent something? · Why is === on a signature a vulnerability, and what is the arithmetic? · What does a Merkle proof cost for one item out of a million? · Why is a hashed email still personal data?

Next: 8.2.3 covers the half that solves key distribution — public keys, RSA and elliptic curves, digital signatures, and what "proves who sent it" actually requires.