11.16 — Flash Sale & Inventory

Ten thousand units go on sale at 10:00:00. Two million people are waiting with their finger on the button. By 10:00:10, all ten thousand are gone and 1,990,000 people have been told no.

Write down the arithmetic before designing anything, because it changes what the system is for: 99.5% of the requests must be rejected. The job is not "process 160,000 purchases a second". It is "reject 1.99 million requests as cheaply and as early as possible, and process ten thousand of them perfectly".

Every design that fails at this does so by getting that sentence backwards — building a purchase path that is correct and then discovering it cannot survive being told no two million times.

This is also the extreme-contention study: one row that every request in the system wants to decrement, where the wrong design either oversells — a business and legal problem — or collapses, taking the rest of the store down with it.

1. Requirements

Functional. Sell N units at a scheduled moment. A purchase reserves stock; payment confirms it. Unpaid reservations expire and return to stock. Per-customer purchase limits. A queue people can wait in.

Non-functional, with numbers.

• Never oversell. Selling 10,001 units is materially worse than selling 9,999, because the first is a promise you cannot keep and the second is a rounding error.
• The rest of the site must stay up, including for customers who have no interest in the sale.
• "Sold out" communicated within 1 second.
• Bot resistance is a first-class requirement, not something to add later. Without it, most of a high-demand sale goes to automation and the sale fails at its actual purpose.

Out of scope today: payment internals (11.14), catalogue and search, shipping.

The clarifying questions, and what each answer changes

"How many units, and how many people?" The ratio is the design. Ten thousand units for twelve thousand people is a queue. Ten thousand for two million is a load-shedding problem wearing a shop's clothing.

"Is overselling recoverable?" Sometimes it is — a digital good, or a product with more stock arriving. Usually it is not, and it becomes a refund, an apology and occasionally a regulator. Establishing this early is what justifies failing closed in section 8.

"Does a click buy the item, or reserve it?" Reserve, almost always, because payment can fail. That single answer introduces expiry, a sweeper, and the idempotency requirement that turns out to be the most common cause of real oversells.

"What does fair mean here?" First come first served, random among those present, or a lottery with advance registration. This is a product and reputation decision, not an engineering one, and it must be published — section 6.3 explains why.

"Is the sale on the same infrastructure as the rest of the store?" If yes, the most likely incident is not the sale failing but the whole store failing with it.

2. Estimation

Arrival rate. 2 million users, about 80% arriving within a 10-second window = ~160,000 requests a second, all targeting one item. What that forces: everything below. This is not a distributed load, it is a single point of contention receiving the whole internet at once.

What one database row can take. A row lock held for the duration of a network round trip plus a transaction commit is roughly 1–5 milliseconds, so a single row supports a few hundred to a couple of thousand serialised updates a second. What that forces: 160,000 against 2,000 is two orders of magnitude over. No amount of database tuning closes that gap, because the limit is serialisation rather than throughput — and serialisation is by definition the opposite of throughput.

The rejection ratio, which reframes the problem. 10,000 units among 2 million people means 99.5% of requests are rejected. What that forces: the design is a funnel. Each layer exists to reject as many requests as it can at the lowest possible cost, so that the expensive, strictly correct layer at the bottom has almost nothing to do. A layer that rejects a million requests for a tenth of a penny each is worth more than any optimisation of the purchase path.

What an in-memory counter can take. A single atomic operation in a memory store handles 100,000 or more a second. What that forces: the contended decision moves out of the database and into a system built for exactly one thing. The database then sees only the ten thousand winners, which is an ordinary insert rate with no contention at all.

Retry amplification, the number people forget. If rejected users retry every two seconds, the effective arrival rate doubles or triples within moments. What that forces: the queue in section 5. An honest "you are 340,000th, about twelve minutes" produces far less traffic than an error page, because an error page is an instruction to press the button again.

3. API

GET /sale/sku_991/ticket                 # served at the edge; no origin involvement
→ 200 { "token": "eyJ…", "position": 341902, "estimatedWaitSeconds": 730 }

GET /sale/sku_991/status?token=eyJ…      # polled by the waiting page, with jitter
→ 200 { "state": "waiting", "position": 118004, "estimatedWaitSeconds": 240 }
→ 200 { "state": "admitted", "admissionToken": "adm_…", "expiresIn": 120 }
→ 200 { "state": "sold_out" }

POST /sale/sku_991/reserve
Authorization: Bearer …
X-Admission-Token: adm_…
Idempotency-Key: 9f2c…
→ 201 { "reservationId": "res_01J…", "expiresAt": "2026-08-01T10:12:00Z" }
→ 409 { "error": { "code": "sold_out" } }
→ 409 { "error": { "code": "limit_reached" } }

POST /reservations/res_01J…/confirm      # after payment succeeds
→ 200 { "orderId": "ord_…" }

The waiting-room token is issued and verified at the edge, with no origin involvement at all. Anything else defeats the purpose: a waiting room served by the infrastructure the sale runs on is not a waiting room, it is the queue for the thing that is already overloaded.

Position lives in a signed token held by the client, not in server-side state. Two million server-side sessions is its own scaling problem, and a signed token carrying {userId, arrivalTime, position} is verifiable at the edge with no lookup at all.

The admission token is single-use and short-lived. Without both properties it will be shared and resold — which genuinely happens for high-demand sales — and the funnel becomes decoration.

POST /reserve rejects any request without a valid admission token. If the purchase endpoint can be called directly, everything above it is theatre.

Reserving returns an expiry. The user has a visible countdown, the reservation is a hold rather than a sale, and section 6.2 covers what happens when it lapses.

4. Data model

sale_events
  sale_id      UUID PRIMARY KEY
  sku          TEXT NOT NULL
  initial_stock INT NOT NULL
  opens_at     TIMESTAMPTZ
  state        SMALLINT            -- scheduled | open | sold_out | paused

reservations
  reservation_id UUID PRIMARY KEY
  sale_id      UUID, user_id UUID
  state        SMALLINT NOT NULL   -- pending | confirmed | expired | cancelled
  created_at   TIMESTAMPTZ NOT NULL
  expires_at   TIMESTAMPTZ NOT NULL
  restocked_at TIMESTAMPTZ NULL    -- the field that prevents a double restock
  UNIQUE (sale_id, user_id)        -- the per-user limit, at the data level

-- in the memory store, during the sale:
stock:{sale_id}        → integer, the authoritative remaining count
buyers:{sale_id}       → set of user ids that have already won

Access patterns:

Query	Frequency	Returns
Decrement stock atomically	up to 15,000/s	won, sold out, or already bought
Insert a reservation	~1,000/s briefly	one row
Confirm a reservation	~1,000/s over minutes	one row
Find expired pending reservations	every few seconds	tens to hundreds
Sum reservations by state	continuously	three numbers

The counter in memory is the gate; the database is the record. That split is the whole architecture, and it works because the two have completely different jobs: the counter must be fast and atomic, and the database must be durable and queryable. Neither can do the other's job at this scale.

UNIQUE (sale_id, user_id) is the per-user limit expressed where it cannot be bypassed. The in-memory buyer set enforces it fast; the constraint enforces it truthfully, and the two together mean a user firing fifty parallel requests wins at most once even if something upstream goes wrong.

restocked_at is a nullable timestamp doing more work than any other field on this page. Section 6.2 explains why, and the staff question at the end is entirely about it.

5. Architecture: the funnel

Figure 1 — The funnel. The design principle in one picture. Since 99.5% of requests must be rejected, every layer exists to reject as many as it can at the cheapest available cost, leaving the strictly-correct layer at the bottom with almost nothing to do.

Layer one, the edge. A fully static sale page and a waiting-room token, served from the content delivery network with no origin involvement. The cheapest possible rejection is one your servers never see.

Layer two, the queue. Admits users in a controlled trickle — say five thousand a second — with an honest position and estimated wait. Everyone else holds a place rather than hammering the retry button.

Layer three, eligibility. Signed in, has not already bought, does not look like automation. All of these are cheap checks against per-user state, which is spread across millions of keys, rather than against the one contended item.

Layer four, the counter. An atomic decrement that returns the remaining count and rejects instantly at zero. This is where overselling is prevented.

Layer five, the database. Records the reservation for the few thousand survivors — an ordinary insert with no contention.

6. Deep dives

6.1 The counter, where overselling is prevented

-- KEYS[1] = stock:{sale}   KEYS[2] = buyers:{sale}   ARGV[1] = userId
local left = tonumber(redis.call('GET', KEYS[1]))                     -- (1)
if not left or left <= 0 then return -1 end                           -- (2)
if redis.call('SISMEMBER', KEYS[2], ARGV[1]) == 1 then return -2 end  -- (3)
redis.call('DECR', KEYS[1])                                           -- (4)
redis.call('SADD', KEYS[2], ARGV[1])                                  -- (5)
return 1

(1) read the remaining stock. (2) sold out means reject immediately, with no side effect at all — which matters, because this branch runs far more often than the winning one and must be as close to free as possible. (3) the per-user check happens inside the same script, which is why a user firing fifty simultaneous requests wins at most once. (4) the only place in the entire system where stock decreases. (5) recording the winner in the same atomic step as the decrement, so the two can never disagree.

Why this cannot oversell. The store executes the whole script without interleaving, so the read, the check and the decrement are one indivisible operation. The check-then-act race that would otherwise let two requests both see "1 remaining" and both decrement is structurally impossible (9.5.1).

What it costs, stated plainly: the memory store becomes a component whose failure stops the sale, which is why section 8 fails closed rather than open. That is an unusual choice in this book, and it is correct here for one reason — the harm from admitting uncounted sales exceeds the harm from pausing.

6.2 Reservations expire, and the restock is where oversells are actually born

A win is a hold, not a sale. It is written to the database as pending with an expiry of ten minutes, and the user has that long to pay. If they do not, a sweeper returns the unit to stock by incrementing the counter back.

That increment is the single most dangerous operation in the entire system. Every real oversell of the kind that appears late in a sale traces back to it running more than once for the same reservation — a retried job, a duplicated schedule, two sweeper instances without coordination. Each spurious increment silently inflates available stock, and the extra units then sell perfectly normally through a perfectly correct purchase path.

The fix is that the restock must be idempotent at the data level, not in the job's logic:

UPDATE reservations
   SET restocked_at = now(), state = 'expired'
 WHERE reservation_id = :id
   AND restocked_at IS NULL;

Increment the counter only if that statement affected a row (10.4). A second sweeper, or a retry of the first, affects zero rows and therefore increments nothing. The idempotency lives in the database rather than in a variable, which is what makes it survive a process dying halfway.

Figure 2 — The reservation lifecycle. The purchase path on the top row is the one everyone designs carefully. The return path on the bottom row is where real oversells come from, because a duplicated increment produces no error and the extra units are then sold entirely correctly.

6.3 Fairness is a published policy, not an emergent property

Pure first-come-first-served at 160,000 requests a second is not achievable, and — more interestingly — it is not obviously desirable. It rewards whoever has the fastest network path, which in practice means automation and expensive connections rather than enthusiasm.

Three real options.

Randomised admission among everyone present at the open. Defensible, and resistant to shaving milliseconds off a network path.

Order of arrival at the edge. Feels fair to users, and requires edge timestamps you trust.

A lottery with advance registration. Register during a window, winners drawn, and this is the fairest option and increasingly the standard for genuinely scarce goods — largely because it removes the value of being fast, which removes most of the value of automation.

Whichever you choose, publish it before the sale. The reputational damage from flash sales comes overwhelmingly from perceived unfairness rather than from people losing. Someone who was told the rules and lost is disappointed; someone who suspects the rules were different for other people is angry in public.

6.4 Bots are the operational reality, not an edge case

Without defences, the majority of a high-demand sale goes to automation. The response is layered, and none of the layers work alone.

Account age and history requirements, so an account created that morning cannot participate.

Rate limits keyed on account and payment instrument, not on network address. Addresses are cheap and disposable; payment instruments are not, and keying on them is one of the few limits that actually costs an attacker money.

A challenge on the admission step, not on every request — one challenge in the funnel is worth more than a hundred challenges scattered through it, and it is far less annoying to legitimate users.

Cancellation after the fact for orders matching automation patterns, with a clear published policy so it does not look arbitrary.

And the honest limit: this is an arms race that is managed rather than won. The strongest single lever is not detection, it is making automation economically unattractive — registration windows, lotteries, per-account history requirements — because those change the economics rather than the difficulty.

6.5 Isolate the sale from everything else

The most common flash-sale incident is not the sale failing. It is the entire store going down with it, converting a disappointing product launch into a revenue-losing outage that affects customers who did not even know there was a sale.

The defence is physical separation, exactly as in 11.6: separate service instances, separate database connection pools, separate cache namespaces, separate rate-limit budgets (10.9). The sale traffic must be structurally incapable of consuming the resources the rest of the site needs.

6.6 Warm everything before the gate opens

A flash sale is the rare event whose peak is known in advance, which makes an unprepared one an unforced error.

Pre-scale, do not autoscale. Autoscaling reacts over minutes; a flash sale is decided in seconds. Waiting for autoscaling is planning to fail.

Pre-warm the caches, so the first thousand requests do not all miss.

Pre-establish connections, so nobody pays for a connection handshake at the worst possible moment.

Load the counter and push the static page to the edges before the announced time.

And rehearse at the real number. Run the funnel against a synthetic two million users before the day, because the layer that fails is never the one you expected — it is usually the queue's own token verification or the identity service behind the sign-in check.

7. Decision Ledger

Decision	Alternatives	Why this	What it costs
A multi-layer funnel	let requests reach the database	rejects 99.5% cheaply, so the database only sees winners	several layers to build, operate and rehearse
An atomic in-memory counter as the gate	a row lock; optimistic retry	100,000+ a second against a few thousand, and overselling becomes structurally impossible	the memory store becomes a component whose failure stops the sale
A queue with an honest position	fail fast with an error	honest waiting produces far less traffic than retry storms	users wait; the queue must itself be edge-served to survive
Reservations that expire	sell instantly on click	payment can fail, so stock must be able to come back	a sweeper, an idempotent restock, and a visible countdown
Restock idempotent at the data level	idempotent in the job's logic	survives a process dying between the two steps	one nullable column and a conditional update
Randomised or lottery admission	strict first come first served	resists latency arbitrage and much automation	some users feel it is less earned, so it must be published
Full isolation from the main site	shared infrastructure	a sale failure can never become a store outage	duplicate capacity for a short-lived event
Fail closed when the counter is unavailable	fail open and reconcile later	admitting uncounted sales is worse than pausing	the sale stops, visibly, and someone must resume it

8. Scale and failure

At 10× — 100,000 units and 20 million users — two moves. Shard the counter: ten shards of 10,000 units each, with users hashed to one. Throughput multiplies, at the cost of one shard selling out while another still has stock, which is fixed by rebalancing or by a final consolidation pass. And push more rejection to the edge, which is always the cheapest layer and the one with the most headroom.

What breaks	Blast radius	How you find out	What keeps it running	Recovery
Counter store unavailable	the sale stops	error rate on reserve	fail closed — show "sale paused"	rebuild as initial − confirmed − pending with the sale paused, then resume deliberately
Restock sweeper runs twice	silent oversell	continuous invariant check	restocked_at conditional update	correct the counter; identify affected orders
Queue itself falls over	everything floods the origin at once	queue error rate	it must be the most robust component: stateless, edge-served	degrade to a static "come back later", never to a pass-through
Payment provider slow	reservations near expiry	payment latency	extend the reservation window rather than mass-cancelling	communicate; do not silently drop holds
Everyone polls at the same instant	a self-inflicted spike every N seconds	request pattern with sharp peaks	jitter the client polling interval	add jitter; the pattern flattens
A second write path to the counter	oversell, from nowhere obvious	an audit log of every counter mutation	the rule that the script is the only writer	remove the path; verify with the invariant
Sale traffic exhausts shared pools	the whole store	connection pool saturation	separate instances, pools and budgets	isolate before the next sale

Fail closed here, and this is the one place in this Part where that is the right answer. Everywhere else — the rate limiter, the cache — availability wins because the harm from being unavailable exceeds the harm from being approximate. Here it inverts: admitting uncounted sales creates promises you cannot keep, and a paused sale is recoverable while an oversold one is not.

And the invariant that must run continuously, not at the end:

counter + confirmed + pending == initial_stock

Check it every few seconds during the sale, with the authority to pause the sale automatically on a mismatch. At 43 units over, the loss is embarrassing. The same defect on a larger sale is a serious financial and legal exposure, and the entire difference between those two outcomes is whether the invariant was checked continuously or once in a report the following morning (10.10).

What the interviewer will push on

"Why not SELECT ... FOR UPDATE and a decrement? It is obviously correct." It is obviously correct and it is fatal. The row lock serialises every buyer, and each holds it for a network round trip plus a commit — a few milliseconds — so the ceiling is a couple of thousand purchases a second against 160,000 arriving. Then walk the cascade: connections pile up, the pool exhausts, requests for other parts of the site cannot get a connection, timeouts start, clients retry and amplify, and the store goes down. The tell is naming that a design correct in isolation became catastrophic under contention.

"State the problem in one sentence." "Reject 1.99 million requests as cheaply and as early as possible, and process ten thousand perfectly." A candidate who says "handle 160,000 purchases a second" has not read the ratio, and everything they design afterwards will be aimed at the wrong target.

"Where exactly is overselling prevented?" In one atomic script, where the read, the sold-out check, the per-user check and the decrement happen without interleaving. Then the follow-up that separates experience from theory: the far more common cause of a real oversell is the restock, when a sweeper returns the same expired reservation to stock twice. Idempotency has to be at the data level — a conditional update on a restocked_at column, incrementing only if a row was affected.

"The counter store dies mid-sale. Fail open or closed?" Closed, and the reasoning must be explicit rather than instinctive: everywhere else in this Part availability beats precision, and here it inverts because uncounted sales create promises you cannot keep while a paused sale is fully recoverable. Then describe the rebuild — initial − confirmed − pending from the database, computed with the sale paused, verified before resuming, and never as an automatic recovery.

"How do you make the sale fair?" The trap is to answer with an algorithm. Fairness here is a published policy: strict arrival order rewards fast networks and automation, so randomised admission or an advance-registration lottery is usually better — and whichever you pick must be announced beforehand, because the reputational damage comes from suspected unfairness rather than from losing.

"Your sale went fine but the checkout for normal customers timed out for twenty minutes. What happened?" Shared infrastructure. Sale traffic consumed the connection pools, the caches or the rate-limit budget that the rest of the store depends on. The fix is physical isolation, and the framing worth offering is that the most common flash-sale incident is not the sale failing but the store failing alongside it.

Volunteer this, because nobody asks: the invariant counter + confirmed + pending == initial must be checked during the sale, every few seconds, and it must have the authority to pause the sale by itself. A correctness property verified only in a post-mortem is a property you do not actually have — you have a report about the property, delivered after the money moved. Giving an automated check the power to stop the system is unusual, and in a system with exactly one hard invariant it is the right call.

Next: 11.17 — from one contended row back out to a spatial question, but a static one this time: not "which driver is near me right now" but "which of a hundred million fixed places are near this point", where the index can be built in advance and the difficulty moves into how you rank what you find.

Recall

• Reframe the problem first: 10,000 units among 2 million people means 99.5% of requests must be rejected. The job is rejecting 1.99 million cheaply and early, not processing 160,000 purchases.
• The arithmetic that kills the obvious design: a row lock held for 1–5 ms caps a single row at a couple of thousand updates a second, against 160,000 arriving — two orders of magnitude, and the failure cascades into a full store outage through pool exhaustion.
• The funnel: edge static page and token → queue with an honest position → eligibility and bot checks → atomic in-memory counter → database reservation. Each layer rejects roughly ten times more cheaply than the next.
• Overselling is prevented by one atomic script doing read, sold-out check, per-user check and decrement without interleaving. But the restock is where real oversells are born — the sweeper must be idempotent at the data level (restocked_at set by a conditional update, incrementing only if a row was affected).
• The queue must be edge-served and stateless: position lives in a signed token held by the client, admission tokens are single-use and short-lived, and the purchase endpoint rejects anything without one.
• Fairness is a published policy. Strict arrival order rewards fast networks and automation; randomised admission or an advance lottery is usually better, and the damage comes from perceived unfairness.
• Isolate the sale physically — separate instances, pools, caches, budgets — because the most common incident is the whole store going down with it. Pre-scale rather than autoscale; autoscaling reacts in minutes and the sale is decided in seconds.
• Fail closed when the counter is unavailable, uniquely in this Part, because uncounted sales are unrecoverable and a paused sale is not.
• Check counter + confirmed + pending == initial continuously during the sale, with the authority to pause automatically.

Self-test: State the problem in one sentence. Why does a row lock fail, and what does the failure cascade into? Where is overselling actually prevented, and where is it actually caused? Why fail closed here when everywhere else fails open? What must the invariant be allowed to do?

Quiz Bank

FoundationalWhy can't you just use a database transaction with SELECT ... FOR UPDATE?

Because it is perfectly correct and fatally slow, and the failure is not local.

SELECT stock FROM inventory WHERE sku = ? FOR UPDATE followed by a decrement genuinely prevents overselling. The row lock serialises every buyer, and no interleaving is possible. That is not the problem.

The problem is that serialising is the opposite of throughput. Each buyer holds the lock for the duration of a network round trip plus a transaction commit — call it 1 to 5 milliseconds — so the maximum is a few hundred to a couple of thousand purchases a second. Against an arrival rate of 160,000 a second, that is two orders of magnitude short, and no amount of tuning closes a gap of that size because the limit is structural rather than a matter of configuration.

What happens next is the part that turns a slow sale into an outage. Connections pile up waiting for the lock. The connection pool exhausts. Requests for other parts of the store — checkout, browsing, account pages — cannot get a connection, because they share the pool. Database processor time rises from managing lock contention rather than doing useful work. Timeouts begin, clients retry, and the retries amplify the load (10.9). A design that was correct in isolation has taken down a business that had nothing to do with the sale.

What to do instead is move the contended decision into a system built for exactly one thing: an atomic in-memory counter doing 100,000 or more operations a second, with the entire check-and-decrement inside a single uninterruptible script. The database then records only the small number of winners, which is an ordinary insert with no contention at all.

The general lesson worth extracting: a database's transactional guarantees are the right tool for correctness under moderate contention and the wrong tool under extreme contention. Not because they are wrong, but because the fix is to shrink the serialised section to microseconds in a system with no disk, no multi-version bookkeeping and no connection pool — and a database cannot become that system by being tuned.

InterviewDesign the waiting room. What must it guarantee, and where does it run?

It runs at the edge, and that is not an optimisation. Its entire purpose is absorbing traffic the origin cannot handle, so a waiting room served by the same infrastructure the sale runs on is not a waiting room — it is the queue for the thing that is already broken. It belongs in the content delivery network or a dedicated edge service.

Five guarantees.

Every arrival gets an immediate, honest answer. A position and an estimated wait. The alternative — an error, or a spinner — is an instruction to press the button again, which multiplies load exactly when it is highest. "You are 341,902nd, about twelve minutes" produces dramatically less traffic than a failure page, and the reason is behavioural rather than technical.

Position is stable. Refreshing must not lose or improve your place. That means position lives in a signed token held by the client, not in server-side state for two million people. The token carries the user, the arrival time, the position and a signature, and the edge can verify it with no lookup at all.

Admission is paced by what the purchase path can absorb, not by a fixed rate chosen in advance. Ideally it is driven by a feedback signal — current latency, current error rate — so it regulates itself as conditions change (10.9).

Admission tokens are single-use and short-lived. Without both, they get shared and resold, which genuinely happens for high-demand items.

It cannot be bypassed. The purchase endpoint rejects any request without a valid admission token, or every layer above it is decoration.

Three details that decide whether it works. The waiting page is fully static, with client-side polling at a jittered interval — synchronised polling creates a thundering herd every N seconds, which is a spike you inflicted on yourself. Wait estimates should be conservative, because under-promising keeps people patient while over-promising makes them refresh. And the queue must handle the pre-open population, which is often larger than the post-open arrival: people queue for an hour beforehand, so what happens at the moment of opening — strict arrival order, or randomised among everyone present — is a fairness decision to make and announce deliberately.

The failure mode to design against: if the queue itself falls over, everyone floods the origin simultaneously and the sale is over before it started. So it must be the most robust component in the system — stateless, edge-distributed, and degrading to a plain "come back shortly" page rather than to a pass-through that lets everyone through at once.

StaffAfter the sale, the numbers do not balance: 10,043 units sold against 10,000 in stock. Find the cause and prevent recurrence.

First, contain. Establish the exact overage and identify the 43 affected orders by reservation time and path. Decide the business response now — honour them and absorb the cost, or cancel with compensation. Honouring is usually right for reputation, but it is a business decision that needs a decision-maker rather than an engineer's default. Freeze further sales of that item.

Then find the cause, and the good news is that a well-built version of this design has only four places an oversell can enter.

One: the restock double-counted. By a wide margin the most likely, and its signature is exactly this — a modest overage, appearing late in the sale. The sweeper that returns expired reservations to stock ran twice over the same reservations, because of a retry, a duplicated schedule, or two instances running without coordination. Each spurious increment silently inflates available stock, and the extra units then sell through a completely correct purchase path.

Confirm by comparing expiry events in the reservation table against counter increments: every reservation must have contributed exactly one restock. Prevent by making the restock idempotent at the data level — UPDATE reservations SET restocked_at = now() WHERE id = :id AND restocked_at IS NULL, incrementing the counter only if that statement affected a row (10.4). Idempotency in the job's logic is not enough, because the job can die between its two steps.

Two: the counter was rebuilt incorrectly after a failure. If the memory store restarted mid-sale and the counter was reconstructed from a stale snapshot or a wrong formula, the rebuilt value can exceed reality. Prevent by reconstructing only from the database — initial − confirmed − pending — with the sale paused during reconstruction, and by treating a counter restart as an incident requiring verification before resuming rather than as an automatic recovery.

Three: a second write path exists. An administration tool, a customer-service "add stock" action, a promotional job, or a retry path that decrements without going through the script. Prevent with the rule that the script is the only writer, enforced by an audit log recording every counter mutation with its source, so a second writer is visible rather than theoretical.

Four: atomicity was broken by a refactor. Someone moved the per-user check outside the script, or replaced the script with a sequence of individual commands in the belief that the store is single-threaded anyway. Individual commands are atomic; a sequence of them is not, and the gap between two of them is exactly where two requests both see one unit remaining.

The systemic prevention that matters more than any of the four fixes: run the invariant counter + confirmed + pending == initial continuously during the sale, every few seconds, and give it the authority to pause the sale automatically on a mismatch. At 43 units the loss is embarrassing. The same defect on a larger or more expensive sale is a serious financial and legal exposure, and the entire difference between those outcomes is whether the invariant was checked while the sale was running or once in a report the next morning.

The staff-level statement: in a system with exactly one hard invariant, that invariant must be asserted in production continuously and must be allowed to stop the system. A correctness property that is only verified afterwards is not a property you have — it is a report about a property, delivered after the money moved.

Flashcards

FlashThe real problem statement

10,000 units, 2 million people ⇒ reject 99.5% as cheaply and early as possible, and process 10,000 perfectly. Not "handle 160,000 purchases a second".

FlashWhy a row lock fails

Correct, but it serialises: 1–5 ms per holder caps one row at a couple of thousand a second against 160,000 arriving. Then pool exhaustion takes down the rest of the store.

FlashThe funnel

Edge static page and signed token → queue with an honest position → eligibility and bot checks → atomic in-memory counter → database reservation. Each layer rejects ~10× more cheaply than the next.

FlashWhere oversells actually come from

Not the purchase path — the restock. A sweeper returning the same expired reservation twice inflates stock silently. Fix at the data level: conditional update on restocked_at, increment only if a row was affected.

FlashFail closed, uniquely

Counter unavailable ⇒ pause the sale, do not admit uncounted sales. Rebuild as initial − confirmed − pending with the sale paused, then resume deliberately. Everywhere else in this Part, availability wins; here it does not.

FlashThe invariant with teeth

counter + confirmed + pending == initial, checked every few seconds during the sale, with the authority to pause automatically. Verified only afterwards, it is a report rather than a property.

Scenario Drill

DrillGeneralise: your system has a hot partition — one tenant, key or resource taking orders of magnitude more traffic than the rest. Enumerate the toolkit from this Part and give the decision rule.

The same pattern has now appeared in six studies under six different names, and recognising it as one problem is the whole point of this drill: the whale customer in 11.3, the hot key in 11.4, the celebrity in 11.8, the viral video in 11.9, the dense city cell in 11.10, and the single item in this one.

The toolkit, from cheapest to most invasive.

Cache in front of it. If the hot thing is read, an in-process cache with a short lifetime plus request coalescing absorbs essentially unlimited read volume with no network hop (11.4). This solves the majority of hot-key problems outright, and it should always be tried first.

Move the work earlier and cheaper. Reject or answer at the edge so the hot resource never sees the traffic — this study's funnel, and the edge-cached redirect in 11.1.

Split the key. key:0 through key:N, each holding a fraction of the capacity, with callers choosing at random (11.3). Aggregate behaviour is preserved; per-shard behaviour becomes lumpy; and anything needing the true total must now fan in.

Lease or batch. Hand out blocks of the contended resource — tokens, identifier ranges, stock — to instances that spend them locally, collapsing round trips by orders of magnitude at the cost of bounded imprecision at the edges (11.3, 11.5).

Invert the strategy for the hot entity specifically. Do not fan out to the celebrity's followers; fan in at read time instead (11.8). This is the highest-value move when the cost asymmetry is severe, and it is why "hybrid" is the standard answer to feed design.

Dedicate capacity. Give the hot tenant its own shard, pool or cluster. Operationally it is special-casing, and it converts a shared-fate problem into an isolated one, which is often the pragmatic answer for a small number of known whales.

Change the contract. Approximate counts instead of exact (11.9), eventual instead of immediate, sampled instead of complete. The cheapest fix is frequently a requirement nobody actually needed, and it is worth asking before building anything.

The decision rule, in order.

Is it read-hot or write-hot? Read-hot is nearly always solved by caching plus coalescing, and you should stop there rather than reaching for anything cleverer.

Write-hot is the hard case, so ask: does the write need a single consistent view? If no — counters, metrics, rate limits — then split or lease, because imprecision is acceptable and throughput multiplies. If yes — stock, balances, assignment — you cannot split the decision at all, so instead shrink the contended section to microseconds with an atomic in-memory operation and shed load before it arrives. Which is precisely this study's entire design.

And the rule worth memorising: you cannot make a single contended point faster than physics allows, so either stop needing a single point (split, lease, approximate) or stop sending traffic to it (cache, reject at the edge, queue). Every solution in this Part is one of those two, and "add more servers" is neither.