Appearance
11.14 — Payment System
You send a charge request to a card processor. Twenty seconds pass. Your HTTP client gives up and throws a timeout.
Three things could be true. The request never arrived, and the customer has not been charged. It arrived and succeeded, and the customer has been charged. It arrived and was declined, and the customer has not been charged.
You cannot tell which, and both obvious reactions are wrong. Retry, and if the first attempt succeeded you have charged someone twice. Mark it failed, and if the first attempt succeeded you have taken money and delivered nothing.
That single ambiguity is why payment systems look different from everything else in this Part. It is not a throughput problem — six hundred requests a second at peak is nothing. It is a problem of never being allowed to guess, and of being able to prove afterwards that you did not.
1. Requirements
Functional. Accept payments through external processors. Authorise, then capture. Refund in full or in part. Pay out to merchants. Keep a ledger that always balances. Handle callbacks from processors. Every mutating endpoint idempotent.
Non-functional, with numbers.
- No double charges. Ever. Not "rarely" — this is the requirement the design exists to satisfy.
- No lost payments.
- Every unit of currency traceable to a transaction. Auditability is a hard requirement with legal weight, not a feature to prioritise.
- p99 under 2 seconds for a payment attempt.
- Card data never touches our servers, which shrinks our compliance obligations from an audit to a questionnaire.
Out of scope today: fraud scoring internals, sourcing exchange rates, and tax computation.
The clarifying questions, and what each answer changes
"Do we hold funds, or does the processor?" If money sits in your account between capture and payout, you are handling other people's money and inherit a set of regulatory obligations that shape the whole design. If the processor holds it, your ledger is a record rather than a bank.
"Can we accept a raw card number?" The answer must be no, and asking establishes it as an architectural constraint rather than a security review comment. The moment a card number enters your servers, your compliance scope expands enormously and your breach exposure changes category.
"Is authorise-then-capture separate, or one step?" Two steps means holding a customer's funds without taking them, which is right for anything that ships later — and it introduces expiry, partial capture, and a whole set of states. One step is simpler and wrong for most commerce.
"How long between capture and settlement?" Days, usually. That gap is why the ledger must distinguish money you have been promised from money you actually have, and it is the entire basis of the payout design in the drill.
"Who is on the hook when a customer disputes a charge?" The answer determines whether you need reserves, and reserves change the merchant balance model. It is a business answer with a direct structural consequence.
2. Estimation
Request volume. 10 million payments a day ÷ 86,400 ≈ 120 a second average, ~600 at peak. What that forces: nothing, and saying so is the point. Six hundred requests a second is a single unremarkable service. Payments are a correctness problem, not a throughput problem, and any design that trades correctness for throughput here has misread the requirements.
Ledger volume. Double-entry means at least two rows per event, and a payment produces several events — authorised, captured, fee taken, settled, paid out. Call it 10 rows per payment × 10 million = 100 million ledger rows a day, at roughly 150 bytes each = 15 GB a day, about 5.5 TB a year, append-only and retained permanently. What that forces: the ledger is the largest storage consumer in the system by far, it is partitioned by time, and nothing in it is ever deleted. Retention is not a tuning decision; in most jurisdictions it is a legal minimum measured in years.
Ambiguous outcomes, which is the number nobody computes. If 0.1% of processor calls time out or return ambiguously — a conservative figure across a year including the processor's own bad days — that is 10,000 payments a day in the unknown state. What that forces: the unknown state cannot be handled by a human. It needs an automated resolver, a queue with an age metric, and an alarm when the age grows. Ten thousand a day is a system, not an exception path.
Idempotency storage. Each key stores a request fingerprint and the response body: 10 million a day at ~2 KB = 20 GB a day, retained for at least 24 hours and preferably longer. What that forces: a retention window that is a deliberate choice. Too short and a client retrying after a long outage creates a second charge; too long and you are storing response bodies indefinitely. Twenty-four hours is a common minimum and seven days is safer.
Reconciliation volume. The processor's daily settlement report lists every transaction it settled. Comparing 10 million lines against your ledger is a batch job over a few gigabytes. What that forces: a real pipeline with a schedule, an owner and an alerting threshold — not a script someone runs when they are worried.
3. API
http
POST /payments
Idempotency-Key: 3f9c1b7a-2e5d-4a80-9c11-6b0f2d7e4a55
{ "amount": 4999, "currency": "GBP", "method": "card",
"paymentToken": "tok_1J9…", "orderId": "ord_8821" }http
201 Created
{ "paymentId": "pay_01J9F6", "status": "authorized",
"amount": 4999, "currency": "GBP",
"processorRef": "ch_3Nx…", "createdAt": "…" }http
POST /payments/pay_01J9F6/capture
Idempotency-Key: …
{ "amount": 4999 }
→ 200 { "paymentId": "pay_01J9F6", "status": "captured" }
POST /payments/pay_01J9F6/refund
Idempotency-Key: …
{ "amount": 1000, "reason": "partial_return" }
→ 201 { "refundId": "ref_01J9G2", "status": "pending" }
GET /payments/pay_01J9F6
→ 200 { …, "status": "unknown", "unknownSince": "…", "resolutionAttempts": 3 }
POST /webhooks/processor # signed callback; verify, store, acknowledge, process laterNote the amount is 4999 and the currency is "GBP", separately. Money is stored and transmitted as an integer number of minor units — pence, cents — with the currency alongside it. Never a floating-point number. 0.1 + 0.2 not being exactly 0.3 is a rounding artefact on a chart and a legal problem in a ledger (3.6.7).
Idempotency-Key is mandatory on every mutating endpoint (10.4), and its semantics are precise. The key is stored with a hash of the request body and with the response that was produced. A replay with the same key and the same body returns the stored response without re-executing anything. A replay with the same key and a different body is a client bug and must return 422 — it must never silently do something new, because a key being reused for a different operation is exactly the situation where guessing causes a wrong charge.
status: "unknown" is a real status the API returns, and that is unusual enough to be worth defending. Most APIs hide uncertainty. Here, hiding it means either lying to the caller or guessing on their behalf, and both are worse than saying "we do not yet know, and here is how long we have not known for".
Webhooks are acknowledged fast and processed later. The handler verifies the signature, writes the event to a durable store, and returns 200 immediately. Doing the work inline means a slow handler causes the processor to time out and retry, which multiplies the load exactly when the system is already struggling.
4. Data model
payments
payment_id UUID PRIMARY KEY
order_id TEXT NOT NULL
amount_minor BIGINT NOT NULL -- integer minor units
currency CHAR(3) NOT NULL
state SMALLINT NOT NULL -- pending|authorized|captured|failed|unknown|refunded
processor TEXT, processor_ref TEXT
idem_key TEXT NOT NULL
unknown_since TIMESTAMPTZ NULL
created_at, updated_at
idempotency
key TEXT PRIMARY KEY
request_hash BYTEA NOT NULL
response_body JSONB
response_code SMALLINT
expires_at TIMESTAMPTZ NOT NULL
ledger_entries -- APPEND ONLY. never updated, never deleted.
entry_id BIGINT PRIMARY KEY
txn_id UUID NOT NULL -- entries of one transaction share this
account_id TEXT NOT NULL
direction CHAR(2) NOT NULL -- 'DR' or 'CR'
amount_minor BIGINT NOT NULL
currency CHAR(3) NOT NULL
created_at TIMESTAMPTZ NOT NULL
accounts
account_id TEXT PRIMARY KEY
type SMALLINT -- asset | liability | revenue | expense
owner_id UUID NULL, currency CHAR(3)
webhook_events
event_id TEXT PRIMARY KEY -- the processor's id; dedupe on this
payload JSONB, received_at, processed_at NULL
outbox
id, aggregate_id, event_type, payload, published_at NULLAccess patterns:
| Query | Frequency | Returns |
|---|---|---|
| Insert an idempotency key | 600/s peak | success, or a conflict |
| Read a payment by id | high | one row |
| Append ledger entries for a transaction | 600/s × ~4 rows | — |
| Sum an account's entries | on demand and by job | a balance |
| Find unresolved unknown payments | continuous | thousands |
| Compare a day of ledger against a report | daily | 10M rows |
ledger_entries is partitioned by time and never modified. No UPDATE, no DELETE, no exceptions. A correction is a new reversing entry, so the history of every unit of currency is complete and reconstructible. This is simultaneously an engineering property — you can answer "what did this account look like at 14:00 last Tuesday" exactly — and a legal one.
A balance is derived, never stored as truth. It is the sum of an account's entries. Materialised snapshots exist for performance, and they are treated as a cache: rebuildable, and never the thing anyone reasons from when the numbers disagree.
The unique index on idempotency.key is the deduplication mechanism, and it must be a constraint rather than a check-then-insert. Two concurrent retries of the same request will both pass a check and both proceed; only an atomic insert stops one of them.
5. The double-entry ledger
Three consequences follow from the diagram.
Conservation becomes checkable. SUM(debits) = SUM(credits) holds per transaction, per account, and globally, and it can be asserted continuously as a monitored number. A bug that would have created money now fails an assertion instead of quietly succeeding.
The modelling is forced to be honest. Money moving must name both ends, which surfaces questions a single balance field lets you postpone: where does the fee come from, which account holds funds between capture and payout, what happens to the money when a refund is issued and the payout has already gone out. Those questions become incidents if they are not answered at design time.
Corrections are additions. A mistake is fixed by writing a reversing entry, not by editing the original. History stays intact, and "what happened to this money" always has an answer.
The costs, stated fairly. An order of magnitude more rows. A mental model the team has to genuinely learn, because debits and credits are not simply additions and subtractions and getting the sign conventions wrong is a real onboarding hazard. And derived balances need materialised snapshots to be fast. All of that is easily worth it, because the alternative is a system whose correctness cannot be checked at all.
6. Architecture
7. Deep dives
7.1 The unknown state, which is the heart of payment engineering
A processor call that times out has three possible truths and the system must not choose between them.
Retrying blindly is how customers get charged twice. Marking it failed is how payments vanish — the customer is charged, your system believes nothing happened, and the goods are never sent. Both errors are discovered by the customer rather than by you, which is the worst possible discovery path.
So the payment moves to unknown, recording the processor reference, the idempotency key, and the time it entered that state. Then it is resolved by one of three mechanisms, in order of speed:
Ask the processor. Every major processor lets you query by your own idempotency key. That is what makes the key valuable at the boundary you do not control — you can ask "what happened to the request I identified as X" rather than "did anything happen".
Wait for the webhook. The processor will tell you the outcome asynchronously, usually within seconds.
Reconciliation. The daily report is the backstop for anything the first two missed.
Passing your idempotency key through to the processor is non-negotiable, because it is what makes a safe retry possible at all. Without it, retrying is a coin flip; with it, the processor deduplicates on your behalf and a retry is genuinely free.
And the unknown state needs an age metric with an alarm. Ten thousand a day (section 2) means a resolver that stops working is invisible in an error rate, because nothing is erroring. What you watch is the age of the oldest unresolved unknown, and it should be measured in seconds.
7.2 The four layers that make a double charge structurally impossible
Layer one: the client generates the key once and persists it. Once per payment attempt, written to local storage before the first request. A key regenerated on each retry provides no protection at all, and this is the single most common implementation error — it produces exactly the symptom "duplicates, but only when the network is bad".
Layer two: the server stores the key atomically. An insert against a unique constraint, not a check followed by an insert, because two concurrent retries both pass a check. Stored alongside a hash of the request body and the response that was produced.
Layer three: the key is forwarded to the processor. If your service crashes after calling the processor and the retry reaches the processor again, the processor deduplicates. This covers the window your own store cannot.
Layer four: the unknown state. Ambiguity is recorded rather than resolved by assumption.
And the backstop: reconciliation. Anything that slips through all four is caught within a day and refunded proactively — which matters, because the customer noticing first is a materially worse outcome than your own job noticing.
7.3 Webhooks are hostile until proven otherwise
They arrive asynchronously, out of order, more than once, and occasionally from someone who is not the processor. Four rules.
Verify the signature before anything else. Before parsing, before logging, before touching the database. An unsigned or wrongly signed callback is an attacker telling you a payment succeeded.
Acknowledge fast, process later. Store the event and return 200 within milliseconds. Processing inline means a slow handler causes the processor to time out and retry, which multiplies inbound load precisely when you are already slow — a feedback loop with your own reliability at the bottom of it.
Deduplicate by the processor's event identifier. Delivery is at-least-once, so the same event will arrive twice, and a second captured event must not write a second set of ledger entries.
Tolerate out-of-order arrival. A captured event can arrive before the authorized event that logically precedes it. Handlers must therefore be state-machine guarded — each transition checks the current state and ignores what it cannot apply — rather than assuming a sequence that the network does not guarantee.
7.4 Reconciliation is the only proof of correctness
Every day, the processor's settlement report is compared line by line against the ledger. Three classes of discrepancy, each needing an owner and a written procedure.
Present at the processor, absent from our ledger. We lost track of a payment — usually an unknown that never resolved, or a webhook that was never processed. The money exists and our records do not know about it.
Present in our ledger, absent at the processor. Worse. We recorded something that did not happen, which means our numbers overstate reality — and if we have already paid a merchant on the strength of it, we are out of pocket.
Amount mismatch. Fees, currency conversion, or a partial capture that was recorded at the wrong value.
The sentence that matters: a payment system without automated daily reconciliation is not "mostly correct", it is unverified — and the difference becomes very concrete the first time a regulator, an auditor or a large merchant asks you to demonstrate that your numbers are right.
7.5 Keeping card data out of your systems
Compliance scope here is an architectural decision, not a paperwork exercise.
The client sends card details directly to the processor — through hosted input fields or the processor's own client library — and receives a token back. Your servers see only the token. Your obligations collapse from the full audited standard to a short self-assessment, and a breach of your database exposes tokens that are useless anywhere else.
Any design that accepts a raw card number into your API has made a very expensive decision, and it is expensive in two directions at once: an ongoing audit burden, and a breach whose consequences are measured in card reissues and fines rather than in apology emails (9.9.5).
7.6 Coordinating across services without distributed transactions
Placing an order touches inventory, payment and fulfilment. A single transaction across all three plus an external processor is not available — the processor is a third party who will not join your transaction, and no protocol changes that.
The workable model is a saga (10.8.4): reserve inventory, charge, confirm the order, with a compensating action for each step if a later one fails — release the reservation, refund the charge.
The costs must be stated rather than hidden. There is a window where the system is temporarily inconsistent and a user can see it — money taken, order not yet confirmed. And a compensation is a business operation, not a rollback: refunding a charge is not the same as it never having happened. It appears on a statement, it may incur a fee, and the customer will see it. Designing as if compensation were undo produces surprises that are visible to customers.
7.7 Money-specific rules that override general engineering instincts
Never a floating-point number. Integer minor units or a decimal type. The rounding error that is invisible in a chart is a discrepancy in a ledger, and discrepancies in ledgers are found by auditors.
Always store the currency with the amount. An amount without a currency is not a quantity of money, and the bug where two currencies are added together does not announce itself.
Never update or delete a ledger row. Corrections are reversing entries.
Never use a local clock to order financial events (10.3). Use the processor's timestamps and your own monotonic sequence, because two servers disagreeing by 200 milliseconds is enough to record a refund before the charge it reverses.
Make every financial state transition guarded and idempotent, so that a retry is always safe and a duplicated message can never move money twice.
8. Decision Ledger
| Decision | Alternatives | Why this | What it costs |
|---|---|---|---|
| Idempotency keys required on every mutating endpoint | best-effort deduplication | double charges become structurally impossible rather than unlikely | clients must generate and persist keys correctly, and many will not at first |
| Key stored atomically with a request hash | check, then insert | two concurrent retries both pass a check | a unique constraint and a 422 path for key reuse |
| The key is forwarded to the processor | keep it internal | covers the window where your own service crashed after calling out | you depend on the processor supporting it, which they all do |
An explicit unknown state | assume failed; retry blindly | ambiguity is real, and guessing loses or duplicates money | more states, a resolver job, and a user-visible "processing" |
| Double-entry, append-only ledger | a balance column | conservation becomes a continuously checkable assertion | ten times the rows, and a model the team must genuinely learn |
| Balances derived, snapshots as cache | store the balance as truth | a stored balance can drift from its own history silently | recomputation cost, and snapshot invalidation |
| Saga with compensations | a distributed transaction | not available across an external processor, at any price | temporary visible inconsistency; compensations are business events |
| Tokenise at the client | accept card data server-side | compliance scope collapses and breach impact shrinks to tokens | client integration work, and closer coupling to one processor |
| Daily automated reconciliation | trust the system | it is the only actual proof that the system is correct | a pipeline, runbooks, and owned discrepancy queues |
9. Scale and failure
| What breaks | Blast radius | How you find out | What keeps it running | Recovery |
|---|---|---|---|---|
| Processor times out | one payment, ambiguously | timeout rate | payment enters unknown; the resolver takes it | status query by key, webhook, or reconciliation |
| Processor down | all new payments | breaker state; error rate | fail fast with an honest message; route to a secondary processor if you have one | breaker recovers; queued attempts resume |
| Unknown resolver stalls | 10,000 a day pile up, silently | age of the oldest unresolved unknown | nothing errors, which is why the age metric exists | restart; the queue drains; reconciliation catches the rest |
| Webhook endpoint down | outcomes stop being recorded | webhook ingest rate falling to zero | processors retry for hours, so a short outage self-heals | reprocess; the endpoint must be idempotent |
| Duplicate webhook | none | deduplication counter | the event-id primary key | none needed |
| Ledger write fails after processor success | one payment's records | ledger error rate; then reconciliation | the outbox and reconciliation both catch it | write reversing or missing entries deliberately |
| Ledger invariant violated | correctness of everything | continuous debits-equal-credits assertion | the assertion is the detector; nothing else would notice | stop writes on that account; correct with reversing entries |
| Clock skew between services | events ordered wrongly | sequence anomalies in the ledger | order by processor timestamp and a monotonic sequence | re-derive ordering; never trust local time |
| Reconciliation finds a spike | trust in the whole ledger | discrepancy count, and its rate of change | see the staff question below | classify, bound, stop the bleeding, then remediate |
Two rows deserve emphasis because neither produces an error. A stalled unknown-resolver looks perfectly healthy: no exceptions, no failed requests, just a growing pile of payments in a state nobody is watching. And a violated ledger invariant is only visible if you assert it — otherwise money is created or destroyed and the system continues cheerfully. Both need a metric that exists specifically to notice a lack of progress (10.10).
Multi-processor routing is worth naming as both a resilience and a cost decision. A second processor gives you somewhere to go when the first is down, and it gives you leverage on pricing — and it costs you a second integration, a second reconciliation pipeline, and a routing decision that must be recorded per payment so that a refund goes back to the processor that took the money.
What the interviewer will push on
"The processor call times out. What do you do?" This is the question the whole design answers, and a one-word answer is wrong. Name the three possible truths, say that both obvious reactions cause a specific harm — blind retry double-charges, assume-failed loses the payment — and then describe the unknown state with its three resolution paths, in order of speed. The tell is mentioning that you pass your idempotency key through to the processor, because that is what makes asking "what happened to request X" possible at all.
"Why not just store a balance?" Because a wrong balance is silent. A bug that adds to one account and forgets the other creates money, and nothing objects. With entries that must sum to zero, the same bug fails a continuously running assertion. Then add the second and third properties — append-only history so a correction is an addition, and the modelling being forced to name both ends of every movement, which surfaces questions a balance field lets you postpone until they become an incident.
"How do you guarantee no double charge?" Four layers, and the interesting part is layer one: the client must generate the key once and persist it before the first attempt. A key regenerated on retry provides no protection, and it produces the exact symptom "duplicates only when the network is bad". Then the atomic server-side insert, the forwarding to the processor, the unknown state, and reconciliation as the backstop.
"A webhook arrives saying captured for a payment you think is still pending. What happens?" Nothing dramatic, and that is the point. Webhooks are out of order and at-least-once, so every handler is guarded by the state machine and idempotent by the event identifier. A captured arriving before authorized is applied if the transition is legal and held otherwise. A candidate who assumes ordered delivery from a third party has not integrated with one.
"Reconciliation found 340 discrepancies this morning, up from three. What is your first move?" Classify before investigating — the distribution across the four discrepancy types usually identifies the cause on its own. Then bound the blast radius in time and money, and stop the bleeding by disabling the affected path rather than the whole system, because payments are revenue and a full stop must be justified against a partial one.
"Where does card data live?" Nowhere in your systems. The client tokenises directly with the processor and you store the token. Then the reason, which is architectural rather than procedural: it collapses your compliance obligations and makes a breach of your database expose values that are useless elsewhere.
Volunteer this, because nobody asks: reconciliation is the last line of defence, and a system that depends on its last line for detection has no defence in depth. The goal is that reconciliation finds nothing — so every discrepancy it does find should be treated as evidence of a missing upstream detector, not merely as a record to correct. The upstream detectors that should have caught the last incident are the webhook handler's error rate, the age of the oldest unresolved unknown, and the continuous debits-equal-credits assertion. Naming those three, and the fact that reconciliation catching something means one of them was absent, is the difference between running a payment system and building one.
Next: 11.15 — payments care that money is never lost. An exchange cares about that and about the order in which two orders arrived, measured in microseconds, where being second is the whole difference between a trade and no trade.
Recall
- Payments are a correctness problem, not a throughput problem — ~600 a second at peak. Any design trading correctness for throughput has misread the brief.
- The unknown state is the heart of it. A timeout has three possible truths. Blind retry double-charges; assume-failed loses the payment. Record
unknownand resolve by status query using your key, webhook, or reconciliation. At 0.1% ambiguity that is ~10,000 a day, so it needs a resolver and an age alarm, not a human. - Four layers against double charging: the client persists the key before the first attempt · the server inserts it atomically with a request hash (same key + different body ⇒
422) · the key is forwarded to the processor · the unknown state. Reconciliation is the backstop. - Double-entry, append-only ledger.
SUM(DR) = SUM(CR)is a continuously asserted invariant, so money-from-nothing fails an assertion instead of succeeding silently. Corrections are reversing entries. Balances are derived, never stored as truth. - Money is integer minor units with the currency alongside, never a float. Never order financial events by a local clock.
- Webhooks: verify the signature first · acknowledge fast and process later (a slow handler causes retries that multiply load) · deduplicate by event id · tolerate out-of-order arrival through a guarded state machine.
- Reconciliation is the only proof of correctness. Three discrepancy classes, each with an owner and a runbook. Without it the system is not mostly correct, it is unverified.
- Tokenise at the client so card data never enters your servers — compliance scope collapses and a breach exposes useless tokens.
- Cross-service flows are sagas with compensations, and a compensation is a business event, not an undo.
Self-test: Give the three truths behind a timeout and the resolution path for each. Why does a balance column fail where paired entries do not? Which layer of idempotency do clients most often get wrong, and what symptom does that produce? Name the four webhook rules. Why is reconciliation finding something a sign of a missing detector?
Quiz Bank
FoundationalHow do you guarantee a customer is never double-charged?
Four layers, each covering the previous one's failure, plus a backstop.
Layer one: the client generates an idempotency key once and persists it. One key per payment attempt, written to local storage before the first request is sent, and reused across every retry including after a crash or an app restart. This is the layer implementations most often get wrong: a key generated freshly on each retry provides no protection whatsoever, and it produces the exact symptom "duplicate charges, but only when the network is unreliable" — because that is when a retry follows a crash.
Layer two: the server stores the key atomically. On receipt it inserts the key together with a hash of the request body, against a unique constraint. If the key exists with a matching hash, the stored response is returned without re-executing anything. If it exists with a different hash, the answer is 422, because the client is reusing a key for a different operation and that is a bug that must be surfaced rather than absorbed. The insert must be atomic rather than a check followed by an insert, or two concurrent retries both pass the check and both proceed (10.4).
Layer three: forward the key to the processor. Every major processor accepts an idempotency key of your own. Forwarding it means that even if your service crashes after calling the processor, a retry reaching the processor a second time is deduplicated by them. That covers the one window your own store cannot see: the gap between the outbound call and your record of it.
Layer four: the unknown state. If the call times out, the payment is recorded as unknown rather than retried blindly or marked failed. Resolution comes from querying the processor by your key, from the webhook, or from reconciliation.
And the backstop. Daily reconciliation compares the processor's settled transactions against the ledger, so a double charge that somehow passed all four layers is detected within a day and refunded proactively. That last word matters: the customer noticing first is a materially worse outcome than the same error being caught and corrected by your own job before they see their statement.
InterviewWhy double-entry bookkeeping instead of just storing balances?
Because a balance column can be wrong silently, and paired entries cannot.
With a balance field, a bug that adds 5,000 to one account and forgets to remove it from another creates money from nothing, and nothing in the system objects. The error is detectable only by comparison with something external, long after the fact, and with no way to determine when it started or how much has accumulated.
Double-entry makes conservation structural. Every transaction writes at least two entries that sum to zero — a £49.99 payment debits customer_receivable 4999 and credits merchant_payable 4699 plus platform_revenue 300 — so SUM(debits) = SUM(credits) becomes an invariant you can assert per transaction, per account, and globally, continuously, as a monitored number in production. The bug that would have created money now fails an assertion instead of quietly succeeding.
Second property: the ledger is append-only. A correction is a new reversing entry rather than an edit, so the complete history of every unit of currency is reconstructible. A balance becomes a derived quantity — the sum of entries — never a stored fact that can drift away from its own history. That gives audit compliance essentially for free, lets you answer "what did this account look like on the third of March" exactly, and removes the entire class of bug where one half of an update succeeds and the other fails.
Third property, and it is underrated: it forces honest modelling. Money moving must name both ends, which surfaces questions a single balance field lets you postpone — where does the fee come from, which account holds funds between capture and payout, what happens when a refund is issued after the merchant has already been paid. Those questions do not go away if you avoid them; they turn into incidents.
The costs, stated fairly. An order of magnitude more rows. A mental model the team must genuinely learn, since debits and credits are not simply additions and subtractions and mixing up the sign conventions is a real onboarding hazard. And derived balances need materialised snapshots for performance, which introduces a cache that must be treated as one.
All of which is easily worth it, because the alternative is a system whose correctness cannot be checked at all — only assumed.
InterviewA processor call times out. Walk me through exactly what happens next.
First, name what you do not know. The request may never have arrived, may have arrived and succeeded, or may have arrived and been declined. There is no information available at this moment that distinguishes them.
So both instinctive reactions are wrong, and each has a specific victim. Retrying blindly charges the customer twice if the first attempt succeeded. Marking it failed means the customer has been charged and your system believes nothing happened, so the goods are never sent — and the customer discovers this rather than you.
The payment moves to unknown, recording the processor reference, the idempotency key, and a timestamp for when the ambiguity started. Nothing is written to the ledger, because the ledger records what happened and nobody knows what happened.
Then three resolution paths, in order of how fast they usually work.
Query the processor using your own idempotency key. This is what the key buys you at a boundary you do not control: you can ask "what became of the request I labelled X" rather than the unanswerable "did anything happen". Most processors expose exactly this.
Wait for the webhook. The processor will report the outcome asynchronously, usually within seconds, and the handler resolves the unknown through the same guarded state machine as any other outcome.
Reconciliation. The daily settlement report is the backstop for anything the first two paths missed — a webhook that never arrived, a status query that kept failing.
The operational point that turns this from a design into a working system: at a conservative 0.1% ambiguity rate, this is ten thousand payments a day, which is far past what anyone will handle by hand. It needs an automated resolver, and — because a stalled resolver produces no errors at all — the metric that matters is the age of the oldest unresolved unknown, alarmed in seconds rather than hours. Watching a count instead of an age is how a stalled resolver runs unnoticed for a week.
StaffReconciliation finds 340 discrepancies this morning, up from a typical 3. Walk through your response.
Treat it as an incident, not a data-quality chore. A hundredfold jump means either the system started behaving differently or the reporting did, and both are urgent, because the ledger's trustworthiness is the product.
First, classify before investigating. Bucket the 340 by type: present at the processor and absent from our ledger; present in our ledger and absent at the processor; amount mismatch; status mismatch. The distribution usually names the cause on its own. A wave of the first means outcomes are being lost — a webhook handler that started failing, or the unknown-state resolver stalling. A wave of the second is worse and means we are recording things that did not happen, typically a bug writing ledger entries before confirmation, or a non-production processor key reaching production. Amount mismatches cluster around fee, rounding or currency-conversion logic. Status mismatches usually mean an ordering or state-machine defect.
Second, bound the blast radius in time and in money. Find the earliest affected transaction, which brackets the change and can be cross-referenced against deploys, processor API changes and configuration edits. Compute the total value at risk and, critically, the direction: customers overcharged is a stop-the-line event; the platform under-collecting is serious but not customer-affecting, and that difference should drive how aggressively you intervene.
Third, stop the bleeding before finding the root cause. If discrepancies are still accruing, disable the specific affected path — one payment method, one processor, one merchant — rather than the whole system. Payments are revenue, and a full stop has to be justified against a partial one.
Fourth, root-cause using the evidence the design already provides. The ledger is append-only and the outbox records every event, so the exact sequence for any affected payment is reconstructible. Replay a handful end to end and the defect usually presents itself. In rough order of likelihood: a webhook handler that started throwing, so outcomes stopped being recorded — and note that its error rate should have alerted independently and did not, which is itself a finding; unknown-state payments never resolving because the resolver silently failed; a processor-side change in fee structure or settlement timing that the comparison logic does not model, which produces real-looking discrepancies from a reporting bug; or a deploy that changed rounding or currency handling.
Fifth, remediate deliberately. Every affected payment gets a decision with a record: refund, re-charge, ledger correction by reversing entry — never an edit — or "no action, reporting artefact". Customer-affecting corrections get proactive communication, because being told by their provider is vastly better than finding it on a statement.
Sixth, and this matters more than the fix: the post-mortem finding. Reconciliation caught this, which is good, and it caught it twenty-four hours late. The upstream signals — webhook handler error rate, age of unresolved unknowns, and the continuous debits-equal-credits assertion — should have caught it within minutes, and their absence or their failure to alert is the actual defect. What comes out of this incident is: alerting on the rate of change of discrepancy count rather than an absolute threshold, a monitored age for unresolved unknowns, continuous ledger-invariant assertions, and reconciliation running hourly for high-value flows.
The staff framing: reconciliation is the last line of defence, and a system that relies on its last line for detection has no defence in depth. The goal is for reconciliation to find nothing — so every discrepancy it does find is evidence of a missing upstream detector, and the post-mortem's real output is the name of that detector.
Flashcards
FlashThe idempotency chain
Client persists the key before the first attempt → server inserts key + request hash atomically against a unique constraint → the key is forwarded to the processor. Same key with a different body returns 422, never a new action.
FlashThe unknown state
A timeout means never-arrived, succeeded, or declined — and you cannot tell. Blind retry double-charges; assume-failed loses the payment. Record unknown; resolve by status query using your key, webhook, or reconciliation. Alarm on the age of the oldest unresolved one.
FlashDouble-entry
Every transaction writes entries summing to zero, so SUM(DR) = SUM(CR) is a continuously checkable assertion and money-from-nothing fails it. Append-only; corrections are reversing entries; the balance is derived, never stored as truth.
FlashWebhook rules
Verify the signature first · acknowledge fast and process asynchronously (a slow handler triggers retries that multiply load) · deduplicate by the processor's event id · tolerate out-of-order arrival with a guarded state machine.
FlashMoney storage rules
Integer minor units, never floating point. Always store the currency alongside the amount. Never update or delete a ledger row. Never order financial events by a local clock.
FlashWhat reconciliation really is
The only proof the system is correct — and the last line of defence. Every discrepancy it finds means an upstream detector is missing: webhook error rate, unknown-state age, or the ledger invariant.
Scenario Drill
DrillDesign the marketplace payout side: merchants paid weekly, minus fees and refunds, across currencies and jurisdictions, with a guarantee that we never pay out money we do not have.
Start with the guarantee, because it dictates the data model. Never pay out funds you do not hold. That means the ledger must distinguish available from pending, and the split is a function of settlement, not of authorisation.
A customer's payment is authorised instantly and captured shortly after, but the money reaches your account only when the processor settles, days later. So a merchant's available balance must reflect settled funds, with everything else sitting in pending. Paying out on captured-but-unsettled money means fronting cash, which is a deliberate financial product with credit risk and a funding requirement — never something that should emerge accidentally from a query joining the wrong state. Model it as distinct ledger accounts per merchant: pending_settlement, available, reserve, paid_out, with money moving between them only on explicit, evidenced events.
Refunds and disputes are why a reserve exists. A merchant paid in full on Monday whose customer disputes the charge on Friday leaves you chasing a merchant who has the money and may not return it. A rolling reserve — a percentage held for a number of days, or a risk-scored amount — is standard. It must be visible to the merchant in their dashboard with a clear release schedule, because an opaque hold is the single largest source of merchant support load in every marketplace that has ever built one.
The payout run is a durable, resumable batch (11.18). Compute each merchant's available balance as of a cutoff, subtract fees and reserve, apply the minimum-payout threshold, and produce a payout record keyed idempotently by merchant and period so that re-running the job can never pay twice. Move the balance from available to paid_out in the same transaction that creates the payout record, then submit to the bank rail.
Bank transfers have their own unknown state — submitted, unknown, settled, returned — so the payout record carries its own state machine and its own reconciliation, this time against bank statements. A returned transfer must reverse the ledger movement with reversing entries and restore the balance, not quietly disappear, because a merchant whose payout failed and whose balance did not come back will notice.
Multi-currency adds one rule that must never bend: you never net across currencies implicitly. Each currency is a separate account with its own balance. Conversion is an explicit transaction with the rate and timestamp recorded, so that the rate used is auditable months later. The spread earned on conversion is its own revenue account. And holding a balance in a currency is a market exposure — a treasury decision that should be surfaced to someone who owns it, rather than emerging as a side effect of where payments happened to come from.
Jurisdiction brings requirements engineering cannot solve but must accommodate. Identity verification gating payouts, so a merchant who has not completed it accrues a balance but cannot withdraw — which is a state on the merchant, checked at payout time, not a filter applied somewhere in a report. Tax withholding and reporting thresholds. Sanctions screening before each transfer. And per-country rails with different formats, cut-off times and failure semantics — which is precisely why the payout rail should sit behind an adapter interface (9.4.7), and why the differences will be in timing and failure behaviour rather than in field names.
The reconciliation obligation doubles. Inbound: processor against your ledger. Outbound: your ledger against bank statements. Money leaving must be verified as carefully as money arriving, and it is the half that teams routinely forget because it feels like the end of the process rather than the beginning of a risk.
The sentence for the design document: payouts are payments inverted, with strictly higher stakes — the failure mode is not an unhappy customer but an unrecoverable transfer, so every movement must be idempotent, evidenced, reversible by a reversing entry, and reconciled against an external record you do not control.