9.5.3 — Deadlock, Livelock and Starvation

At 02:14 the payments service stops responding. No errors, no crash, no memory spike. CPU is at 3 percent. Every request just hangs. The logs end mid-sentence, and the last two lines are from two different transfers:

02:14:07.221  transfer  from=wallet_88  to=wallet_12  locked wallet_88
02:14:07.223  transfer  from=wallet_12  to=wallet_88  locked wallet_12

Nothing after that, forever. The two transfers are each holding the row the other one needs. Neither will ever give it up, because each is waiting for the other to finish first. The rest of the service piles up behind them until every connection is used and the whole thing is dead.

This is a deadlock: a set of threads where each is waiting for something held by another member of the set, so none can ever proceed. It is the most feared concurrency failure because, unlike a race, it does not corrupt data or produce a wrong answer. It produces nothing at all, quietly, and usually at the worst possible time.

1. Seeing it: the wait-for graph

The clearest way to think about deadlock is as a picture. Draw one node per thread, and an arrow from thread A to thread B whenever A is waiting for something B holds. A deadlock exists exactly when that graph contains a cycle.

Figure 1 — Deadlock is a cycle in the wait-for graph. This is not a metaphor; it is the actual definition, and it is how databases detect deadlocks at runtime.

The cycle can be longer than two. Four threads each waiting on the next in a ring is the same failure, and it is far harder to spot in a code review because no two files look suspicious together.

2. The four conditions, and the four cures

In 1971 Edward Coffman showed that a deadlock requires all four of the following to be true at once. That is the useful part: break any single one, and deadlock becomes impossible. Each condition maps to a real engineering technique.

Condition one — mutual exclusion. Some resource can only be held by one thread at a time. Without this there is nothing to wait for.

Breaking it: make the resource shareable. Immutable data can be read by everyone at once. A read-write lock lets all readers in together. Giving each thread its own copy removes the contention completely. This is the strongest cure available, and it is why 9.5.1 ranked "don't share" and "don't mutate" above locking.

Condition two — hold and wait. A thread holds one resource while requesting another.

Breaking it: acquire everything you need in one atomic step, or acquire nothing. If you cannot get all of them, release what you have and start over. Databases expose this as SELECT ... FOR UPDATE over multiple rows in one statement. The cost is reduced concurrency, because you hold things longer than strictly needed, and the risk is livelock, which section 5 covers.

Condition three — no preemption. A resource cannot be taken away from the thread holding it; only the holder may release it.

Breaking it: use timed lock acquisition. tryLockFor(200ms) means a thread gives up rather than waiting forever, releases whatever it holds, and retries. This does not prevent the deadlock from forming, but it guarantees the system escapes it. It is the cheapest safety net in this whole chapter and the one most often missing.

Condition four — circular wait. The wait-for graph contains a cycle.

Breaking it: impose a global ordering on all lockable resources and require every thread to take locks in that order. If everyone always locks the lower id first, a cycle cannot form, because a cycle would require somebody to hold a higher id while waiting for a lower one. This is the single most practical deadlock cure in existence, it costs nothing at runtime, and it is what the transfer code in 9.5.2's drill did with [fromId, toId].sort().

Figure 2 — Lock ordering, the cure worth memorising. Sorting the resources before locking them makes the fourth Coffman condition unsatisfiable, at zero runtime cost.

3. The four strategies, and which ones real systems use

Textbooks list four approaches to deadlock. It is worth knowing all four and being honest about which ones anybody actually uses.

Prevention — design so one of the four conditions can never hold. Lock ordering, timeouts, single-lock designs. This is what real systems do, and it is where you should spend your effort.

Avoidance — at runtime, refuse any lock request that could lead to a future deadlock. The famous algorithm here is Banker's, which requires every thread to declare its maximum resource needs up front so the system can check whether granting a request leaves a safe ordering. It is a beautiful piece of theory and it is essentially never used in application code, because nobody knows their maximum needs in advance and the checking is expensive. Know the name, know why it is impractical, and move on.

Detection and recovery — allow deadlocks to happen, notice them, and break them by killing a participant. This is what databases do, and it works well because a database can see the whole wait-for graph and can roll a transaction back cleanly. PostgreSQL runs a cycle check on its lock graph after a short wait and aborts one transaction with deadlock detected. Your application's job is to catch that error and retry, because the abort was not your fault and the retry will usually succeed now that the other transaction is gone.

The ostrich algorithm — ignore the problem and reboot when it happens. Do not laugh: for a batch job that runs nightly and deadlocks once a quarter, a supervisor that restarts it is a legitimate engineering decision, and pretending otherwise wastes effort that belongs elsewhere. The condition for it being legitimate is that the failure is rare, obvious, and harmless to restart. If any of those three is untrue, you need one of the other strategies.

4. The deadlocks you will actually meet

Textbook examples use two threads and two mutexes. Real ones rarely look like that. Here are the five that show up in production, in rough order of frequency.

Two rows, opposite order

The transfer at the top of this page. Two database transactions lock the same two rows in opposite orders. The database detects it and kills one after a second or two, so it appears in your logs as an occasional deadlock detected error rather than a hang. Teams often "fix" this by adding a retry, which does work, but the real fix is to sort the ids before locking so the deadlock never forms and you stop paying for the aborted work.

A callback that takes a lock

class Inventory {
  #lock = new Mutex();
  #listeners: Array<(sku: Sku) => Promise<void>> = [];

  async adjust(sku: Sku, delta: number): Promise<void> {
    await this.#lock.withLock(async () => {
      this.#apply(sku, delta);
      for (const listener of this.#listeners) {
        await listener(sku);              // ← calling unknown code while holding the lock
      }
    });
  }
}

The listener is code you did not write. Six months from now, somebody registers a listener that calls inventory.adjust() for a related SKU, or that takes the pricing service's lock while the pricing service is taking yours. The deadlock arrives through a file that nobody reviewing this class will ever open.

The fix is a rule, not a patch: collect the events inside the lock, and fire them after releasing it.

async adjust(sku: Sku, delta: number): Promise<void> {
  const events = await this.#lock.withLock(async () => {
    this.#apply(sku, delta);
    return [{ sku }];                     // (1) decide inside, do nothing outside your state
  });
  for (const e of events) {
    for (const listener of this.#listeners) await listener(e.sku);   // (2) lock is released
  }
}

(1) The critical section now touches only your own data and returns a description of what happened. (2) The notifications run with no lock held, so a listener may take any lock it likes, including yours, without a cycle being possible.

The connection pool deadlock

This one is worth its own section because it catches experienced engineers, and because nothing in the code looks like a lock at all.

A service has a pool of 10 database connections. A request handler takes a connection, and partway through, calls a helper that also needs a connection. With 10 requests in flight, all 10 connections are held by handlers that are each waiting for an eleventh connection that will never exist. The pool is not a mutex, but it is a resource with mutual exclusion, hold-and-wait, and a circular wait — all four conditions, satisfied by a pool.

async function generateReport(userId: UserId): Promise<Report> {
  return pool.withConnection(async conn => {              // holds connection #1
    const user = await conn.query("SELECT ...", userId);
    const orders = await loadOrders(user.id);             // ← this also calls pool.withConnection
    return build(user, orders);
  });
}

The tells are always the same: throughput drops to zero, CPU is idle, and the pool's "waiting for connection" metric equals the pool size. The fixes, in order of preference: pass the connection down so the inner call reuses the one you already hold; restructure so the two queries are not nested; or, as a last resort, size the pool above the maximum nesting depth times the concurrency, which is a fragile number that will be wrong again after the next feature.

The same shape appears with any bounded pool — HTTP client connections, worker threads, semaphore permits. A general rule falls out of it: never wait on a pool while holding something from the same pool.

The lock you take twice

A method takes a lock and calls another method in the same class that takes the same lock. With a non-reentrant mutex, the thread waits for itself, forever. It usually appears through refactoring: an inner method gains a lock, and nobody notices that an outer method already holds it.

The fix is not simply "make the lock reentrant", although that works. The better fix is the convention that only public methods lock, and private methods assume the lock is already held. Name the private ones so it is impossible to miss — #applyLocked(...) — and the ambiguity disappears from the code rather than being handled at runtime.

The await inside a lock, in single-threaded code

Node has no threads, so surely it cannot deadlock. It can, and the mechanism is instructive.

const lock = new Mutex();

async function outer(): Promise<void> {
  await lock.withLock(async () => {
    await inner();                 // inner() also does lock.withLock(...)
  });
}

There is exactly one thread, and it is now awaiting a promise that will only resolve when the lock is released, which will only happen when the current callback returns, which it cannot do because it is awaiting. The event loop is fine, other requests continue, but this chain of work is stopped forever and the lock is never released — so every future caller of outer hangs too. What began as one stuck request becomes a permanently broken endpoint. Single-threaded does not mean deadlock-free; it means the deadlock takes a promise-shaped form.

5. Livelock: everybody busy, nobody progressing

A livelock is a deadlock where the threads are not stuck but are still getting nowhere. They keep responding to each other, keep changing state, keep burning CPU, and never complete.

The classic picture is two people meeting in a narrow corridor. Both step left. Both step right. Both step left again. Everybody is moving politely and nobody gets past.

In software it usually arrives as a retry storm. Two transactions conflict; both detect the conflict; both roll back and retry immediately; both conflict again at exactly the same moment, because they restarted in lockstep. Add a hundred clients and the system is at full CPU with a throughput of zero.

The cure is randomness. Retry after a delay, make the delay grow with each attempt, and add a random component so the retriers stop marching in step:

async function withRetry<T>(fn: () => Promise<T>, attempts = 5): Promise<T> {
  for (let attempt = 0; attempt < attempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (!isTransientConflict(err) || attempt === attempts - 1) throw err;  // (1)
      const base = 50 * 2 ** attempt;                                        // (2) 50, 100, 200...
      await sleep(base / 2 + Math.random() * base);                          // (3) jitter
    }
  }
  throw new Error("unreachable");
}

(1) Only retry things that are actually retryable. Retrying a validation error is a bug that hides the real failure. (2) Exponential backoff gives the system time to drain instead of hammering it while it is already struggling. (3) The random component — called jitter — is the part that actually breaks the livelock. Without it, all the retriers wake at the same instant and collide again, having achieved nothing except a longer pause. This exact function reappears at cluster scale in 10.9, where the failure it prevents is called a thundering herd.

A subtler livelock comes from the hold-and-wait cure in section 2. If a thread that cannot get all its locks releases everything and retries, and two threads do this in perfect symmetry, they can release-and-retry forever. Same fix: randomised backoff, so one of them wins.

6. Starvation: the queue that some threads never reach the front of

Starvation is when a thread is permanently denied a resource it needs, even though the system as a whole is making progress. Nothing is stuck. Work is being done. It is just never this thread's work.

Three common causes.

An unfair lock. Most mutexes make no promise about who gets it next. When one is released, whichever thread the scheduler happens to run first may take it, including one that just arrived and never waited. Under continuous pressure a particular waiter can be passed over indefinitely. A fair lock hands the lock to the longest waiter instead, which removes starvation and costs throughput, because it forces a context switch to the specific thread that is owed the lock rather than letting whoever is already running proceed. Most libraries default to unfair for that reason and offer fairness as an option.

Read-preferring read-write locks. Covered in 9.5.2: a steady stream of readers means the writer never gets in. The symptom in production is not an error but a config change that silently never takes effect.

Priority scheduling. If the scheduler always runs the highest-priority runnable thread, a low-priority thread can wait forever whenever higher-priority work keeps arriving. The fix is aging: raise a thread's effective priority the longer it waits, so everything eventually reaches the front.

And the famous compound failure, priority inversion (2.3): a low-priority thread holds a lock that a high-priority thread needs. A medium-priority thread, which needs no lock at all, preempts the low-priority holder. Now the high-priority thread is effectively blocked by the medium one, which is exactly backwards. This is not a thought experiment; it is what repeatedly reset the Mars Pathfinder rover in 1997. The cure is priority inheritance: while a low-priority thread holds a lock that a high-priority thread wants, it temporarily inherits the higher priority so it can finish and get out of the way. Note that this needs the lock to know its owner, which is why a semaphore cannot offer it and a mutex can.

One more worth knowing by name, the convoy effect. A slow thread holds a hot lock. Everyone queues behind it. When it finally releases, all the waiters wake, and because they now run in lockstep they collide on the next shared resource together, forming a new queue there. The system's threads end up travelling as a clump rather than spreading out, and throughput stays low even though the original slow operation is long finished. The cure is the same as everywhere else on this page: shorter critical sections, so no clump ever forms.

7. Finding these in production

Deadlock looks like: requests hang, CPU near zero, no errors, and the pending-request count climbing steadily. The diagnostic is a thread dump — in a threaded runtime, capture it and look for threads in BLOCKED state; most runtimes will even print "found one Java-level deadlock" and name the cycle for you. For database deadlocks, the database log has already told you: deadlock detected with both statements printed. For pool deadlocks, watch the pool's waiting count sitting at exactly the pool size.

Livelock looks like: CPU pinned high, throughput near zero, and a retry or rollback counter climbing fast. The distinguishing feature from deadlock is that the CPU is busy.

Starvation looks like: overall throughput normal, but a latency histogram with a long tail — the median is fine and the 99th percentile is enormous. Averages hide this completely, which is why percentiles are the metric that matters (10.10).

Three metrics worth exporting on any system with real locking: time spent waiting to acquire, not just time held; queue depth at each pool; and the longest current wait, which is the one that goes to infinity when you deadlock and is therefore the cleanest alert.

8. The rules that keep you out of all of this

Take locks in a consistent global order, always. Sort by id. Write it down as a rule for the whole codebase, because a rule that only some files follow provides no protection at all.

Hold one lock at a time if you possibly can. Most two-lock situations dissolve under a little redesign, and no cycle can form with one lock.

Never call out while holding a lock. No I/O, no callbacks, no plugin code, no event handlers. Collect what you need to do, release, then do it.

Always use a timeout. A timed acquisition turns a permanent hang into a visible error. This is your safety net for the deadlock you did not anticipate, and there will be one.

Keep critical sections tiny. Every rule above is easier to obey when the locked region is four lines long.

Prefer designs where the question does not arise. Immutable data, one owner per entity, conditional writes at the database. Every deadlock in this chapter needed at least two locks, and code that takes no locks cannot deadlock.

Next: 9.5.4 stops looking at what goes wrong and starts assembling the shapes that go right — the handful of structures that almost all concurrent systems are built from.

Recall

• Deadlock is a cycle in the wait-for graph: each thread waits for something another member holds. Symptom is a hang with idle CPU.
• Coffman's four conditions, all required: mutual exclusion · hold-and-wait · no preemption · circular wait. Break any one and deadlock is impossible.
• The cure that matters in practice is global lock ordering — sort the resource ids and lock in that order. Zero runtime cost, and the business operation's direction need not match the locking order.
• Real systems use prevention (ordering, timeouts); databases use detection and recovery (they kill a victim, you retry). Banker's avoidance is theory you should name and not implement.
• The five real ones: two rows locked in opposite orders · a callback that locks while you hold a lock · the connection pool deadlock (never wait on a pool while holding from that pool) · taking a non-reentrant lock twice · awaiting a lock you already hold in single-threaded code.
• Livelock: busy, changing state, never progressing — retry storms in lockstep. Symptom is high CPU, zero throughput. Cure is exponential backoff with jitter.
• Starvation: the system progresses but one thread never gets served. Causes: unfair locks, read-preferring read-write locks, priority scheduling. Cures: fair locks, aging, priority inheritance (needs an owner, so a mutex can and a semaphore cannot).
• Metrics that catch all three: lock wait time, pool queue depth, longest current wait.

Self-test: Name the four Coffman conditions and one way to break each. Why does sorting ids prevent deadlock? Describe the connection pool deadlock and its three fixes. How do you tell deadlock from livelock at a glance? What is priority inversion and why can't a semaphore fix it?

Quiz Bank

FoundationalState the four conditions required for deadlock, and give a concrete engineering technique that breaks each one.

All four must hold simultaneously, which is what makes the list useful: you only have to defeat one.

Mutual exclusion — the resource can be held by only one thread. Break it by removing the exclusivity: make the data immutable so every reader can have it at once, use a read-write lock so readers share, or give each thread its own copy. This is the strongest cure because it eliminates the contention rather than scheduling it.

Hold and wait — a thread holds one resource while asking for another. Break it by all-or-nothing acquisition: take every lock you will need in a single step, and if you cannot get them all, release everything and start again. Databases expose this as locking several rows in one statement. The costs are lower concurrency, since you hold resources longer than strictly necessary, and a risk of livelock if two threads keep releasing and retrying in step.

No preemption — a resource cannot be taken from its holder. Break it with timed acquisition: tryLockFor(200ms) means a waiter gives up, releases what it holds, and retries. This does not stop the deadlock forming, but it guarantees the system escapes rather than hanging, and it converts an invisible outage into a countable error. It is the cheapest insurance available.

Circular wait — the wait-for graph has a cycle. Break it with a global ordering: number every lockable resource and require all code to acquire in ascending order. A cycle would require some thread to hold a higher number while waiting for a lower one, which the rule forbids. In practice this is [a, b].sort() before locking, and it is the technique that solves the overwhelming majority of real deadlocks.

The one to reach for by default is ordering, because it costs nothing at runtime, needs no timeouts to be tuned, and prevents the failure instead of recovering from it. The one to always have anyway is the timeout, because it protects you from the lock ordering somebody forgot.

FoundationalDistinguish deadlock, livelock and starvation. How would you tell them apart from a dashboard, without reading any code?

Deadlock — a set of threads each waiting for something another member of the set holds. Nobody moves, ever, without intervention.

Livelock — threads keep executing and keep changing state, but the state changes cancel out and no work completes. They are responding to each other forever.

Starvation — the system as a whole progresses fine, but a particular thread or class of request never gets its turn.

From a dashboard, the single most discriminating signal is CPU.

Deadlock: throughput at zero, CPU near idle, request queue growing without limit, no errors in the logs. The idle CPU is the giveaway, because a service that is failing is usually working hard at failing. Here it is doing nothing at all.

Livelock: throughput at or near zero, CPU pinned high, and a retry or rollback or conflict counter climbing rapidly. Lots of work, no progress.

Starvation: throughput looks normal and the average latency looks fine. The tell is entirely in the distribution: p50 healthy, p99 enormous or unbounded. Averages actively hide this, which is why any latency panel should show percentiles (10.10).

The follow-up you should be ready for is what to do next in each case. For deadlock, take a thread dump and look for the cycle, or check the database log, which has usually already printed both offending statements. For livelock, add jittered exponential backoff to the retry path, then find why the conflict rate is so high in the first place. For starvation, look for an unfair lock or a read-preferring read-write lock, and consider whether the starved class of work needs its own queue rather than fighting for a shared one.

AppliedA report endpoint hangs under load. CPU is 2 percent, the database is idle, and the connection pool metric shows waiting equals 10 with pool size 10. What happened, and what are the fixes in priority order?

What happened: a connection pool deadlock. The handler takes a connection from the pool and then, partway through its work, calls a helper that takes a second connection from the same pool. With ten requests in flight, all ten connections are held by handlers that are each waiting for an eleventh, which does not exist. No thread will ever release, because releasing requires finishing, and finishing requires the connection nobody can get.

Notice this satisfies all four Coffman conditions even though there is not a single lock() call anywhere. A connection is exclusive while checked out. The handler holds one and waits for another. It cannot be taken back. And every waiter is waiting on a resource held by another waiter. A bounded pool is a lock with a count, and it deadlocks like one.

The signature is exactly the metrics given: waiting equals pool size, database idle because nobody is actually running a query, and CPU near zero because everyone is blocked. That combination has essentially one cause, which makes it a fast diagnosis once you have seen it.

Fix one, the correct one: pass the connection down. The helper should accept the connection it is supposed to use rather than reaching into the pool for its own. This removes the nesting entirely, and it has a second benefit that matters more than the deadlock: the outer and inner queries now run on the same connection, which means they can be in the same transaction and see a consistent snapshot. The nested version could not, so it had a subtle correctness problem as well.

Fix two: do not nest at all. Often the two queries do not need to overlap. Fetch the user, release the connection, then fetch the orders. Shorter checkouts mean higher effective pool capacity as well.

Fix three, and only as a stopgap: raise the pool size above the maximum nesting depth times the maximum concurrency. This works arithmetically and is fragile in practice, because the safe number depends on a call-graph property that any future commit can change. If you do this, add a comment naming the constraint so the next person understands what the number means.

What to add so it never surprises you again: a checkout timeout on the pool, so waiting for a connection produces a clean error rather than an unbounded hang, and an alert on waiting-count approaching pool-size. Then state the general rule for the team, because it covers thread pools and semaphores as well: never wait on a pool while holding something from the same pool.

InterviewYour team's fix for repeated database deadlocks is to catch the error and retry three times. It works. Explain why you would still change the design, and what you would change it to.

Granting the point first: the retry is not wrong, and you should keep it. Database deadlock aborts are a legitimate, expected outcome, the database has already rolled the transaction back cleanly, and a retry usually succeeds because the other transaction has now finished. Any system doing multi-row transactions needs that retry path.

But leaving it as the only response is a design decision to pay a recurring cost forever, and here is what that cost is.

Every deadlock throws away real work. The aborted transaction did its reads, took its locks, and did its writes, and all of it is discarded. Under load, the wasted work is proportional to the conflict rate, and the conflict rate grows faster than traffic does, because more concurrent transactions means quadratically more chances to overlap. A system that deadlocks occasionally at current traffic can deadlock constantly at three times current traffic. The retry hides the trend right up until it stops working.

Detection is not instant. The database waits a configured interval — a second by default in PostgreSQL — before running its cycle check, precisely because checking is expensive. So every deadlock adds that latency to a user-facing request, plus the retry's own execution time. Your p99 carries the cost even though your error rate looks clean.

Retries amplify under stress. When the system is already struggling, retries add load exactly when there is least capacity for it, which is how a mild degradation becomes an outage. If the retries are immediate rather than jittered, the retrying transactions restart in lockstep and collide again, which is a livelock wearing a retry's clothing.

The change: make the deadlock impossible instead of survivable. Find the transactions involved — the database log names both statements — and identify the resources they lock in different orders. Then impose a global ordering: sort the ids before locking, so every transaction in the system acquires in ascending order. A cycle then cannot form, and the retry path becomes something that fires almost never instead of constantly.

A few related moves usually come with it. Shorten the transactions, since a transaction that holds locks for 200 ms has forty times the collision window of one that holds them for 5 ms — and the usual culprit is an HTTP call or a slow computation inside the transaction, which should be moved outside. Take the locks in the same order the statements will need them, rather than acquiring one row early and another late. And check whether you need row locks at all: many of these transactions turn out to be expressible as a single conditional update, which cannot deadlock with itself.

The sentence to close on: the retry is the safety net and the ordering is the fix, and a team that has only the safety net has decided to keep paying for a problem they could have deleted.

StaffDesign a review checklist your team can apply to any pull request that touches concurrency, and justify why each item earns its place.

The goal is a short list that a non-specialist reviewer can apply mechanically, because a checklist only works if it does not require the expertise it is substituting for. Seven items.

1. Is every lock released in a finally? A lock released only on the success path wedges the service permanently the first time the body throws, and that first time will be in production because the happy path is what testing covers. This is the single highest-yield item and takes two seconds to check.

2. Are two or more locks ever held at once, and if so, is the order fixed and documented? Two locks is where deadlock becomes possible at all. If the answer is yes, the acquisition must be by a sorted key, with a comment saying so, because the next person to touch this file will not deduce the convention.

3. Does anything inside a critical section do I/O, or call a callback, or fire an event? Both turn a bounded wait into an unbounded one, and the callback case invites a cycle through code that nobody reviewing this file will read. The required shape is: decide inside the lock, act outside it.

4. Is there a timeout on every blocking acquisition — locks, pools, remote calls? This is the safety net for everything the first three items missed. It converts a silent hang into a countable error, and the difference between those two in an incident is roughly an hour of diagnosis time.

5. Does the code take a resource from a pool while already holding one from that pool? The connection-pool deadlock is invisible in a diff because there is no lock keyword to grep for. It has to be looked for deliberately, by following the call graph one level down.

6. Is the invariant also enforced where the state lives? An in-process lock protects only this process. If the change relies on a mutex for correctness rather than for load shaping, ask what happens at two replicas, and push the guarantee into a conditional write or a constraint. This is the item that prevents a class of bug that only appears after a scaling change, which is the worst time to find it.

7. Is there a retry, and if so is it jittered, bounded, and limited to genuinely transient errors? Unjittered retries livelock. Unbounded retries turn a downstream blip into an amplification attack on your own infrastructure. Retrying a validation error hides a real bug behind five identical failures.

Why a checklist rather than expertise. Concurrency bugs are invisible in a diff and do not reproduce in review, so the usual reviewer instinct — read it and see whether it looks right — does not work here. Every item above is a structural question with a yes-or-no answer that can be checked without simulating the interleaving in your head. That property is what makes the list usable by the whole team rather than by one person, which in turn is what makes it get applied at all.

And one process item that belongs with it: when a concurrency bug does reach production, the follow-up is not only to fix that instance but to ask which checklist item would have caught it, and to add one if none would have. A checklist that never grows is a checklist that has stopped learning from the incidents.

Flashcards

FlashCoffman's four

Mutual exclusion · hold-and-wait · no preemption · circular wait. All four needed; break one and deadlock is impossible. Circular wait is the cheapest to break: sort the ids.

FlashDeadlock versus livelock at a glance

Deadlock: zero throughput, idle CPU, growing queue, no errors. Livelock: zero throughput, pinned CPU, climbing retry counter. Starvation: normal throughput, healthy p50, unbounded p99.

FlashConnection pool deadlock

Handler holds a connection and waits for a second from the same pool; all N held, all waiting for N+1. Fix: pass the connection down. Rule: never wait on a pool while holding from that pool.

FlashPriority inversion

Low-priority thread holds a lock the high-priority thread needs; a medium-priority thread preempts the holder. Cure is priority inheritance, which needs a lock that knows its owner — so a mutex can and a semaphore cannot.

FlashJitter

Exponential backoff alone keeps retriers in lockstep, so they collide again. The random component is what actually breaks the livelock and the thundering herd.