10.13 — Blockchain & Web3: An Honest Engineering Tour

Part 10 closes with the technology that took distributed-systems ideas mainstream and simultaneously generated the most confusion about them. This page is an engineering treatment: what a blockchain is mechanically, what its consensus buys that 10.7.2's does not, the real trade-offs in numbers, smart contracts and what they can and cannot do, and — the section that matters most for your career — an honest assessment of where the technology fits and where a database is simply better. ^{⚑What is Web3 and blockchain, honestly assessed? [EQ-149b]}

1. What a blockchain actually is

Mechanically, a blockchain is a replicated append-only log (10.8.1's structure) with three additions:

1. Hash chaining — each block contains the cryptographic hash of the previous block, so altering an old block changes its hash, invalidating every subsequent block. This gives tamper evidence: modification is detectable by anyone holding the chain. (Inside a block, transactions are summarized by a Merkle tree so a single transaction's inclusion can be proven with a logarithmic-size proof rather than the whole block — the same structure used for anti-entropy in 10.5.)
2. Digital signatures — each transaction is signed by the holder of a private key (Part 8.2), so authorization is cryptographic rather than administrative: no server decides whether you may spend; your key does.
3. Byzantine fault tolerant consensus — the new part, and the reason blockchains exist.

The genuinely novel property is not "distributed database" (we've had those for decades) and not "immutable log" (an append-only log with hashes is old). It is: a set of mutually distrusting parties, with no shared administrator, agreeing on an ordered history. That is the problem Bitcoin solved, and it's worth stating precisely because it's the only property that justifies the costs.

2. Byzantine consensus: what it adds over Raft

10.7.2's Raft tolerates crash faults: nodes may stop or be slow, but the ones that respond are assumed honest. Byzantine fault tolerance (BFT) tolerates nodes that lie — sending different messages to different peers, forging, or colluding. The classical result (Lamport's Byzantine Generals) is that BFT consensus needs 3f+1 nodes to tolerate f malicious ones (versus 2f+1 for crash faults), and classical BFT protocols (PBFT and descendants) require knowing the participant set in advance.

Nakamoto consensus (Bitcoin, 2008) solved the open-membership case — anyone may join, nobody is identified — by making participation costly and agreement probabilistic:

• Proof of work — miners race to find a nonce whose block hash meets a difficulty target. Finding it is expensive (energy); verifying it is trivial. The longest (most-work) chain wins, so rewriting history requires redoing all that work faster than the honest network — the famous "51% attack" threshold.
• Probabilistic finality — a block is never certain, only exponentially unlikely to be reversed as more blocks build on it (hence "wait for 6 confirmations"). This is a genuinely different guarantee from Raft's deterministic commit, and it's the price of open membership.

Proof of stake (Ethereum since 2022) replaces energy with capital at risk: validators bond tokens, are selected to propose/attest, and are slashed (lose stake) for provable misbehavior. It achieves comparable security assumptions at ~99.9% less energy, and modern designs add finality gadgets that give deterministic finality after a couple of epochs — moving closer to classical BFT guarantees while keeping open participation.

3. The trade-offs, in numbers

The honest comparison, which the field's marketing usually omits:

Property	Blockchain (public L1)	Conventional distributed DB
Throughput	~15–7,000 TPS (chain-dependent)	10⁴–10⁶ TPS
Latency to finality	seconds to minutes	milliseconds
Cost per write	cents to dollars (gas)	~free
Storage	every node stores everything	partitioned
Trust model	no trusted party required	operator is trusted
Governance	protocol rules + social consensus	whoever has admin rights

The trade is stark and singular: you sacrifice three to five orders of magnitude of throughput, latency, and cost — in exchange for removing the need to trust an operator. That's it. Every architectural decision should follow from asking whether trustlessness is worth those orders of magnitude in your specific case. The scaling responses are real but partial: layer 2 (rollups batching thousands of transactions into one L1 commitment — optimistic with fraud proofs, or ZK with validity proofs), sharding, and app-specific chains all improve throughput while adding complexity and, often, reintroducing trusted components.

4. Smart contracts and their limits

A smart contract is code deployed to the chain that executes deterministically on every node as transactions invoke it — programmable, self-enforcing logic without an operator. Genuine capabilities: escrow and conditional payment without an intermediary, token issuance and transfer rules, automated market makers, and auditable, unchangeable rules (which is either the feature or the flaw, depending on the bug).

The limits engineers must know: determinism is mandatory (no randomness, no clock, no network calls — every node must compute identically), so any external data requires an oracle (a trusted feed — reintroducing exactly the trust the chain removed, and the source of a large share of real-world exploits); immutability cuts both ways (a bug is permanent unless an upgrade proxy was designed in, and upgrade proxies reintroduce an admin key); everything is public (contract state and transactions are visible — privacy requires additional cryptography); gas costs make computation and storage expensive enough to shape algorithms (loops over unbounded arrays are a denial-of-service vector); and the security bar is brutal — reentrancy, integer issues, front-running (MEV), and access-control mistakes have cost billions, because the code holds the money directly and is adversarially exercised by anyone.

5. Honest assessment: where it fits

Where the trade-off genuinely pays — all cases where no trusted operator is available or acceptable: cryptocurrencies (the original, and the clearest fit — a bearer asset with no issuer to trust); cross-border value transfer where correspondent banking is slow and expensive; assets on public chains where custody and provenance matter and the parties are strangers; decentralized finance for composable, permissionless financial primitives (with the honest note that much of it re-adds trust via oracles and governance keys); and multi-party consortiums with genuine mutual distrust — several banks or logistics firms sharing a ledger none of them will let another operate.

Where it doesn't — and this is the majority of proposals: any application where you already have a trusted operator (which is nearly every business system), where throughput or latency matter, where data must be private or deletable (GDPR's right to erasure is fundamentally at odds with an immutable public ledger — a legal reality, not an engineering preference), or where the real requirement is auditability, which an append-only log with signed entries and third-party attestation delivers at a millionth of the cost (10.8.4's event sourcing plus signatures covers "prove this wasn't tampered with" without a chain). The blunt test: "who is the untrusted party, and why can't we trust them?" If there isn't a specific answer, a database is better on every axis.

What generalizes to your work regardless — and this is the section to remember: Merkle trees (efficient verification of large datasets — used in Git, Cassandra's anti-entropy 10.5, and certificate transparency); content addressing (identify data by its hash — Git, container image digests, 9.9.7's immutable assets); cryptographic signatures as authorization (Part 8.2/8.4); append-only ledgers as a modeling discipline (9.7.29); and BFT thinking (designing for participants who may be adversarial, not merely failed — increasingly relevant in multi-tenant and partner-facing systems).

6. The expert lens

Blockchain is a governance technology wearing a database costume. Its distinguishing property is not technical performance — it's who decides. That reframing resolves most debates instantly: if your organization can legitimately decide (you own the data and the rules), a database is superior in every measurable dimension; if no single party may decide, you're in blockchain's problem domain and the costs may be justified. Engineers who evaluate it on throughput miss the point; engineers who ignore the throughput cost get burned.

Most "blockchain" enterprise projects were auditability projects. The requirement was usually "prove records weren't altered" or "share data between organizations without one owning it," and the first is solved by hash-chained append-only logs with external attestation (10.8.4), the second often by a well-governed shared service or signed data exchange. Being able to make that distinction — and to say it kindly to an enthusiastic stakeholder — is genuine senior value, and it is exactly the 10.1 question ("what limit are we hitting?") applied to a hyped technology.

Learn it for the ideas, evaluate it on the trade. The cryptographic and consensus concepts here are permanently useful and appear across systems that have nothing to do with cryptocurrency. The architecture is appropriate rarely and expensively. Holding both facts simultaneously — genuine intellectual respect, ruthless applicability judgment — is the professional stance, and it's the same one this book takes toward every technology.

Next: 10.14.1 — caching, which is the single largest performance lever in most systems and the source of some of the most interesting failures in this Part.

Recall

• A blockchain = replicated append-only log + hash chaining (tamper evidence; Merkle trees for logarithmic inclusion proofs) + signatures (cryptographic authorization) + BFT consensus. The novel property: mutually distrusting parties with no administrator agreeing on an ordered history — not "distributed database," not "immutable log."
• BFT tolerates lying nodes (3f+1 for f malicious, vs 2f+1 crash-only in Raft). Nakamoto consensus solved open membership via proof of work (costly to produce, cheap to verify; longest-work chain wins) with probabilistic finality ("6 confirmations"); proof of stake substitutes bonded capital + slashing, ~99.9% less energy, with finality gadgets approaching deterministic finality.
• The trade in numbers: ~15–7,000 TPS, seconds-to-minutes finality, cents-to-dollars per write, every node storing everything — versus 10⁴–10⁶ TPS, millisecond latency, ~free writes. You buy exactly one thing: no trusted operator. Layer 2 (rollups), sharding, and app-chains improve throughput while adding complexity and often re-adding trust.
• Smart contracts are deterministic on-chain code: no randomness/clock/network ⇒ external data needs an oracle (re-adding trust, a major exploit source); immutability makes bugs permanent (upgrade proxies re-add admin keys); state is public; gas shapes algorithms; and the adversarial security bar (reentrancy, MEV/front-running, access control) is brutal.
• Fits: cryptocurrencies, cross-border value transfer, public-chain assets, DeFi primitives, genuinely mutually-distrusting consortiums. Doesn't fit: anything with a legitimate trusted operator, throughput/latency-sensitive workloads, private or deletable data (GDPR erasure vs immutability), or requirements that are really auditability (hash-chained signed logs + attestation, at ~a millionth of the cost). Test: "who is the untrusted party, and why can't we trust them?"
• Transferable ideas: Merkle trees (Git, anti-entropy, certificate transparency), content addressing, signatures as authorization, append-only ledgers, and BFT thinking for adversarial participants.

Self-test: Name the three additions over a plain replicated log and the one novel property. Why 3f+1 rather than 2f+1? State the trade in orders of magnitude. Why can't a smart contract call an API, and what does the workaround cost? Give the one-question test for whether blockchain fits.

Quiz Bank

FoundationalWhat is a blockchain mechanically, and what property genuinely distinguishes it?

Mechanically: a replicated append-only log where (1) each block includes the hash of its predecessor, so any modification to history invalidates every later block — tamper evidence detectable by anyone (and within a block, a Merkle tree lets you prove a transaction's inclusion with a logarithmic-size proof); (2) transactions are digitally signed, so authorization comes from key possession rather than from an administrator's decision; and (3) agreement on the log's contents is reached by Byzantine fault tolerant consensus among parties who may be adversarial. The genuinely distinguishing property is the third one, and it is not "distributed database" (decades old), "immutable log" (also old — 10.8.4's event sourcing), or "encrypted" (mostly it isn't — chains are public).

It is: mutually distrusting parties, with no shared administrator and open membership, agreeing on a single ordered history. Everything expensive about blockchains — low throughput, high latency, full replication of all data to all nodes, energy or capital costs — is the price of that property, which is why the only sound way to evaluate the technology is to ask whether that specific property is required.

FoundationalHow does Byzantine fault tolerance differ from crash fault tolerance, and what did Nakamoto consensus add?

Crash fault tolerance (Raft, Paxos — 10.7.2) assumes failed nodes simply stop or lag; responding nodes are honest, so 2f+1 nodes tolerate f failures via majority. Byzantine fault tolerance assumes nodes may behave arbitrarily — sending contradictory messages to different peers, forging, colluding — which requires 3f+1 nodes to tolerate f malicious ones, and classical BFT protocols (PBFT lineage) additionally require a known, fixed participant set.

Nakamoto consensus (Bitcoin) solved the case classical BFT couldn't: open membership, where anyone may participate anonymously and Sybil attacks (creating many identities) are otherwise free. It does so by making participation costly — proof of work requires finding a nonce meeting a difficulty target, expensive to produce and trivial to verify — and by making agreement probabilistic: the chain with the most cumulative work wins, so reversing a block requires out-computing the honest network, and confidence grows exponentially with each subsequent block (hence "wait for N confirmations" rather than Raft's deterministic commit). Proof of stake achieves similar Sybil resistance with bonded capital and slashing rather than energy, cutting energy use by ~99.9%, and modern implementations add finality gadgets that restore deterministic finality after a bounded number of epochs — converging back toward classical BFT guarantees while keeping open participation.

AppliedState the blockchain trade-off in concrete numbers and explain what you actually buy.

Numbers: public layer-1 chains process roughly 15 TPS (Bitcoin) to a few thousand TPS (high-throughput chains), with finality from seconds to tens of minutes, per-write costs from cents to dollars, and full replication — every node stores the entire history. A conventional distributed database handles 10⁴–10⁶ TPS with millisecond latency, effectively free writes, and partitioned storage (10.6). That's three to five orders of magnitude worse on throughput, latency, and cost.

What you buy for it: the removal of a trusted operator. No administrator can alter history, censor a transaction, freeze an account, or be compelled to; authorization is cryptographic; and any participant can independently verify the entire history. That is a real and sometimes decisive property — but it is singular, and it is not "security," "reliability," or "auditability," each of which conventional systems provide more cheaply.

Partial mitigations: layer-2 rollups batch many transactions into a single L1 commitment (optimistic rollups with fraud proofs; ZK rollups with validity proofs), raising throughput by orders of magnitude at the cost of complexity and, frequently, new trust assumptions (sequencer centralization, upgrade keys); sharding and app-specific chains trade global composability for capacity. The evaluation question stays the same at every layer: is trustlessness worth this, here?

InterviewA stakeholder proposes blockchain for supply-chain traceability across your company and three partners. How do you evaluate it?

Start with the test: who is the untrusted party, and why can't we trust them? In most supply-chain proposals the honest answer is that the partners do trust each other enough to trade daily, but nobody wants a competitor operating the shared system — which is a governance problem, and worth taking seriously. So evaluate three options against the actual requirements.

(1) A shared service with strong governance — hosted neutrally (a consortium entity or a cloud tenancy with contractual controls), with an append-only, hash-chained, signed event log (10.8.4) and periodic third-party attestation. This gives tamper evidence and auditability at ~a millionth of the cost, with normal throughput, private data, and the ability to delete when law requires.

(2) Signed data exchange — each party keeps its own records and signs the events it publishes; verification is cryptographic without any shared ledger; this fits when parties mainly need to prove their own claims. (3) A permissioned blockchain — appropriate only if there is genuine, articulable mutual distrust and no acceptable neutral operator; note honestly that permissioned chains are BFT protocols with known members — closer to a jointly-operated database than to Bitcoin — and that the hard problems (data quality at entry, partner onboarding, governance of schema changes) remain untouched by the technology.

The decisive engineering point to raise regardless of choice: a ledger guarantees that what was recorded wasn't altered; it says nothing about whether what was recorded was true. Supply-chain fraud is overwhelmingly a data-entry and physical-verification problem (the "oracle problem" — section 4), so the project's value depends on sensors, audits, and process — not on the ledger technology. Deliver that assessment first; it is usually the observation that reframes the whole initiative.

StaffWhich ideas from this space should a backend engineer with no crypto interest actually learn, and why?

Five, all with immediate non-crypto applications. (1) Merkle trees — hash trees enabling efficient verification and diffing of large datasets: Git's object model is one, Cassandra's anti-entropy repair compares replicas with them (10.5), certificate transparency logs use them, and any system that must prove "this dataset is unchanged" or "these two replicas differ here" is reaching for the same structure.

(2) Content addressing — identifying data by its hash rather than a location: Git commits, container image digests, and immutable asset URLs (9.9.7) all gain deduplication, integrity checking, and cache-forever semantics from it.

(3) Signatures as authorization — moving from "the server decides who you are" to "you present a cryptographic proof": JWTs, webhook signature verification, and mutual TLS are the everyday forms (Part 8.2/8.4), and the mental model — authorization as verifiable claim rather than as lookup — is transferable.

(4) Append-only ledgers as a modeling discipline — immutable events with derived state (9.7.29, 10.8.4), which gives auditability, replay, and correction-by-compensation in ordinary systems; blockchains merely apply this at maximum rigor.

(5) Byzantine thinking — designing for participants who may be adversarial rather than merely faulty; increasingly relevant outside crypto in multi-tenant platforms, partner integrations, and anything where a client can lie (9.9.2's "everything from the wire can lie").

What not to spend time on unless the domain demands it: tokenomics, specific chains' APIs, and DeFi mechanics. The professional stance to hold: learn the primitives because they're permanently useful; evaluate the architecture on its trade, which rarely favors it.

Flashcards

FlashBlockchain = ?

Replicated append-only log + hash chaining (tamper evidence, Merkle proofs) + signatures + BFT consensus. Novel property: mutually distrusting, open-membership parties agreeing on ordered history.

FlashBFT vs crash faults

Crash: nodes stop (2f+1, Raft). Byzantine: nodes lie (3f+1, known members). Nakamoto adds open membership via costly participation + probabilistic finality.

FlashPoW vs PoS

PoW: energy-costly nonce search, longest-work chain, probabilistic finality. PoS: bonded capital + slashing, ~99.9% less energy, finality gadgets approach deterministic.

FlashThe trade

3–5 orders of magnitude worse throughput/latency/cost; every node stores everything. Bought: no trusted operator. That's the entire purchase.

FlashSmart contract limits

Deterministic only (no clock/random/network ⇒ oracles re-add trust); immutable bugs (upgrade proxies re-add admin keys); public state; gas shapes algorithms; adversarial security bar.

FlashThe one-question test

"Who is the untrusted party, and why can't we trust them?" No specific answer ⇒ a database wins on every axis. Most enterprise proposals are auditability projects.

Scenario Drill

DrillA fintech CTO asks you to assess three proposals: (a) settle inter-bank payments on a public chain, (b) store customer KYC documents on-chain for auditability, (c) issue loyalty points as tokens on a public chain. Evaluate each with the trade-off framework, and say what you'd build instead where the answer is no.

(a) Inter-bank settlement — the strongest case, with caveats. The untrusted-party question has a real answer: banks in different jurisdictions with no common operator, where correspondent banking is slow (days), expensive, and opaque. Trustless settlement genuinely addresses that.

But evaluate honestly: public-chain throughput and finality (seconds to minutes) are acceptable for settlement (unlike retail payments), yet volatility, on/off-ramp custody, regulatory reporting, and privacy (competitors seeing your flows on a public ledger) are decisive obstacles — which is why real deployments use permissioned chains or tokenized deposits rather than public L1s, and why the interesting question is jurisdictional and legal rather than technical. Verdict: plausible, but the design constraint is regulation and privacy, and the honest architecture is likely a permissioned ledger or a regulated stablecoin rail, not a public chain. (b) KYC documents on-chain — a clear no, with a legal reason that ends the discussion. Blockchains are immutable and (on public chains) fully visible; KYC documents are personal data subject to erasure rights (GDPR Article 17 and equivalents) and strict confidentiality. Immutability and the right to be forgotten are fundamentally incompatible — you cannot delete from an immutable ledger, and "we'll store only hashes" means the ledger holds no documents, at which point it is doing nothing a signed audit log couldn't.

Build instead: documents in encrypted object storage with strict access control; an append-only, hash-chained audit log of access and verification events, signed and periodically attested by a third party (10.8.4) — which delivers "prove these records weren't altered and show who touched them" at negligible cost, while remaining deletable and private. This is the standard case of an auditability requirement wearing blockchain clothing (section 6).

(c) Loyalty points as tokens — a no on the trade, and the reasoning generalizes. You are the trusted operator: you issue the points, you honor them, you can revoke them for fraud, and customers already trust you enough to hold their money and data. Tokenizing adds per-transaction cost and latency to what is currently a database increment, exposes your liability publicly, creates a secondary market you must then have a legal position on, and — the killer — makes fraud reversal (a routine, necessary operation) either impossible or dependent on an admin key that re-centralizes everything anyway.

Build instead: a conventional ledger-modeled points system (9.7.29's append-only entries with derived balances), which gives auditability, correction-by-compensation, and instant free transactions. The only variant worth revisiting is if the strategic goal is interoperability with other companies' programs — that's a genuine multi-party trust question, and it would reopen (a)'s consortium conversation rather than justifying a public token.

The pattern across all three: ask who the untrusted party is; check whether the requirement is really auditability, interoperability, or governance; and price the trade in orders of magnitude before discussing technology. Two of three proposals dissolve into cheaper, better systems — and saying so clearly, with alternatives attached, is the value a senior engineer adds to a hype cycle.