10.13 — Blockchain & Web3: An Honest Engineering Tour

Blockchain made distributed-systems ideas famous, and also caused more confusion about them than anything else. Here it is from an engineer's point of view:

• what a blockchain is, mechanically;
• what its kind of consensus buys that the consensus of 10.7.2 does not;
• the real trade-offs, in numbers;
• smart contracts, and what they can and cannot do;
• and the part that matters most for your career, an honest assessment of where it fits and where a plain database is simply better.

^{⚑What is Web3 and blockchain, honestly assessed? [EQ-149b]}

1. What a blockchain actually is

Mechanically, a blockchain is a replicated append-only log (10.8.1's structure) with three additions:

1. Hash chaining — each block contains the cryptographic hash of the previous block, so altering an old block changes its hash, invalidating every subsequent block. This gives tamper evidence: modification is detectable by anyone holding the chain. (Inside a block, transactions are summarized by a Merkle tree so a single transaction's inclusion can be proven with a logarithmic-size proof rather than the whole block — the same structure used for anti-entropy in 10.5.)
2. Digital signatures — each transaction is signed by the holder of a private key (Part 8.2), so authorization is cryptographic rather than administrative: no server decides whether you may spend; your key does.
3. Byzantine fault tolerant consensus — the new part, and the reason blockchains exist.

The genuinely novel property is not "distributed database" (we've had those for decades) and not "immutable log" (an append-only log with hashes is old). It is: a set of mutually distrusting parties, with no shared administrator, agreeing on an ordered history. That is the problem Bitcoin solved, and it's worth stating precisely because it's the only property that justifies the costs.

2. Byzantine consensus: what it adds over Raft

10.7.2's Raft tolerates crash faults: nodes may stop or be slow, but the ones that respond are assumed honest. Byzantine fault tolerance (BFT) tolerates nodes that lie — sending different messages to different peers, forging, or colluding. The classical result (Lamport's Byzantine Generals) is that BFT consensus needs 3f+1 nodes to tolerate f malicious ones (versus 2f+1 for crash faults), and classical BFT protocols (PBFT and descendants) require knowing the participant set in advance.

Nakamoto consensus (Bitcoin, 2008) solved the open-membership case — anyone may join, nobody is identified — by making participation costly and agreement probabilistic:

• Proof of work — miners race to find a nonce whose block hash meets a difficulty target. Finding it is expensive (energy); verifying it is trivial. The longest (most-work) chain wins, so rewriting history requires redoing all that work faster than the honest network — the famous "51% attack" threshold.
• Probabilistic finality — a block is never certain, only exponentially unlikely to be reversed as more blocks build on it (hence "wait for 6 confirmations"). This is a genuinely different guarantee from Raft's deterministic commit, and it's the price of open membership.

Proof of stake (Ethereum since 2022) replaces energy with capital at risk: validators bond tokens, are selected to propose/attest, and are slashed (lose stake) for provable misbehavior. It achieves comparable security assumptions at ~99.9% less energy, and modern designs add finality gadgets that give deterministic finality after a couple of epochs — moving closer to classical BFT guarantees while keeping open participation.

3. The trade-offs, in numbers

The honest comparison, which the field's marketing usually omits:

Property	Blockchain (public L1)	Conventional distributed DB
Throughput	~15–7,000 TPS (chain-dependent)	10⁴–10⁶ TPS
Latency to finality	seconds to minutes	milliseconds
Cost per write	cents to dollars (gas)	~free
Storage	every node stores everything	partitioned
Trust model	no trusted party required	operator is trusted
Governance	protocol rules + social consensus	whoever has admin rights

The trade is stark and singular: you sacrifice three to five orders of magnitude of throughput, latency, and cost — in exchange for removing the need to trust an operator. That's it. Every architectural decision should follow from asking whether trustlessness is worth those orders of magnitude in your specific case. The scaling responses are real but partial: layer 2 (rollups batching thousands of transactions into one L1 commitment — optimistic with fraud proofs, or ZK with validity proofs), sharding, and app-specific chains all improve throughput while adding complexity and, often, reintroducing trusted components.

4. Smart contracts and their limits

A smart contract is code deployed to the chain that executes deterministically on every node as transactions invoke it — programmable, self-enforcing logic without an operator. Genuine capabilities: escrow and conditional payment without an intermediary, token issuance and transfer rules, automated market makers, and auditable, unchangeable rules (which is either the feature or the flaw, depending on the bug).

The limits engineers must know:

• Every node must get exactly the same result. So there is no randomness, no clock and no network calls. Any outside data needs an oracle, a trusted feed. But that brings back exactly the trust the chain was meant to remove, and oracles are behind a large share of real-world hacks.
• Code that cannot change cuts both ways. A bug is permanent, unless an upgrade mechanism was built in from the start. And an upgrade mechanism brings back an admin key.
• Everything is public. Contract data and transactions are visible to all. Privacy needs extra cryptography.
• Gas costs make computing and storage expensive enough to change how you write algorithms. A loop over an array with no size limit lets an attacker block the contract.
• The security bar is brutal. Re-entry bugs, number overflows, front-running (MEV) and permission mistakes have cost billions. The code holds the money directly, and anyone in the world can attack it.

5. Honest assessment: where it fits

Where the trade-off really pays. In every case, there is no one everybody can trust to run it:

• Cryptocurrencies. The original use, and the clearest fit: something of value that nobody issues, so there is nobody to trust.
• Sending money across borders, where the chain of banks in between is slow and expensive.
• Assets on public chains, where who holds what and where it came from matter, and the people trading are strangers.
• Decentralised finance, financial building blocks that anyone can use and combine. To be honest, much of it brings trust back through oracles and admin keys.
• Groups that truly distrust each other, such as several banks or shipping firms sharing a ledger that none of them will let another one run.

Where it does not pay, which is most proposals:

• You already have someone trusted to run it. That is nearly every business system.
• Speed or volume matters.
• Data must be private, or must be deletable. The GDPR right to be forgotten clashes head-on with a public ledger that can never change. That is a legal fact, not an engineering preference.
• What you really need is an audit trail. An append-only log with signed entries, checked by an outside party, gives you that at a tiny fraction of the cost. Event sourcing plus signatures (10.8.4) proves "nobody tampered with this" without any chain.

The blunt test: "who is the untrusted party, and why can't we trust them?" If there isn't a specific answer, a database is better on every axis.

What carries over to your work anyway. This is the part to remember:

• Merkle trees, which check huge amounts of data cheaply. Git uses them, Cassandra uses them to repair copies (10.5), and so does certificate transparency.
• Content addressing, naming data by its hash. Git does it, container image digests do it, and so do the never-changing build files of 9.9.7.
• Digital signatures as permission (Part 8.2 and 8.4).
• Append-only ledgers as a way to model data (9.7.29).
• Byzantine thinking: designing for participants who may lie on purpose, not just crash. That matters more and more in systems shared by many customers or partners.

6. The expert lens

Blockchain is a governance technology wearing a database costume. Its distinguishing property is not technical performance — it's who decides. That reframing resolves most debates instantly: if your organization can legitimately decide (you own the data and the rules), a database is superior in every measurable dimension; if no single party may decide, you're in blockchain's problem domain and the costs may be justified. Engineers who evaluate it on throughput miss the point; engineers who ignore the throughput cost get burned.

Most "blockchain" enterprise projects were auditability projects. The requirement was usually "prove records weren't altered" or "share data between organizations without one owning it," and the first is solved by hash-chained append-only logs with external attestation (10.8.4), the second often by a well-governed shared service or signed data exchange. Being able to make that distinction — and to say it kindly to an enthusiastic stakeholder — is genuine senior value, and it is exactly the 10.1 question ("what limit are we hitting?") applied to a hyped technology.

Learn it for the ideas, evaluate it on the trade. The cryptographic and consensus concepts here are permanently useful and appear across systems that have nothing to do with cryptocurrency. The architecture is appropriate rarely and expensively. Holding both facts simultaneously — genuine intellectual respect, ruthless applicability judgment — is the professional stance, and it's the same one this book takes toward every technology.

Next: 10.14.1 — caching, which is the single largest performance lever in most systems and the source of some of the most interesting failures in this Part.

Recall

• A blockchain = replicated append-only log + hash chaining (tamper evidence; Merkle trees for logarithmic inclusion proofs) + signatures (cryptographic authorization) + BFT consensus. The novel property: mutually distrusting parties with no administrator agreeing on an ordered history — not "distributed database," not "immutable log."
• BFT tolerates lying nodes (3f+1 for f malicious, vs 2f+1 crash-only in Raft). Nakamoto consensus solved open membership via proof of work (costly to produce, cheap to verify; longest-work chain wins) with probabilistic finality ("6 confirmations"); proof of stake substitutes bonded capital + slashing, ~99.9% less energy, with finality gadgets approaching deterministic finality.
• The trade in numbers: ~15–7,000 TPS, seconds-to-minutes finality, cents-to-dollars per write, every node storing everything — versus 10⁴–10⁶ TPS, millisecond latency, ~free writes. You buy exactly one thing: no trusted operator. Layer 2 (rollups), sharding, and app-chains improve throughput while adding complexity and often re-adding trust.
• Smart contracts are code on the chain that must give the same result on every node:
  • No randomness, clock or network, so outside data needs an oracle. That brings trust back, and it is a major source of hacks.
  • Bugs are permanent. Upgrade mechanisms bring admin keys back.
  • All data is public.
  • Gas costs shape the algorithms.
  • The security bar is brutal: re-entry bugs, front-running (MEV) and permission mistakes.
• Fits: cryptocurrencies, cross-border value transfer, public-chain assets, DeFi primitives, genuinely mutually-distrusting consortiums. Doesn't fit: anything with a legitimate trusted operator, throughput/latency-sensitive workloads, private or deletable data (GDPR erasure vs immutability), or requirements that are really auditability (hash-chained signed logs + attestation, at ~a millionth of the cost). Test: "who is the untrusted party, and why can't we trust them?"
• Transferable ideas: Merkle trees (Git, anti-entropy, certificate transparency), content addressing, signatures as authorization, append-only ledgers, and BFT thinking for adversarial participants.

Self-test: Name the three additions over a plain replicated log and the one novel property. Why 3f+1 rather than 2f+1? State the trade in orders of magnitude. Why can't a smart contract call an API, and what does the workaround cost? Give the one-question test for whether blockchain fits.

Quiz Bank

FoundationalWhat is a blockchain mechanically, and what property genuinely distinguishes it?

Mechanically, it is a log that is only ever added to, with a copy on many machines. Three things make it special:

1. Each block contains the hash of the block before it. So changing anything in the past breaks every later block, and anyone can see the tampering. Inside a block, a Merkle tree lets you prove a transaction is there with a very small proof.
2. Transactions are digitally signed. Permission comes from holding the key, not from an administrator deciding.
3. The machines agree on what is in the log using Byzantine fault tolerant consensus, which works even when some participants are lying on purpose.

The genuinely distinguishing property is the third one, and it is not "distributed database" (decades old), "immutable log" (also old — 10.8.4's event sourcing), or "encrypted" (mostly it isn't — chains are public).

It is: mutually distrusting parties, with no shared administrator and open membership, agreeing on a single ordered history. Everything expensive about blockchains — low throughput, high latency, full replication of all data to all nodes, energy or capital costs — is the price of that property, which is why the only sound way to evaluate the technology is to ask whether that specific property is required.

FoundationalHow does Byzantine fault tolerance differ from crash fault tolerance, and what did Nakamoto consensus add?

Crash fault tolerance (Raft, Paxos — 10.7.2) assumes failed nodes simply stop or lag; responding nodes are honest, so 2f+1 nodes tolerate f failures via majority. Byzantine fault tolerance assumes nodes may behave arbitrarily — sending contradictory messages to different peers, forging, colluding — which requires 3f+1 nodes to tolerate f malicious ones, and classical BFT protocols (PBFT lineage) additionally require a known, fixed participant set.

Nakamoto consensus (Bitcoin) solved the case classical BFT couldn't: open membership, where anyone may participate anonymously and Sybil attacks (creating many identities) are otherwise free. It does two things.

It makes taking part expensive. Proof of work means finding a number, the nonce, that makes the block's hash meet a difficulty target. Finding it takes enormous computing. Checking it takes a moment.

It makes agreement a matter of probability. The chain with the most total work wins. To reverse a block, an attacker must out-compute everyone honest. And confidence grows very fast with each block added after it. That is why you "wait for N confirmations", instead of getting a definite commit as in Raft.

Proof of stake gets a similar protection against fake identities by making participants lock up money, which they lose if they cheat, instead of burning energy. That cuts energy use by about 99.9%. Modern versions add a step that makes a block definitely final after a set number of rounds. That brings them back close to the guarantees of classic Byzantine fault tolerance, while still letting anyone join.

AppliedState the blockchain trade-off in concrete numbers and explain what you actually buy.

Numbers: public layer-1 chains process roughly 15 TPS (Bitcoin) to a few thousand TPS (high-throughput chains), with finality from seconds to tens of minutes, per-write costs from cents to dollars, and full replication — every node stores the entire history. A conventional distributed database handles 10⁴–10⁶ TPS with millisecond latency, effectively free writes, and partitioned storage (10.6). That's three to five orders of magnitude worse on throughput, latency, and cost.

What you buy for it: the removal of a trusted operator. No administrator can alter history, censor a transaction, freeze an account, or be compelled to; authorization is cryptographic; and any participant can independently verify the entire history. That is a real and sometimes decisive property — but it is singular, and it is not "security," "reliability," or "auditability," each of which conventional systems provide more cheaply.

Partial fixes:

• Layer-2 rollups bundle many transactions into one entry on the main chain. Optimistic rollups rely on anyone being able to prove fraud. ZK rollups attach a mathematical proof that every transaction was valid. Throughput rises by orders of magnitude. The cost is complexity, and often new things you have to trust, such as one company ordering the transactions, or an upgrade key.
• Sharding and chains for one application buy capacity, but programs on different chains can no longer work together as easily.

The evaluation question stays the same at every layer: is trustlessness worth this, here?

InterviewA stakeholder proposes blockchain for supply-chain traceability across your company and three partners. How do you evaluate it?

Start with the test: who is the untrusted party, and why can't we trust them? In most supply-chain proposals the honest answer is that the partners do trust each other enough to trade daily, but nobody wants a competitor operating the shared system — which is a governance problem, and worth taking seriously. So evaluate three options against the actual requirements.

(1) A shared service with strong governance — hosted neutrally (a consortium entity or a cloud tenancy with contractual controls), with an append-only, hash-chained, signed event log (10.8.4) and periodic third-party attestation. This gives tamper evidence and auditability at ~a millionth of the cost, with normal throughput, private data, and the ability to delete when law requires.

(2) Signed data exchange — each party keeps its own records and signs the events it publishes; verification is cryptographic without any shared ledger; this fits when parties mainly need to prove their own claims. (3) A permissioned blockchain. It fits only if the parties really do distrust each other, for reasons you can state, and no neutral operator is acceptable to all of them. Be honest about two things. A permissioned chain is a Byzantine fault tolerant protocol between known members, so it is closer to a database run jointly than to Bitcoin. And the hard problems remain untouched by the technology: the quality of data when it is entered, bringing partners on board, and agreeing on changes to the data format.

The decisive engineering point to raise regardless of choice: a ledger guarantees that what was recorded wasn't altered; it says nothing about whether what was recorded was true. Supply-chain fraud is overwhelmingly a data-entry and physical-verification problem (the "oracle problem" — section 4), so the project's value depends on sensors, audits, and process — not on the ledger technology. Deliver that assessment first; it is usually the observation that reframes the whole initiative.

StaffWhich ideas from this space should a backend engineer with no crypto interest actually learn, and why?

Five, all with immediate non-crypto applications. (1) Merkle trees — hash trees enabling efficient verification and diffing of large datasets: Git's object model is one, Cassandra's anti-entropy repair compares replicas with them (10.5), certificate transparency logs use them, and any system that must prove "this dataset is unchanged" or "these two replicas differ here" is reaching for the same structure.

(2) Content addressing — identifying data by its hash rather than a location: Git commits, container image digests, and immutable asset URLs (9.9.7) all gain deduplication, integrity checking, and cache-forever semantics from it.

(3) Signatures as authorization — moving from "the server decides who you are" to "you present a cryptographic proof": JWTs, webhook signature verification, and mutual TLS are the everyday forms (Part 8.2/8.4), and the mental model — authorization as verifiable claim rather than as lookup — is transferable.

(4) Append-only ledgers as a modeling discipline — immutable events with derived state (9.7.29, 10.8.4), which gives auditability, replay, and correction-by-compensation in ordinary systems; blockchains merely apply this at maximum rigor.

(5) Byzantine thinking — designing for participants who may be adversarial rather than merely faulty; increasingly relevant outside crypto in multi-tenant platforms, partner integrations, and anything where a client can lie (9.9.2's "everything from the wire can lie").

What not to spend time on unless the domain demands it: tokenomics, specific chains' APIs, and DeFi mechanics. The professional stance to hold: learn the primitives because they're permanently useful; evaluate the architecture on its trade, which rarely favors it.

Flashcards

FlashBlockchain = ?

Replicated append-only log + hash chaining (tamper evidence, Merkle proofs) + signatures + BFT consensus. Novel property: mutually distrusting, open-membership parties agreeing on ordered history.

FlashBFT vs crash faults

Crash: nodes stop (2f+1, Raft). Byzantine: nodes lie (3f+1, known members). Nakamoto adds open membership via costly participation + probabilistic finality.

FlashPoW vs PoS

PoW: energy-costly nonce search, longest-work chain, probabilistic finality. PoS: bonded capital + slashing, ~99.9% less energy, finality gadgets approach deterministic.

FlashThe trade

3–5 orders of magnitude worse throughput/latency/cost; every node stores everything. Bought: no trusted operator. That's the entire purchase.

FlashSmart contract limits

Deterministic only (no clock/random/network ⇒ oracles re-add trust); immutable bugs (upgrade proxies re-add admin keys); public state; gas shapes algorithms; adversarial security bar.

FlashThe one-question test

"Who is the untrusted party, and why can't we trust them?" No specific answer ⇒ a database wins on every axis. Most enterprise proposals are auditability projects.

Scenario Drill

DrillA fintech CTO asks you to assess three proposals: (a) settle inter-bank payments on a public chain, (b) store customer KYC documents on-chain for auditability, (c) issue loyalty points as tokens on a public chain. Evaluate each with the trade-off framework, and say what you'd build instead where the answer is no.

(a) Inter-bank settlement — the strongest case, with caveats. The untrusted-party question has a real answer: banks in different jurisdictions with no common operator, where correspondent banking is slow (days), expensive, and opaque. Trustless settlement genuinely addresses that.

But judge it honestly. For settling between banks, a public chain's speed and time to finality, seconds to minutes, are acceptable, unlike for shop payments. Yet four things block it: prices that swing, who holds the money when converting in and out, regulatory reporting, and privacy, because competitors could see your payments on a public ledger. That is why real deployments use permissioned chains or token versions of bank deposits, not public chains. And it is why the interesting questions are legal ones, not technical ones. Verdict: plausible, but the design constraint is regulation and privacy, and the honest architecture is likely a permissioned ledger or a regulated stablecoin rail, not a public chain. (b) KYC documents on-chain — a clear no, with a legal reason that ends the discussion. Blockchains are immutable and (on public chains) fully visible; KYC documents are personal data subject to erasure rights (GDPR Article 17 and equivalents) and strict confidentiality. Immutability and the right to be forgotten are fundamentally incompatible — you cannot delete from an immutable ledger, and "we'll store only hashes" means the ledger holds no documents, at which point it is doing nothing a signed audit log couldn't.

Build instead: documents in encrypted object storage with strict access control; an append-only, hash-chained audit log of access and verification events, signed and periodically attested by a third party (10.8.4) — which delivers "prove these records weren't altered and show who touched them" at negligible cost, while remaining deletable and private. This is the standard case of an auditability requirement wearing blockchain clothing (section 6).

(c) Loyalty points as tokens — a no on the trade, and the reasoning generalizes. You are the trusted operator: you issue the points, you honor them, you can revoke them for fraud, and customers already trust you enough to hold their money and data. Tokenizing adds per-transaction cost and latency to what is currently a database increment, exposes your liability publicly, creates a secondary market you must then have a legal position on, and — the killer — makes fraud reversal (a routine, necessary operation) either impossible or dependent on an admin key that re-centralizes everything anyway.

Build instead: a conventional ledger-modeled points system (9.7.29's append-only entries with derived balances), which gives auditability, correction-by-compensation, and instant free transactions. The only variant worth revisiting is if the strategic goal is interoperability with other companies' programs — that's a genuine multi-party trust question, and it would reopen (a)'s consortium conversation rather than justifying a public token.

The pattern across all three: ask who the untrusted party is; check whether the requirement is really auditability, interoperability, or governance; and price the trade in orders of magnitude before discussing technology. Two of three proposals dissolve into cheaper, better systems — and saying so clearly, with alternatives attached, is the value a senior engineer adds to a hype cycle.