Appearance
7.1 — Relational Theory and the Dialect Map
A team keeps orders in one spreadsheet. Each row holds the order, and — because it is convenient — the customer's name, email and delivery address too.
Then a customer moves house. Someone updates the address on their most recent order. The other 40 rows still carry the old address. Three months later a refund is posted to the old address, and nobody can say which row is the truth.
That is an update anomaly, and the relational model exists to make it impossible rather than to make it unlikely. Every rule in this chapter is a defence against one of three failures: the same fact stored twice and drifting apart, a fact that cannot be recorded because nothing else exists to hang it on, and a fact that disappears when an unrelated row is deleted.
1. What a relation actually is
Edgar Codd's 1970 paper A Relational Model of Data for Large Shared Data Banks proposed something that sounds unremarkable now and was radical then: store data as plain values in tables, and let the machine work out how to fetch it. Before that, a program navigated pointers between records by hand, and changing the storage layout meant rewriting the programs.
A relation is a set of tuples, all with the same named attributes. In table language: a table is a set of rows, and every row has the same columns.
Three consequences fall straight out of the word set, and they surprise people who think of a table as a file:
Rows have no order. A table is not sorted. If you run SELECT * FROM orders twice and get different orderings, the database is not broken — you did not ask for an order. The only thing that guarantees row order is ORDER BY. This is the single most common source of a test that passes for a year and then fails after an upgrade.
Columns have no order either, in theory. SQL gives them a position, which is why SELECT * and INSERT without a column list are fragile: add a column and both change meaning.
There are no duplicate rows in a relation. SQL breaks this one deliberately: a SQL table is a bag (a multiset), so duplicates are allowed. That divergence is the reason SELECT DISTINCT exists at all, and the reason UNION removes duplicates while UNION ALL — which is faster, because it does not sort or hash to find them — does not.
A value is atomic. One cell holds one value, not a list. This is the first normal form, and it is the rule modern databases bend most: PostgreSQL arrays and jsonb columns hold structure inside a cell on purpose. Section 6 is honest about when that is right.
2. Keys: how a row is identified
A candidate key is a minimal set of columns whose values are unique across the table. Minimal matters: if (email) is unique, then (email, name) is also unique but is not a candidate key, because it contains a smaller one.
The primary key is the candidate key you nominate as the row's identity. The database enforces two things about it: unique, and not null.
A foreign key is a column whose value must exist as a primary key in another table. orders.customer_id must name a real customer. This is referential integrity, and it is the database refusing to let your application create an order for a customer who was deleted last Tuesday.
Natural versus surrogate keys is a real decision with a clear default:
| Natural key | Surrogate key | |
|---|---|---|
| Example | ISBN, email | id BIGSERIAL, UUID |
| Meaning | Carries business meaning | Meaningless by design |
| Risk | Business rules change | Needs a separate unique index on the natural key |
| Default | Rare | Usually right |
The reason surrogate keys usually win is that business identifiers change. ISBNs get reissued, email addresses get transferred, national identity numbers get corrected. When the primary key changes, every foreign key pointing at it changes too, and a mistake there is silent. A meaningless key never changes because it never meant anything. Keep the natural key as a UNIQUE constraint so the business rule is still enforced — you are choosing where identity lives, not abandoning the rule.
Chapter 7.2.4 covers the follow-up nobody thinks about until it hurts: whether that surrogate key should be a sequential integer or a UUID, and what each choice does to your indexes.
3. NULL, and the logic that trips everyone
NULL is not a value. It is a marker meaning no value here, and it turns SQL's logic from two-valued into three-valued logic: true, false, and unknown.
Every comparison with NULL produces unknown, including comparing it with itself.
sql
SELECT NULL = NULL; -- NULL (not true!)
SELECT NULL <> NULL; -- NULL
SELECT NULL IS NULL; -- true ← the only way to test it
SELECT 5 > NULL; -- NULLWHERE keeps a row only when the condition is true. Unknown is not true, so the row is dropped. That single rule explains the classic bug:
sql
-- Orders that were not cancelled.
SELECT * FROM orders WHERE status <> 'cancelled';Every order whose status is NULL is missing from that result. The author asked for "not cancelled" and got "known to be something other than cancelled". The fix is explicit:
sql
SELECT * FROM orders WHERE status IS DISTINCT FROM 'cancelled';IS DISTINCT FROM (PostgreSQL, SQL standard) compares treating NULL as an ordinary value, so a null status counts as different from 'cancelled' and the row is kept. MySQL spells the equality version <=>.
Two more consequences that cost people hours.
NOT IN with a subquery that returns any NULL returns no rows at all. x NOT IN (1, 2, NULL) expands to x <> 1 AND x <> 2 AND x <> NULL, whose last term is unknown, so the whole thing is never true. Use NOT EXISTS, which asks a different question and is immune.
Aggregates skip nulls, except COUNT(*). COUNT(email) counts rows where email is not null; COUNT(*) counts rows. AVG(score) over ten rows where four are null divides by six, not ten — which is usually what you want, and is never what you assumed.
A UNIQUE constraint allows multiple nulls, because two unknowns are not known to be equal. So a unique index on deleted_at plus email does not do what a soft-delete design usually wants; Chapter 7.2.4 shows the partial-index fix.
The practical rule: make columns NOT NULL by default, and justify each nullable one. Every nullable column is a branch in every query that touches it, and most nullable columns exist because nobody decided what the absence meant.
4. The three levels of a design: conceptual, logical, physical
Ask three people to "design the orders database" and you get three different documents, all correct, because a database design exists at three levels of detail. Knowing which level you are working at stops most schema arguments before they start, because two people arguing about varchar(255) versus text are having a physical argument, and two people arguing about whether an address belongs to a customer or to an order are having a conceptual one. Those are not the same conversation.
The easiest way in is a house. Someone sketches the house on a napkin — four bedrooms, kitchen at the back, garage on the left. That sketch says nothing about wall thickness or where the pipes run, and everybody in the family can read it. Then an architect draws a floor plan with real room shapes, doors, and dimensions. Then a builder produces the working drawings: this wall is 100 mm blockwork, this beam is steel, this cable is 2.5 mm². All three describe the same house. Each one is unusable for the job the others do.
The conceptual model names the things and the relationships between them, and nothing else. Customers, orders, products, shipments. A customer places many orders; an order contains many products; a product appears on many orders. This is the level where you argue about meaning: is a "delivery address" something a customer has, or something an order has? The answer changes the whole design, and it is a business question, not a technical one. If a customer moves house, should their old orders show the old address or the new one? Almost always the old one, which tells you the address belongs to the order. You cannot fix that mistake with an index. This is why the conceptual model is drawn first and shown to somebody who does not write SQL.
The usual drawing here is an entity-relationship diagram (an ER diagram): boxes for the things, lines between them, and a marking on each line saying how many of one goes with how many of the other. "One customer, many orders" is a one-to-many relationship. "Many orders, many products" is many-to-many, and that one is worth watching, because a many-to-many relationship always turns into an extra table at the next level down.
The logical model turns each of those things into tables, columns, keys and foreign keys, and it is where all the work of section 5 — normalization — happens. Here the many-to-many between orders and products becomes the order_lines table, holding one row per product per order, because that is the only way to record "three of SKU-9 on order 1001" in a relational database. Here you decide that orders.customer_id references customers.id, and that email is unique. Here you decide what may be null. What you do not decide here is anything a specific engine cares about: not bigint versus uuid, not whether there is an index, not how the rows are stored.
The physical model is the actual CREATE TABLE for one particular engine, plus everything around it that affects speed rather than meaning: exact data types, indexes, partitioning, storage settings, where the files live. Two teams can take the same logical model and produce very different physical models — one uses a bigint identity key on PostgreSQL with a partial index on pending rows, the other uses a UUID on MySQL with a covering secondary index — and both are implementing the same design.
Here is the same small design at all three levels, so the difference is concrete rather than abstract.
text
CONCEPTUAL
Customer ──places──< Order ──contains──< Product
(a customer has many orders; an order has many products)text
LOGICAL
customers (customer_id PK, email UNIQUE NOT NULL, name NOT NULL)
orders (order_id PK, customer_id FK -> customers, placed_at NOT NULL,
ship_address NOT NULL) ← lives on the order, not the customer
order_lines (order_id FK, sku FK -> products, qty, unit_price)
PK (order_id, sku)
products (sku PK, name NOT NULL)sql
-- PHYSICAL (PostgreSQL)
CREATE TABLE orders (
order_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, -- (1)
customer_id bigint NOT NULL REFERENCES customers(id),
placed_at timestamptz NOT NULL DEFAULT now(), -- (2)
ship_address text NOT NULL,
status text NOT NULL DEFAULT 'pending'
) PARTITION BY RANGE (placed_at); -- (3)
CREATE INDEX orders_pending_idx ON orders (placed_at)
WHERE status = 'pending'; -- (4)(1) The logical model said "a primary key". This line picks which kind — a database-generated 64-bit integer — and that choice belongs to the physical level because it changes performance and storage and changes no answer to any question. (2) timestamptz stores an instant that is unambiguous across time zones, where plain timestamp stores a wall-clock reading with no zone attached and is a reliable source of one-hour bugs twice a year. (3) Splitting the table into monthly chunks so that a query about last week never touches 2019's rows. Nothing about the meaning of an order changed. (4) An index over only the pending orders. Pure speed, zero meaning.
Everything in that CREATE TABLE can change without a single application query changing, and that is the point. The property has a name — physical data independence — and it is the thing Codd's model bought that navigating pointers by hand did not. You add the index, partition the table, move it to a faster disk, and every query keeps returning the same rows. Compare that to the pre-relational world described in section 1, where the program contained the traversal, so changing the storage meant rewriting the program.
There is a weaker sibling called logical data independence: changing the tables without changing the applications that read them. This one is only partly achievable, and the tool for it is the view, which is a stored query that applications read as though it were a table. Split one wide customers table into customers and customer_addresses, then define a view named customers that joins them back into the old shape, and the code that selects from customers never notices. Chapter 7.2.6 builds views properly, including the point at which this trick stops working.
One word of warning about the vocabulary, because it collides. In PostgreSQL, MySQL and SQL Server, the word schema also means a namespace — a folder for tables, created with CREATE SCHEMA reporting; so you can have reporting.orders and billing.orders side by side. That is a completely different thing from "the logical schema" in this section. When someone says "put it in the analytics schema" they mean the namespace. When someone says "review the logical schema" they mean the table design. Both are standard usage and the sentence around them tells you which.
Where this actually earns its keep is a migration review. A change that only touches the physical level — adding an index, changing a fill factor, partitioning — cannot break correctness, so it is reviewed for lock time and disk cost. A change at the logical level — splitting a table, making a column nullable, adding a foreign key — can break correctness and needs the expand-and-contract discipline from Chapter 7.2.4. A change at the conceptual level — deciding the address now belongs to the customer rather than the order — is not a migration at all, it is a product decision that arrives disguised as one. Sorting an incoming change into the right level in the first minute tells you how much care it needs, and the most expensive schema mistakes are conceptual ones that got reviewed as if they were physical.
The last thing worth saying is where this often goes wrong in practice. A code-first ORM generates the physical model directly from classes in the application, which quietly skips the logical level: nobody ever wrote down the tables and keys, so nobody ever checked them against the normalization rules that follow. The tables are then whatever the class hierarchy happened to look like. Chapter 7.2.4 gives the honest assessment of ORMs; the specific damage here is that the level where correctness of the design is checked got skipped.
5. Normalization, derived rather than recited
Normalization is the process of splitting tables so that each fact is stored in exactly one place. The forms are usually memorised as a list; they are much easier to keep if you derive them from that one sentence.
Start with the broken table from the opening.
orders
order_id | customer_email | customer_address | product_sku | product_name | qty
---------+----------------+------------------+-------------+--------------+----
1001 | ana@x.com | 12 Oak St | SKU-9 | Kettle | 1
1002 | ana@x.com | 12 Oak St | SKU-4 | Toaster | 2
1003 | ben@y.com | 7 Pine Rd | SKU-9 | Kettle | 1Three failures live in that table:
- Update anomaly — Ana moves house; two rows must change, and one might not.
- Insertion anomaly — a new product cannot be recorded until somebody orders it, because the only place a product name exists is on an order row.
- Deletion anomaly — delete order 1003 and, if it was the only Kettle order, the fact that SKU-9 is called "Kettle" is gone.
A functional dependency is written A → B and means: if you know A, you know B. Here customer_email → customer_address, and product_sku → product_name. Those two dependencies are the whole diagnosis.
First normal form (1NF): one value per cell. No comma-separated lists, no repeating phone1, phone2, phone3 columns. Both make querying awkward and indexing impossible.
Second normal form (2NF): no non-key column depends on only part of a composite key. This only applies when the primary key is made of several columns. If order_lines has the key (order_id, sku) and stores product_name, then product_name depends on sku alone — half the key. Split it out.
Third normal form (3NF): no non-key column depends on another non-key column. In the original table customer_address depends on customer_email, which is not the key. Split it out. The old mnemonic — every non-key column depends on "the key, the whole key, and nothing but the key" — is 1NF, 2NF and 3NF in that order, and it is worth keeping because it is accurate.
Boyce-Codd normal form (BCNF) tightens 3NF for the rare case where a column that is part of a candidate key depends on a non-key column. You will meet it in an exam more often than in a schema; 3NF is where practical design lives.
Higher forms (4NF, 5NF) handle multi-valued dependencies and are genuinely rare. If someone asks, the honest answer is: 3NF or BCNF for transactional systems, then deliberate denormalization where measurement justifies it.
Denormalization is storing a fact twice on purpose, to avoid a join. It is not a failure — it is a trade you make with your eyes open, and it always costs the same thing: you now own the job of keeping the copies in agreement. Store order.total alongside the lines it is computed from and you have bought a fast read and taken on a consistency duty. Do it when a join is measurably too slow, not when you imagine it might be, and write down which process is responsible for the copy. Chapter 7.8.1 shows the analytics side, where denormalized shapes are the normal design rather than an exception.
6. When to break 1NF on purpose
Modern PostgreSQL and MySQL both store JSON in a column with indexing support. That is a direct violation of "one value per cell", and it is often the right call.
Use a JSON column when the structure genuinely varies per row and you never need to join or aggregate on the inside. Product attributes across thousands of categories, webhook payloads kept for audit, per-tenant custom fields.
Use real columns when you query it. The moment a field inside the JSON appears in a WHERE, an ORDER BY or a join, it wants to be a column: the planner has proper statistics for a column and much weaker ones for a JSON path, constraints can be enforced, and a typo in a key name is a compile-time-ish error instead of a silently empty result.
sql
-- jsonb: query it, and index the path you query
CREATE TABLE products (
sku text PRIMARY KEY,
attrs jsonb NOT NULL DEFAULT '{}'::jsonb -- (1)
);
CREATE INDEX ON products USING gin (attrs); -- (2)
SELECT sku FROM products WHERE attrs @> '{"colour":"red"}'; -- (3)(1) The column holds a whole object per row, with different keys per product category. (2) A GIN index (generalised inverted index — the same inverted-index idea Chapter 7.7 uses for text search) indexes every key and value inside the document, so containment queries do not scan the table. (3) @> means "contains this fragment". This prints the SKUs of red products, using the index.
The trap is the middle ground: a JSON column that everybody queries, which nobody indexed, on a table with ten million rows. That is a sequential scan on every request, and it is one of the most common causes of a database that was fine at 100k rows and fell over at 10 million.
7. SQL is relational algebra with a friendlier face
Codd also gave the model an algebra — a small set of operations that take relations and return relations. That closure property is why operations compose, and it is why SQL has subqueries and CTEs at all.
| Algebra | Meaning | SQL |
|---|---|---|
| Selection σ | Keep some rows | WHERE |
| Projection π | Keep some columns | SELECT a, b |
| Cartesian product × | Every pair | CROSS JOIN |
| Join ⋈ | Product then filter | JOIN … ON |
| Union ∪ | Rows from both | UNION |
| Difference − | Rows in A not in B | EXCEPT |
| Rename ρ | Rename attributes | AS |
The deeper point is that these operations have algebraic laws, and the laws are what make a query optimiser possible. Filtering then joining gives the same answer as joining then filtering, so the database is free to push a WHERE clause down below a join — turning a million-row intermediate result into a thousand-row one. You wrote what you want; the planner picks how.
That is why SQL is a declarative language. You never write the loop. Chapter 7.2.3 reads the plan the optimiser produced and shows what to do when it picks badly.
A join is a Cartesian product with a filter, conceptually. Nobody executes it that way — Chapter 7.2.1 covers the three algorithms that actually run — but it explains the shape of the result, and it explains exactly what a missing ON clause does to your server.
8. "SQL versus PostgreSQL" — what the question means
People say "should we use SQL or PostgreSQL", and the question is confused in a way worth untangling once.
SQL is a language standard. ISO/IEC 9075, revised roughly every few years — SQL-92 is the version everyone half-remembers, then SQL:1999 added recursion, SQL:2003 added window functions, SQL:2016 added JSON, SQL:2023 added property graph queries.
PostgreSQL, MySQL, Oracle Database, SQL Server and SQLite are database engines — programs that implement a large subset of the standard, plus their own extensions. No engine implements the whole standard, and every engine has features the standard does not describe.
So the standard is a common core, and everything around the edge is a dialect. The differences that actually bite:
| Task | PostgreSQL | MySQL | SQL Server / Oracle |
|---|---|---|---|
| Auto id | GENERATED … AS IDENTITY | AUTO_INCREMENT | IDENTITY / sequence |
| Limit rows | LIMIT 10 OFFSET 20 | same | OFFSET … FETCH NEXT |
| String join | || | CONCAT() | + or || |
| Quote identifier | "col" | `col` | [col] / "col" |
| Upsert | ON CONFLICT DO UPDATE | ON DUPLICATE KEY UPDATE | MERGE |
| Case sensitivity | Strings case-sensitive | Depends on collation | Depends on collation |
Three of those cause real production incidents.
Identifier case. PostgreSQL folds unquoted identifiers to lower case; quoting "userName" makes it permanently case-sensitive. A tool that quotes everything creates a schema you must quote forever.
String comparison. MySQL's common default collation compares strings case-insensitively, so WHERE email = 'Ana@X.com' finds ana@x.com. PostgreSQL does not. A migration between the two silently changes login behaviour, in the direction that lets people in.
Division. In PostgreSQL, 5 / 2 on integers is 2. So is it in most engines — but the surprise is common enough that averages computed in SQL are worth checking once.
The practical stance: write standard SQL where it costs nothing, use your engine's good extensions deliberately, and never assume a query is portable because it looks plain. If portability is a hard requirement, that requirement is expensive, and it should be a written decision rather than an assumption.
Which engine, in one paragraph each. PostgreSQL is the sensible default now: strict correctness, rich types, jsonb, extensions, a strong optimiser. MySQL is fast and widely operated, with a huge hosting ecosystem and historically looser defaults. SQLite is a library, not a server — the whole database is one file in your process, and it is the right answer for local applications, tests and embedded use, and wrong for concurrent writes from many machines. Oracle and SQL Server are enterprise engines with deep tooling and licensing costs, usually present because the estate already has them, and Chapter 13.9 covers that world honestly.
9. Constraints: pushing rules into the database
A constraint is a rule the engine enforces on every write, from every client, forever — including the one-off script somebody ran at 2am.
sql
CREATE TABLE order_lines (
order_id bigint NOT NULL REFERENCES orders(id) ON DELETE CASCADE, -- (1)
sku text NOT NULL REFERENCES products(sku), -- (2)
qty int NOT NULL CHECK (qty > 0), -- (3)
unit_price_minor int NOT NULL CHECK (unit_price_minor >= 0), -- (4)
PRIMARY KEY (order_id, sku) -- (5)
);(1) A line cannot name a missing order, and ON DELETE CASCADE says deleting the order removes its lines. The alternatives are RESTRICT (refuse the delete while children exist — the safer default for anything money-related) and SET NULL. (2) A line cannot name a missing product, and there is no cascade here on purpose: deleting a product that appears on orders should fail loudly. (3) Quantity zero and negative quantities are now unrepresentable, so no code path can create one. (4) Money is stored as an integer number of minor units — pence, cents — never as a float. Chapter 9.7.29 shows what floating-point money does to a ledger. (5) The composite primary key says a SKU appears at most once per order, which quietly removes an entire class of double-add bug.
The argument against constraints is that "the application validates it". The application is one of several writers: a background job, a data fix, an import, a second service, a future rewrite. The database is the only place where a rule holds for all of them. Validate in the application for a good error message; constrain in the database for the guarantee.
What the interviewer will push on
"Talk me through how you would design a schema from scratch." They are checking whether you work top-down through the three levels or start typing CREATE TABLE. Conceptual first — what things exist, how they relate, and the meaning questions like whether an address belongs to the customer or the order. Then logical — tables, keys, foreign keys, normalized. Then physical — types, indexes, partitioning for this engine. The tell is naming a decision that belongs at each level. The weak answer describes tables immediately, which means the meaning questions were never asked out loud.
"What is physical data independence?" Changing how data is stored — an index, a partition, a different disk — without changing a single query, because you asked for what and the engine decided how. Then contrast it with logical data independence, which is only partly achievable and is what views are for.
"What is normalization, and would you always normalize?" They are checking whether you can state the goal rather than recite forms: each fact in one place, to kill update, insertion and deletion anomalies. Then say where you would stop — 3NF for transactional tables, deliberate denormalization when a measured read path needs it, and always naming who keeps the copies in agreement. The weak answer recites 1NF-2NF-3NF and cannot say what a functional dependency is.
"Why is NULL difficult?" Because it turns logic three-valued, so WHERE status <> 'cancelled' silently drops null rows and NOT IN with a null returns nothing at all. The tell is knowing NULL = NULL is not true, and reaching for NOT EXISTS or IS DISTINCT FROM rather than patching it with COALESCE everywhere.
"Natural or surrogate primary key?" Surrogate by default, with the natural key kept as a UNIQUE constraint. The reason is that business identifiers change, and when a primary key changes so does every foreign key referencing it. The common wrong answer is "natural keys save a join" — they rarely do, because you still index the surrogate.
"Why put constraints in the database when the app validates?" Because the app is not the only writer. Name the others: migrations, backfills, a second service, an incident fix typed by hand. Then make the distinction — application validation for the message, database constraint for the guarantee.
"What does the relational model actually give you over files?" Physical independence: you state what you want and the engine decides how, so storage can change without rewriting programs, and the optimiser can rewrite your query using algebraic laws. That is the answer that shows you know why the model won, rather than that it exists.
One thing to volunteer: mention that a SQL table is a bag, not a set — duplicates are allowed, which is why DISTINCT exists and why UNION ALL is faster than UNION. It is a small point that shows you know where SQL and the relational model diverge, and it leads naturally into why UNION costs a sort or a hash.
Recall
- A relation is a set of tuples: no row order without
ORDER BY, and — in theory — no duplicates. SQL tables are bags, which is whyDISTINCTexists andUNION ALLbeatsUNION. - Choose a surrogate key by default and keep the natural key as
UNIQUE. Business identifiers change; a meaningless key cannot. NULLmakes logic three-valued.NULL = NULLis unknown,WHERE x <> 'a'drops nulls,NOT INwith a null returns no rows, aggregates skip nulls butCOUNT(*)does not, andUNIQUEpermits many nulls.- A design has three levels: conceptual (what things exist and how they relate), logical (tables, columns, keys — where normalization happens), physical (types, indexes, partitions for one engine). Physical changes cannot change an answer, only a speed — that is physical data independence. Schema also means a namespace; context tells you which.
- Normalization means each fact in one place, killing update, insertion and deletion anomalies. 1NF atomic cells, 2NF no dependence on part of a composite key, 3NF no dependence on another non-key column. Denormalization is a trade, not a sin — it buys a fast read and costs you a consistency duty.
- A functional dependency
A → B("knowing A tells you B") is the tool that diagnoses which form is broken. - SQL is declarative because relational algebra has laws — filter-then-join equals join-then-filter — which is exactly what lets an optimiser rewrite your query.
- SQL is a standard; PostgreSQL, MySQL, Oracle, SQL Server and SQLite are engines. Real portability traps: identifier case folding, collation-dependent string comparison, and upsert syntax.
- Constraints are rules enforced against every writer including the 2am script. Validate in the app for the message, constrain in the database for the guarantee.
Self-test: Why can the same SELECT return rows in a different order tomorrow? · What exactly does WHERE status <> 'cancelled' miss? · State the functional dependency that 3NF forbids · When is a JSON column the right choice, and what makes it the wrong one? · Why does a query optimiser have the freedom to reorder your query? · Name a MySQL-to-PostgreSQL migration difference that changes login behaviour.
Next: 7.2.1 starts SQL itself, with the joins visualised properly — including why LEFT JOIN plus a WHERE on the right table silently becomes an inner join.