Appearance
10.23 — Storage Systems: Block, File, Object & Erasure Coding
When designing systems, where you put the bytes is just as important as how you move them. Data storage is not a monolith. Different workloads demand entirely different architectures for writing, organizing, and retrieving bytes.
Three Paradigms: Block, File, and Object
The foundational choice in storage is the interface paradigm.
Block Storage (The Parking Lot)
The intuition: Imagine a vast parking lot where every space is numbered. You don't name your car; you just remember you parked in space 402.
Block storage divides data into fixed-size chunks (e.g., 4KB or 8KB), called blocks. Each block has an address. There is no concept of a "file" or "folder" at this layer. It is just a raw array of bytes.
Under the hood: Block storage is what your operating system sees when it looks at a hard drive or SSD. Cloud block storage (like AWS EBS) exposes a virtual drive over a high-speed network using protocols like iSCSI or NVMe-oF.
- Pros: Blazing fast, low latency. It is exactly what databases need to control their own destiny (managing B-Tree pages directly).
- Cons: Typically attached to a single machine at a time. The OS must format it with a filesystem (ext4, NTFS) to make it usable for humans.
File Storage (The Filing Cabinet)
The intuition: A standard office filing cabinet. Drawers contain folders, folders contain subfolders, and eventually, documents.
File storage (like AWS EFS, NFS, or SMB) provides a POSIX-compliant hierarchical namespace.
Under the hood: Multiple machines can mount a network file system concurrently. The storage system handles file locks, directory trees, and metadata (permissions, creation dates).
- Pros: Highly compatible with legacy software. If an app expects to read
C:\data\config.json, it works. Great for shared workspaces. - Cons: Metadata operations (like listing a directory with millions of files) are notoriously slow. The hierarchical tree becomes a bottleneck at scale.
Object Storage (The Valet Warehouse)
The intuition: A massive automated warehouse. You hand the valet a box (data) and a sticky note with a name on it (metadata). The valet hands you back a receipt (a unique ID or URL). You have no idea where in the warehouse the box is, but presenting the receipt gets it back instantly.
Object storage (like AWS S3) flattens the hierarchy. Everything is stored in a flat namespace (a bucket) identified by a key.
Under the hood: Objects are immutable; you cannot modify a single byte in the middle of a 5GB video. You must upload a new version. Access is entirely over HTTP (REST APIs).
- Pros: Infinite horizontal scalability. You never run out of "disk space." Extremely cheap. Custom metadata tags.
- Cons: High latency (HTTP overhead). No partial edits. Cannot be mounted as a traditional hard drive natively.
Durability: Replication vs. Erasure Coding
Hard drives fail constantly. To prevent data loss, storage systems use redundancy.
Replication (The Brute Force Approach)
Replication is straightforward: copy the exact same data to multiple independent disks or servers. The standard in distributed systems (like HDFS or early S3) is 3x replication.
The math: If a file is 1GB, storing it with 3x replication takes 3GB of raw disk space.
- Overhead: 200%
- Durability: High. You can lose any 2 machines and keep the data.
- Performance: Excellent for reads, as you can read from whichever replica is closest or least busy.
Erasure Coding (The Mathematician's Approach)
At exabyte scale, a 200% storage overhead is bankruptingly expensive. Enter Erasure Coding (EC).
The intuition: Instead of making copies of your passport, you shred it into 10 pieces. You then use math to generate 4 "magic" pieces. You hide these 14 pieces globally. If you lose any 4 pieces (data or magic), you can mathematically reconstruct the entire passport from the remaining 10.
Under the hood: EC uses algorithms like Reed-Solomon. Data is split into k data chunks, and the system computes m parity chunks using matrix multiplication (Vandermonde matrices over Galois fields).
For a 10+4 scheme (10 data, 4 parity):
- Overhead: 40% (14 total chunks for 10 chunks of data).
- Durability: It survives the loss of any 4 of the 14 chunks. To survive 4 losses with plain copies you would need 5 copies (lose 4, keep 1), which is 400% overhead. So 10+4 matches the failure tolerance of 5× replication at 40% overhead, roughly ten times cheaper in extra disk.
- Performance: CPU intensive. Reconstructing a missing chunk requires reading 10 other chunks over the network and solving algebraic equations. This is called the "rebuild penalty."
The Multipart Upload Pattern
When dealing with Object Storage, uploading a 50GB 4K video over a flaky home internet connection as a single HTTP request is suicidal. If the connection drops at 99%, you restart from zero.
The solution is Multipart Upload.
- Initiate: Tell the storage system you are starting a large upload. It returns an
UploadId. - Upload Parts: Break the file into chunks. S3 sets two limits that decide the chunk size: every part except the last must be at least 5 MiB, and one upload may have at most 10,000 parts. A 50 GiB file cut into 5 MiB parts would need 10,240 parts, which S3 rejects, so the part size has to grow with the file. Upload them independently, in parallel, tagging each with the
UploadIdand aPartNumber. If part 7 fails, you only retry part 7. - Complete: Tell the storage system "I'm done, here is the list of parts." The server concatenates them into a single object.
TypeScript Implementation: Orchestrating a Multipart Upload
A client that uploads a large video, written so that it works for a 50 GiB file on an ordinary laptop. Reading the whole file into memory first is not an option: fs.readFile refuses anything over 2 GiB with ERR_FS_FILE_TOO_LARGE, and even below that limit, holding a whole video in memory starves the machine. So the code reads one part at a time from disk, keeps only a few parts in memory, and retries any part that fails.
ts
import { open } from "node:fs/promises";
const MIN_PART = 5 * 1024 * 1024; // (1) S3: every part except the last is at least 5 MiB
const MAX_PARTS = 10_000; // S3: at most 10,000 parts per upload
const PARALLEL = 4; // (2)
interface UploadedPart { partNumber: number; eTag: string; }
interface StorageClient {
initiate(key: string): Promise<string>; // returns uploadId
uploadPart(uploadId: string, partNumber: number, data: Buffer): Promise<string>; // returns eTag
complete(uploadId: string, parts: UploadedPart[]): Promise<void>;
}
export async function uploadVideo(client: StorageClient, path: string, key: string): Promise<number> {
const file = await open(path, "r");
try {
const { size } = await file.stat();
const partSize = Math.max(MIN_PART, Math.ceil(size / MAX_PARTS)); // (3)
const partCount = Math.max(1, Math.ceil(size / partSize));
const uploadId = await client.initiate(key);
const parts: UploadedPart[] = [];
let next = 1;
const worker = async () => { // (4)
while (next <= partCount) {
const partNumber = next++;
const offset = (partNumber - 1) * partSize;
const length = Math.min(partSize, size - offset);
const buf = Buffer.alloc(length);
await file.read(buf, 0, length, offset); // (5)
const eTag = await withRetry(() => client.uploadPart(uploadId, partNumber, buf)); // (6)
parts.push({ partNumber, eTag });
}
};
await Promise.all(Array.from({ length: PARALLEL }, worker));
parts.sort((a, b) => a.partNumber - b.partNumber); // (7)
await client.complete(uploadId, parts);
return partCount;
} finally {
await file.close();
}
}
async function withRetry<T>(fn: () => Promise<T>, attempts = 5): Promise<T> {
for (let i = 1; ; i++) {
try { return await fn(); }
catch (err) {
if (i === attempts) throw err;
await new Promise(r => setTimeout(r, 2 ** i * 100 * Math.random())); // backoff with jitter
}
}
}- The two S3 limits from the step list above, written as constants so the arithmetic below can respect them.
- How many parts are in flight at once. Four keeps a home connection busy without holding much in memory: at most four parts are buffered at any moment.
- The part size grows with the file. It is at least 5 MiB, and large enough that the file fits in 10,000 parts. A 1 GiB file uses 5 MiB parts (205 of them). A 50 GiB file needs 5.12 MiB parts, which gives exactly 10,000. A 100 GiB file gets 10.24 MiB parts.
- Four workers share one counter. Each worker takes the next part number, uploads it, and comes back for another, so a slow part never holds up the others.
- Each part is read straight from its position in the file. Nothing else of the file is in memory.
- A failed part is retried on its own with exponential backoff and random jitter (10.9); the parts that already succeeded are untouched. This is the whole reason multipart exists.
- Workers finish in any order, but the final "complete" call must list the parts in order, so sort before sending.
Run against a fake storage client with a 23 MiB file, where part 3 fails once, it uploads five parts (four of 5 MiB and a last one of 3 MiB), never has more than four in flight, retries part 3, and completes with parts 1,2,3,4,5 in order.
One more production detail: an upload that is started and never completed leaves its parts stored, and you are billed for them. Configure a lifecycle rule that aborts incomplete multipart uploads after a few days.
Storage Tiers: Hot, Warm, and Cold
Not all bytes are created equal. The value of data typically decays rapidly over time.
- Day 1: A user uploads a profile picture. It is requested thousands of times. (Hot)
- Day 30: The picture is requested rarely. (Warm)
- Day 365: The user deleted their account, but compliance laws require keeping the data for 7 years. (Cold)
Storage systems offer tiers to optimize cost.
| Tier | Media | Cost | Retrieval Time | Use Case |
|---|---|---|---|---|
| Hot | NVMe SSDs | Very High | Milliseconds | Active database files, real-time analytics. |
| Warm | Standard HDDs | Medium | Milliseconds | Standard S3 objects, user-uploaded assets. |
| Cold | Tape / SMR Drives | Ultra Low | Minutes to Hours | Compliance archives, backups (e.g., S3 Glacier). |
To save money, systems implement Lifecycle Policies—automated background jobs that migrate objects from Hot to Warm to Cold as they age.
What the interviewer will push on
"We need to allow users to upload 10GB video files. Should we stream the file through our backend API server to S3?" Absolutely not. Routing massive files through your Node.js or Go application servers will exhaust their memory, saturate their network interfaces, and require huge scale-out costs just to act as a proxy. The correct answer: Use Pre-signed URLs. Your backend authenticates the user and generates a temporary, signed URL. The client (browser/mobile app) uploads the bytes directly to S3 using that URL. Your backend is bypassed entirely for the heavy lifting.
"Why would we ever use 3x Replication if Erasure Coding is so much cheaper?" Erasure coding trades compute (CPU) and latency for storage efficiency. If you are building an ultra-low latency system (like a caching layer or a hot database transaction log), the overhead of calculating parity and network scatter/gather to read/write chunks is unacceptable. Use replication for hot, latency-sensitive data. Use erasure coding for warm/cold object storage.
"How does an object store track where all these distributed chunks actually live on disks?" They use a separate metadata tier. Usually, a highly consistent key-value store (like FoundationDB or Paxos-based metadata servers) maps ObjectId to [Physical Disk 1, Physical Disk 2, ...].
Recall
- Block Storage: Raw bytes, fixed chunks, attached to single hosts. Best for databases.
- File Storage: Hierarchical folders, POSIX permissions, network-attached. Best for legacy apps.
- Object Storage: Flat namespace, immutable, accessed via HTTP API. Best for scale and media.
- Replication: Simple copies (e.g., 3x). Fast, durable, expensive.
- Erasure Coding: Math-based redundancy (e.g., 10+4). Slower rebuilds, highly cost-effective.
- Multipart Upload: Splitting large objects into parallel uploads to tolerate network failures.
Quiz
Why is Object Storage a poor choice for hosting a relational database's active data files? a) It cannot store files larger than 5GB. b) Objects are immutable; databases need to perform small, random reads and overwrites. c) Object storage limits the number of files you can store. d) It does not support encryption.
If an Erasure Coding scheme uses 8 data chunks and 3 parity chunks, how many simultaneous chunk failures can the system survive without losing data? a) 8 b) 11 c) 3 d) 5
What is the primary benefit of the Multipart Upload pattern? a) It converts object storage into block storage. b) It reduces the overall size of the file being uploaded. c) It allows resuming failed uploads and speeds up transfers via parallelization. d) It automatically encrypts data in transit.
Answers: 1: b, 2: c, 3: c
Next: 10.24 asks what happens when one business action has to change data in several of these stores at once.