11.2 — File Storage & Resumable Upload

A user is 3.8 GB into a 4 GB upload when their train enters a tunnel. Thirty seconds later the connection comes back. If your system makes them start again, the product is broken — not slow, broken, because on a mobile connection the upload will never finish and they will stop trying.

That single sentence contains the entire design. Everything below follows from refusing to let a network drop cost more than a few seconds of work: the file has to be split into pieces that succeed independently, the pieces have to be tracked somewhere the client can ask about after a restart, and the bytes cannot travel through your application servers, because your application servers restart during deploys and a 4 GB upload cannot survive a deploy if it is being held in one of them.

This study also establishes the metadata and blob split — small structured rows in a database, large opaque bytes in object storage — which a dozen later studies reuse without re-deriving.

1. Requirements

Functional. Upload files up to 5 GB. Download them. List a user's files. Delete. Share by link. Uploads must be resumable across network drops and application restarts. Identical content stored twice should cost storage once.

Non-functional, with numbers.

• Durability of eleven nines — 99.999999999%, which means that if you store 100 million objects you expect to lose one every hundred thousand years. This is the number the industry quotes, and it is worth understanding what it costs to reach rather than repeating it.
• 99.99% availability for reads. Being unable to upload for ten minutes is annoying. Being unable to download is the product being down.
• Upload throughput limited only by the client's connection, which means no per-file serialisation and no application-server bottleneck in the middle.
• Cost-efficient at petabyte scale, because at this size storage cost is not a line item, it is the business.

Out of scope today: real-time collaborative editing (11.13), full-text search inside file contents, and the versioning user interface — though I will keep the data model able to support versions.

The clarifying questions, and what each answer changes

"What is the largest file, and what is the median file?" These two numbers pull in opposite directions and both matter. A 5 GB maximum forces multipart uploads and resumability. A median of 2 MB means most uploads are a single request and the multipart machinery is pure overhead for them, so the client needs a threshold below which it does the simple thing.

"Are files immutable once uploaded, or edited in place?" Immutable files make everything easy: no coordination, trivial caching, safe deduplication. Editable files turn this into a distributed-state problem with conflicts, which is a different system (and the subject of the drill at the end).

"Is deduplication across users acceptable?" Cross-user deduplication saves an enormous amount of storage and leaks a fact: an attacker who can time uploads can learn whether a specific file already exists in the system. For consumer photo storage that is usually fine. For a service holding legal documents it is not, and deduplication has to be scoped per tenant.

"How long do deleted files stay recoverable?" This decides whether delete is a real deletion or a tombstone with a retention window, and it interacts with deduplication in a way that surprises people: with reference counting, one user deleting a file must not delete the bytes another user still points at.

"Do we need to serve downloads globally?" If yes, a content delivery network is not an optimisation, it is where the read path lives, and the access-control design has to work through it.

2. Estimation

Total stored data. 10 million users at an average of 100 GB each = 1 exabyte. What that forces: everything. One exabyte is not a database. It is object storage, erasure-coded across many failure domains, with a lifecycle policy moving cold data to cheaper tiers. This number eliminates every design where bytes live in the same system as the metadata, and it eliminates it in the first minute rather than the thirtieth.

Ingest rate. 10M users × 5 files a day × 2 MB average = 100 million files and 100 TB a day. Divide by 86,400 seconds: ~1.2 GB per second sustained, and with a 3× evening peak, ~3.5 GB/s. What that forces: the bytes cannot pass through application servers. Proxying 1.2 GB/s means provisioning application capacity purely to relay input to output — capacity that the storage service already has and sells more cheaply — and it means an application deploy interrupts every in-flight transfer.

Read rate. At a 5:1 read-to-write ratio, ~6 GB/s of downloads, concentrated on recent and shared files. What that forces: a content delivery network in front of the storage, and an access-control scheme that works when the API is not on the byte path.

Metadata volume. 10M users × 500 files each × ~1 KB per row = 5 TB. What that forces: almost nothing, and that contrast is the lesson of this study. The metadata is five terabytes and fits comfortably in a partitioned database; the blobs are an exabyte and need a completely different system. The two are separated because their numbers are six orders of magnitude apart, not because separation is a fashionable pattern.

Request rate on the metadata API. 100M uploads a day ≈ 1,200 a second, each involving an init and a complete, plus listings and permission checks — call it 5,000 requests a second at peak. Small, ordinary, stateless work.

Part count arithmetic, which decides the part size. With 8 MB parts, a 5 GB file is 640 parts. Object stores typically cap a multipart upload at 10,000 parts, so 8 MB parts support files up to 80 GB — comfortable. Halve the part size to 4 MB and you double the request count for no benefit on a mobile connection; raise it to 64 MB and a dropped connection now costs 64 MB of re-upload. 8–16 MB is the sweet spot because it is large enough that per-part overhead is negligible and small enough that losing one is cheap.

3. API

The shape of this API is unusual and worth stating before the endpoints: the application never sees the bytes. It issues credentials, verifies outcomes, and records metadata. That is three small, fast, stateless operations, and everything expensive happens between the client and the storage service.

POST /uploads
Content-Type: application/json

{ "fileName": "holiday.mp4", "size": 4294967296,
  "contentHash": "sha256:7d3f...", "mimeType": "video/mp4" }

201 Created
{ "uploadId": "up_01J9F3K7Q",
  "partSize": 8388608,
  "parts": [ { "n": 1, "url": "https://blob.example/...&partNumber=1", "expiresAt": "..." } ],
  "alreadyStored": false }

If contentHash is already present in the blob store, the response is instead:

201 Created
{ "uploadId": "up_01J9F3K7R", "alreadyStored": true, "fileId": "f_01J9F3M22" }

— and the upload is finished, having transferred zero bytes of file content.

PUT <presigned part URL>          # client → object storage, directly
→ 200 OK
  ETag: "9b2c1f..."               # the storage service's receipt for this part

GET /uploads/up_01J9F3K7Q         # after a crash or a tunnel
→ 200 OK
  { "uploadId": "up_01J9F3K7Q", "partSize": 8388608,
    "receivedParts": [1,2,3,5,6], "missingParts": [4,7,8],
    "expiresAt": "2026-08-07T00:00:00Z" }

POST /uploads/up_01J9F3K7Q/complete
{ "parts": [ { "n": 1, "etag": "9b2c1f..." } ] }
→ 201 Created
  { "fileId": "f_01J9F3M22", "size": 4294967296, "contentHash": "sha256:7d3f..." }

GET /files/f_01J9F3M22/content
→ 302 Found
  Location: https://cdn.example/blob/ab/cd/7d3f...?token=...&expires=1754000000

DELETE /files/f_01J9F3M22
→ 204 No Content

Idempotency. complete is idempotent on uploadId: calling it twice returns the same fileId rather than creating a second file (10.4). This matters more here than in most systems, because completion is exactly the request most likely to be retried — it happens at the end of a long, flaky transfer when the client has every reason to be uncertain whether its request arrived.

The server verifies, it does not trust. On complete, the API asks the storage service which parts exist and what their etags are, and compares against both the client's claim and the size declared at init. A client that lies — or a buggy client that reports success for a part that failed — must not be able to create a files row pointing at an incomplete object. This is the difference between an API that observes an upload and one that authorises it.

Errors, one envelope:

409 Conflict
{ "error": { "code": "upload_expired",
             "message": "This upload expired. Start a new one.",
             "requestId": "req_01J9F3P5X" } }

400 for a size or hash that fails validation, 409 for an expired or already-completed upload, 413 for a file over the limit, 422 when parts are missing at completion (with the missing part numbers in the body so the client can finish rather than restart), and 507 when the account is out of quota.

4. Data model

files
  file_id        UUID PRIMARY KEY
  owner_id       UUID NOT NULL
  name           TEXT NOT NULL
  size           BIGINT NOT NULL
  content_hash   CHAR(64) NOT NULL      -- points into blobs
  status         SMALLINT NOT NULL      -- active | trashed | broken
  created_at     TIMESTAMPTZ NOT NULL
  deleted_at     TIMESTAMPTZ NULL

uploads
  upload_id      UUID PRIMARY KEY
  owner_id       UUID NOT NULL
  expected_size  BIGINT NOT NULL
  content_hash   CHAR(64) NOT NULL
  storage_key    TEXT NOT NULL
  status         SMALLINT NOT NULL      -- open | completed | aborted
  expires_at     TIMESTAMPTZ NOT NULL

blobs
  content_hash   CHAR(64) PRIMARY KEY
  storage_key    TEXT NOT NULL
  size           BIGINT NOT NULL
  refcount       BIGINT NOT NULL        -- how many files point here
  created_at     TIMESTAMPTZ NOT NULL

Access patterns, written before the partition key is chosen:

Query	Frequency	Returns
List one owner's files, newest first	high	50–200 rows
Fetch one file by id (with a permission check)	high	one row
Look up a blob by content hash	1,200/s	one row
Resume: fetch an upload by id	moderate	one row
Decrement a blob refcount on delete	moderate	one row

Partition files by owner_id. Every high-volume query on this table is owner-scoped — listing, quota, permission checks — so co-locating a user's rows means those queries touch one partition (10.6). The risk to name yourself is a hot owner: a shared team account with ten million files makes one partition much larger than the others, and the mitigation is a composite key of (owner_id, bucket) where bucket is derived from the file id, splitting a large owner across several partitions at the cost of a small scatter on listing.

Partition blobs by content_hash. Every access is a point lookup by hash, so hashing spreads perfectly and there is nothing that wants a range.

Indexes and their queries:

• files (owner_id, created_at DESC) — the file listing, which is always "this user, newest first".
• files (content_hash) — needed when a blob's refcount drops to zero and you want to confirm nothing points at it.
• uploads (expires_at) — the cleanup job that aborts abandoned uploads.
• blobs (refcount) where refcount = 0 — the garbage collector's work queue.

The blobs table is the deduplication mechanism, and refcount is what makes deletion safe. Two users uploading the same holiday video share one stored object; when one deletes, the refcount drops to one and the bytes stay; when it reaches zero, the bytes become eligible for collection after a grace period.

5. Architecture

Three paths, three panels. The upload path first, because it is the one with the interesting protocol.

Figure 1 — The upload path. The API issues part-scoped credentials and verifies the outcome; the bytes travel client to storage directly, so a 5 GB file consumes no application memory, no application bandwidth and no file descriptors. An application deploy in the middle of this transfer is invisible to the user.

The download path is a different shape, and drawing it separately shows where access control actually happens.

Figure 2 — The download path. The API is on the path exactly once, to decide. The bytes then flow through a cache that has never heard of your permission model, which is only safe because the signed URL is short-lived and scoped to one object.

And the reconciliation path, which exists because two stores that must agree will eventually disagree.

Figure 3 — Reconciliation. Three sweeps, each with a different action. The one that costs real money is the third: abandoned multipart parts are stored and billed but appear in no object listing, so they are invisible until the bill arrives.

6. Deep dives

6.1 What actually makes an upload resumable

Three properties, and all three are required. Remove any one and resumability disappears.

The file is split into parts that succeed or fail independently. Fixed-size parts of 8 MB, numbered from one. Each part is a separate HTTP request to a separate presigned URL. Nothing about part 5 depends on part 4 having arrived, which is also why parts can be uploaded in parallel — three or four at a time saturates a home connection without the head-of-line blocking a single stream would suffer.

Each part's success is recorded by a system that is not the client. When a part lands, the storage service returns an etag — a receipt, usually a hash of the part's bytes — and remembers that the part exists. This is the property that survives an application restart, a client crash, and a phone running out of battery, because the record lives with the bytes.

The client can ask which parts exist. After the tunnel, the client calls GET /uploads/{id}, gets missingParts: [4,7,8], and uploads three parts instead of 4 GB. For this to work across an app restart, the client must have persisted the uploadId to local storage the moment it was issued — which is the one client-side requirement the server cannot enforce and the one most often missed.

The cost of a dropped connection is therefore at most one part in flight per parallel stream, so with four parallel streams of 8 MB, a tunnel costs at most 32 MB of re-upload. Compare with the single-PUT design, where the same tunnel costs 3.8 GB.

Completion is the dangerous moment, and it is worth walking slowly. The client says "all 640 parts are up, here are their etags." The server must not write the files row on the strength of that claim. It asks the storage service to assemble the object, which fails if any part is missing, then verifies the assembled size against expected_size recorded at init. Only then does it write metadata. And because the client may retry complete — it is the request most likely to time out, arriving at the end of a long flaky transfer — the endpoint looks up uploadId, finds status = completed, and returns the existing fileId unchanged.

6.2 Deduplication, and what it leaks

The client hashes the file before uploading and sends the hash at init. If that hash is already in blobs, the server increments the refcount, writes a files row pointing at the existing blob, and returns alreadyStored: true. The upload finishes in one round trip having moved none of the file's bytes. This is the feature that made early sync products feel impossible: re-uploading a 700 MB video you already have takes half a second.

The saving is real. In a consumer product, a large fraction of stored bytes are duplicates — the same viral video, the same installer, the same photo shared into twenty group chats — and deduplication commonly removes 20–40% of stored bytes with no user-visible change.

Two things must be said, and interviewers wait for both.

Never trust the client's hash for integrity. A malicious client could claim its 4 GB of malware hashes to the same value as a popular installer, then point a files row at content it never uploaded — or worse, in the other direction, poison a blob everyone shares. The rule is that the client's hash is a lookup hint only. When bytes are actually uploaded, the storage service computes the hash itself, and that computed value is what goes in blobs. If the client's hint was wrong, you have simply stored the file normally.

Cross-user deduplication leaks existence. If uploading a file completes instantly, the uploader has learned that someone, somewhere, has that exact file. For most content this is harmless. For a leaked document, a specific medical record, or a file that identifies its holder, it is a real disclosure — and it is testable at scale, because an attacker can generate candidate files and check which ones deduplicate. The mitigations, in order of cost: scope deduplication per account (no cross-user saving, no leak), scope it per tenant (saving inside an organisation, no leak across organisations), or keep global deduplication but always transfer the bytes and deduplicate server-side, which removes the timing signal while keeping the storage saving. That last option is the one most large providers actually chose, and saying so shows you know the difference between saving storage and saving bandwidth.

6.3 Where eleven nines come from

Eleven nines is not achieved by copying the file three times. Here is the arithmetic and the machinery.

Erasure coding. Split each object into k data fragments and compute m parity fragments, so that any k of the k+m fragments can rebuild the object. With k=10 and m=4 you store 14 fragments totalling 1.4× the original size, and you survive the loss of any four of them. Compare with 3× replication: three times the storage to survive the loss of any two copies. Erasure coding gives more redundancy for less than half the storage, and at exabyte scale that difference is the entire cost structure.

What it costs, honestly: reading requires fetching k fragments from k different machines and doing a little arithmetic, so a small-object read has higher latency and more moving parts than reading one replica. That is why very small objects are sometimes replicated instead and only larger ones erasure-coded.

Spread across failure domains. Fourteen fragments on fourteen disks in one rack survive disk failure and not a rack losing power. Fragments are placed so that no single rack, and ideally no single availability zone, holds enough of them to matter.

Continuous scrubbing. Disks do not only fail loudly; they also return wrong bytes quietly, which is why every fragment carries a checksum and a background process continuously re-reads and re-verifies stored data. When a fragment is found bad or missing, it is rebuilt from the others immediately. The durability number is a statement about repair speed, not about redundancy alone — an object with four spare fragments and a repair process that takes a month is far less durable than one with two spares repaired in an hour. This is the sentence that shows you understand the number rather than quote it.

Lifecycle tiering, which is the cost lever. Data untouched for 90 days moves to a colder, cheaper storage class with slower first-byte latency. At an exabyte, moving 60% of data down one tier is the single largest saving available, and it is a policy change rather than an engineering project.

6.4 Sharing, access control, and why the redirect exists

Downloads flow through a content delivery network that knows nothing about your permission model. Access control therefore happens exactly once, when the URL is issued.

GET /files/{id}/content hits the API, which checks that the caller may read this file, then returns a 302 to a signed URL valid for a few minutes and scoped to one object. The client follows it, the CDN serves the bytes, and your API sees none of the 6 GB/s.

Why the indirection rather than storing the storage URL directly? Because a permanent URL is a permanent permission. If a share link were the storage URL, revoking access would be impossible — the recipient already has it, and there is no request to intercept. The redirect keeps a checkpoint you control: every download begins with a request to you, and that request is where revocation, expiry, password checks and rate limits are enforced.

Public share links are capability URLs: the unguessable token is the permission, so anyone holding the link can read the file. That is a deliberate product choice, and it comes with the obligations a capability implies — the token must be long enough that guessing is impractical, links must be revocable (see above), and a link should support an expiry and optionally a password. The failure to name is that capability URLs leak through Referer headers, browser history, and screenshots in group chats, so anything genuinely sensitive should require identity rather than a link.

CDN caching and privacy. The CDN caches by blob key, not by user, which is what makes a popular shared file cheap. But it also means two different users' signed URLs resolve to the same cached object, so the signature must be validated at the edge and the cached object must not be servable without one. Getting this wrong is how private files become publicly readable, and it is worth stating the rule out loud: the cache key is the blob, the authorisation is the signature, and the edge must check the signature on every request even when the object is already cached.

6.5 Two stores that must agree

The design's chief hazard is not load, it is drift. Metadata and bytes live in different systems, and no single operation can update both atomically.

Write order: bytes first, metadata second. If the process dies between them you get an orphan blob — bytes with nothing pointing at them, invisible to users, costing money until a sweep collects them. That is a cost problem. Reverse the order and a crash gives you dangling metadata — a file the user can see and cannot open. That is a correctness problem and a support ticket. Between a cost problem and a correctness problem, choose the cost problem every time; this is the same reasoning that puts the journal before the shutter in the ATM design in 9.7.8.

Reconciliation runs in both directions, continuously. One sweep lists blob keys and checks each has a referencing row, collecting orphans after a grace period long enough that an upload in flight is never collected. The other samples files rows and confirms their blob exists, marking any failures broken and surfacing them to the user honestly with an offer to re-upload — because a file that silently fails to download is worse than a file that says it is damaged.

Abandoned multipart uploads are the expensive one. Parts from an upload that was never completed are stored and billed but appear in no object listing, so nobody sees them until the invoice does. A lifecycle rule aborting incomplete uploads after seven days is a one-line policy that has saved organisations very large sums, and forgetting it is one of the most common real-world mistakes in this whole design.

7. Decision Ledger

Decision	Alternatives	Why this	What it costs
Presigned direct-to-storage upload	proxy bytes through the API	zero application bandwidth or memory; deploys do not interrupt transfers	more complex client; the API verifies rather than observes
Multipart, 8 MB parts	single PUT; 64 MB parts	a drop costs ≤ 8 MB per stream; 640 parts for 5 GB stays under the 10,000 cap	more requests; orphaned parts need lifecycle cleanup
Content-hash deduplication	store every upload	20–40% storage saved; instant re-upload of known files	existence leak unless scoped; hash must be recomputed server-side
Erasure coding k=10 m=4	3× replication	1.4× storage for survival of 4 losses	higher read latency and CPU; poor fit for tiny objects
Metadata in a database, blobs in object storage	one system for both	the two workloads differ by six orders of magnitude	drift between stores; permanent reconciliation cost
302 to a short-lived signed URL	hand out permanent storage URLs	revocation and expiry stay possible	one extra round trip per download
Bytes written before metadata	metadata first	a crash leaves an orphan (cost) not a broken file (correctness)	garbage collection sweep required forever

8. Scale and failure

At 10×. The metadata tier shards further by owner, which is easy because the partition key was chosen for it. Ingest at 12 GB/s is a matter of more parallel clients, since no shared component sits on the byte path. The thing that breaks first is the blob lookup by content hash at deduplication time, which is now 12,000 point reads a second on the write path — cache it, since the answer for a given hash never changes.

At 100×. Two things change qualitatively. Storage becomes multi-region for the files that need it, which doubles cost and introduces replication lag on the read path — so it is a per-bucket policy, not a global one. And the metadata database's listing query becomes the pressure point, because users with millions of files exist at this scale; the answer is the composite (owner_id, bucket) key mentioned in section 4, plus cursor pagination that never offsets.

The hot-object problem — one shared file downloaded by ten million people — is solved by the CDN, not by the storage tier. Worth naming because candidates often reach for storage replication when the answer is a cache that already exists.

What breaks	Blast radius	How you find out	What keeps it running	Recovery
Client dies mid-upload	one upload	none needed	parts persist; client resumes from missingParts	lifecycle aborts after 7 days
Metadata database down	uploads, listings, new downloads fail	API error rate	already-issued signed URLs keep working, so downloads in progress continue	promote a replica; uploads retry naturally
Object storage degraded in one zone	none, if erasure fragments span zones	storage service health; repair queue depth	remaining fragments rebuild the object	background repair; watch repair queue, not error rate
CDN outage	downloads slow, origin load spikes	edge error rate; origin bandwidth	serve from origin with a signed URL	restore edge; expect a cold cache
Reconciliation job stopped	orphans accumulate silently, costing money	orphan-count metric flat or missing	nothing user-visible breaks	restart; the sweep is idempotent
No lifecycle rule on incomplete uploads	pure cost, invisible in listings	storage bill vs stored-bytes metric	nothing	add the rule; the saving is immediate

The detection lesson of this study: two of the six failure rows are invisible to users and visible only on a finance or metrics dashboard. In a two-store system, drift and cost are the failure modes that do not page anyone, which is exactly why they need explicit metrics — orphan count, dangling-metadata count, and incomplete-upload bytes — reported daily with alarms on each (10.10).

What the interviewer will push on

"Why can't the bytes go through your API? It would be simpler." They are checking whether you can price a simplification. Four independent reasons, any one sufficient: buffering a 5 GB upload exhausts application memory, and even streaming it consumes file descriptors and event-loop time that serving traffic needs (9.9.4); at 1.2 GB/s you would be provisioning application servers purely as a byte relay; resumability is already implemented by the storage protocol and re-implementing it means re-implementing part tracking, durability and cleanup; and a deploy or a crash would kill every in-flight upload. The tell is the fourth reason, because it is about operations rather than performance and most candidates never reach it.

"The client tells you the file's hash. What could go wrong?" This is a trust-boundary probe. The good answer separates the two uses of the hash: as a lookup hint it is harmless, and as a statement about content it is unverified input that must never be written into blobs. Then name the second-order issue, which is that instant completion on a hash match leaks the existence of files across users, and give the three mitigations with their costs. The common wrong answer treats deduplication as purely a storage optimisation and never mentions that it is also an oracle.

"Where exactly do eleven nines come from?" They want to know whether the number is understood or recited. Strong answer: erasure coding for redundancy (k+m fragments, storage overhead 1.4× for four tolerated losses), placement across failure domains so correlated failures do not take out enough fragments, and — the part people miss — continuous scrubbing with fast repair, because durability is a function of how quickly you restore redundancy after losing it, not just how much redundancy you started with. Weak answer: "we replicate it three times."

"A user says their file downloads fine but a colleague gets a 403. Debug it." They are probing whether you understand where authorisation lives in this design. The answer walks the download path: permission is checked once, at URL issue time, so a 403 from the API means the permission model, and a 403 from the CDN means the signature — expired, scoped to a different object, or being validated against a cached object incorrectly. Naming that these are two different systems producing the same status code is the tell.

"Your storage bill tripled. Where do you look first?" Three suspects in order: incomplete multipart uploads with no lifecycle rule (invisible in object listings — check the incomplete-upload bytes metric specifically), deduplication silently not working (compare distinct content_hash count against files count and see whether the ratio moved), and no tiering policy so everything sits in the hottest class. The order matters, because the first one is both the most common and the hardest to see.

"Why not just use last-writer-wins if two clients upload the same path?" This is the setup for the sync problem. The answer is that files here are immutable blobs addressed by content, so "the same path" is a naming question in the metadata layer, not a byte question — and that once files become editable, last-writer-wins silently destroys work and stops being acceptable. Getting to "this is a different system once files are mutable" quickly is the point.

Volunteer this, because nobody asks: the metric that would have caught most of these problems is not a latency graph, it is a daily reconciliation report with three counts on it — orphan blobs, dangling metadata rows, and bytes held in incomplete uploads. In a two-store design those three numbers are the health of the system, they never appear in an error rate, and if nobody owns them they drift for months. Saying that out loud signals that you have operated one of these rather than designed one.

Next: 11.3 — from storing bytes to refusing requests. The rate limiter is the smallest system in this Part and the one whose correctness is hardest to define, because "10 requests per second" turns out to mean four different things depending on which algorithm you pick.

Recall

• The split: blobs in object storage (1 EB), metadata in a partitioned database (5 TB). Six orders of magnitude apart, which is why they are separated. files partitioned by owner_id, blobs by content_hash.
• The API never touches bytes. It issues credentials, verifies outcomes, records metadata. Consequence: a deploy cannot interrupt an upload.
• Resumability needs three things: independent fixed-size parts (8–16 MB), per-part receipts (etags) held by the storage service, and a "which parts exist?" query. Client must persist uploadId locally. A tunnel costs ≤ one part per parallel stream.
• Complete verifies, never trusts: check parts and total size against storage, then write metadata; idempotent on uploadId because this is the request most likely to be retried.
• Deduplication by content hash gives zero-byte uploads for known files, saving 20–40% of stored bytes. Two rules: the client hash is a hint only (server computes the real one), and instant completion leaks existence — scope per account or per tenant, or always transfer and deduplicate server-side.
• Eleven nines = erasure coding (k=10, m=4 → 1.4× storage, survives 4 losses) + placement across failure domains + continuous scrubbing with fast repair. Durability is about repair speed, not redundancy alone. Tiering is the cost lever.
• Downloads: 302 to a short-lived signed URL. Permission checked once, at issue. The indirection is what makes revocation possible; permanent storage URLs are permanent permissions.
• Two-store rule: bytes first, metadata second (an orphan costs money, a dangling row breaks a file), then reconcile both directions forever. Lifecycle-abort incomplete uploads or pay for them indefinitely.

Self-test: What three properties make an upload resumable? Why is the client's hash untrustworthy, and what does deduplication leak? Where do eleven nines actually come from? Which store is written first and why? Name the three numbers on the daily reconciliation report.

Quiz Bank

FoundationalExplain the resumable upload protocol end to end, including what happens after a crash.

Init. The client sends {fileName, size, contentHash}. The server creates an uploads row with an expiry, asks object storage to begin a multipart upload, and returns an uploadId, a part size (8–16 MB) and presigned URLs for the parts. If the content hash is already known, it instead returns alreadyStored: true with a fileId, and the upload is over.

Transfer. The client uploads parts directly to object storage, typically three or four in parallel. Each successful part returns an etag, and the storage service records that the part exists. Nothing about part 5 depends on part 4, which is what makes parallelism and independent retry possible.

Interruption. The connection drops, or the application is killed, or the phone reboots. Because the client persisted the uploadId at init, on restart it calls GET /uploads/{id} and receives receivedParts and missingParts. It uploads only the missing ones. A drop costs at most one part per parallel stream — about 32 MB with four streams of 8 MB — rather than the whole file.

Complete. The client posts its part list. The server asks storage to assemble the object, which fails if anything is missing, then compares the assembled size against the expected_size recorded at init. Only after both checks does it write the files row. The endpoint is idempotent on uploadId: a retry finds status = completed and returns the same fileId (10.4). This matters because completion is precisely the request most likely to be retried — it arrives at the end of a long, unreliable transfer when the client cannot tell whether its request landed.

Cleanup. A lifecycle rule aborts incomplete multipart uploads after seven days. Without it, parts from abandoned uploads are stored and billed forever while appearing in no object listing.

InterviewWhy must file bytes never pass through your application servers?

Memory and process health. Buffering a 5 GB upload in application memory is an immediate out-of-memory kill (9.9.4). Streaming it to disk avoids that but still consumes file descriptors, disk throughput and event-loop time that the process needs for the thousands of small requests it is actually there to serve (3.8.2).

Bandwidth economics. At 1.2 GB/s sustained ingest, proxying means provisioning a fleet of application servers whose entire job is copying input to output. The storage service already owns that capacity and sells it more cheaply, and it does not also run your business logic.

Resumability is already solved by the storage protocol. Multipart upload with per-part etags and a part-listing query is a feature of the storage service. Re-implementing it in your own API means re-implementing part tracking, durable staging of partial data, and cleanup of abandoned parts — three hard problems bought back for nothing.

Failure isolation, which is the reason people forget. With presigned URLs, an application deploy, restart, crash or scale-down does not interrupt a single in-flight upload, because the bytes are flowing between the client and a different system entirely. In a proxying design, every routine deploy kills every large upload in progress, which for a 4 GB file on a mobile connection means it may never complete.

The architectural consequence to state plainly: your API becomes an authority that issues credentials, verifies outcomes and records metadata — small, fast, stateless handlers whose slowest operation is a database write — while the storage layer does the byte transfer it was built for.

InterviewDeduplication by content hash: how it works, what it saves, and what it gives away.

How. The client hashes the file and sends the hash at init. The server looks it up in blobs. On a hit it increments refcount, writes a files row pointing at the existing blob, and returns immediately — an upload that transferred no file bytes. On a miss the normal multipart flow runs, and the storage service computes the real hash as the bytes land.

What it saves. In consumer products a substantial share of stored bytes are duplicates: the same installer, the same viral video, the same photo forwarded through twenty conversations. Twenty to forty per cent of stored bytes is a normal saving, and at an exabyte that is a very large number. It also produces the experience users notice most — re-uploading a large file you already have takes half a second.

What it gives away. If completion is instant, the uploader has learned that this exact file already exists somewhere in the system. That is a testable oracle: generate candidate files, upload them, see which complete instantly. For a leaked document or a file that identifies its holder, that is a genuine disclosure.

Three mitigations with their costs. Scope deduplication per account: no leak, no cross-user saving. Scope per tenant: saving inside an organisation, no leak across organisations, and this is usually the right default for business products. Or keep global deduplication but always transfer the bytes and deduplicate on the server: you keep the storage saving, lose the bandwidth saving and the instant-upload experience, and the timing oracle disappears. Large providers generally chose the third, and knowing that distinguishes saving storage from saving bandwidth.

The integrity rule, separately. The client's hash is a lookup hint and nothing more. It must never be written into blobs as the identity of stored content, because a malicious client could otherwise point a row at content it did not upload, or claim an identity for content that does not match it. The hash of record is always the one the storage service computed from the bytes it received.

StaffA month after launch, storage costs are three times projection and 15% of files fail to download. Diagnose both.

The cost overrun. Three suspects, each diagnosable from a metric.

Abandoned multipart uploads. Parts from interrupted uploads are stored and billed but belong to no completed object, and — this is why the problem persists — they do not appear in a normal object listing. Check the storage service's incomplete-multipart-bytes metric specifically. Without an abort-incomplete lifecycle rule they accumulate indefinitely, and at 100 TB a day of ingest with even a 5% abandonment rate that is 5 TB a day of invisible cost.

Deduplication silently not working. Compare the count of distinct content_hash values against the count of files rows. If that ratio moved toward 1:1, deduplication has stopped happening — usually because a client update changed how the hash is computed, or because a fallback path skips hashing entirely for large files and nobody noticed.

No lifecycle tiering. Everything sitting in the hottest storage class. Cold data untouched for 90 days should move down automatically, and at this scale that is normally the single largest saving available.

The unreadable files. Fifteen per cent is far too high to be storage-side loss — eleven nines means essentially never — so the fault is in the metadata, which is pointing at objects that do not exist. The most likely cause is that the completion path wrote the files row before verifying the parts, or trusted the client's part list, so uploads that failed partway through produced metadata for objects that were never assembled.

Confirm it cheaply: sample files rows and issue a HEAD against each storage_key. Then group the failure rate by created_at — the date the failures start will point directly at the deploy that introduced the bug, which turns a debate into a diff.

The fixes. Enforce the write order — storage verified, then metadata written. Make completion verification mandatory and idempotent. Run a two-way reconciliation: dangling metadata rows get marked broken and surfaced to the user honestly with an offer to re-upload, and orphan blobs are collected after a grace period long enough that an in-flight upload is never destroyed.

Then make it permanent. A daily reconciliation report with three monitored numbers — orphan blob count, dangling metadata count, incomplete-upload bytes — and an alarm on each. In a two-store design, drift is not an incident that happens once; it is a background condition that must be continuously measured, because nothing about it shows up in a latency graph or an error rate (10.4).

Flashcards

FlashThe two-store split, and why

Blobs → object storage (1 EB). Metadata → partitioned database (5 TB). Separated because the numbers differ by six orders of magnitude. files by owner_id, blobs by content_hash. The API never touches bytes.

FlashResumable upload = which three properties?

Independent fixed-size parts (8–16 MB) · per-part etags recorded by the storage service · a query for which parts exist. Plus: client persists uploadId, and complete is idempotent and server-verified.

FlashDeduplication: saving and leak

Known hash ⇒ zero-byte upload, 20–40% of stored bytes saved. Leaks file existence across users via timing — scope per account or tenant, or transfer always and deduplicate server-side. Client hash is a hint, never the identity.

FlashWhere eleven nines come from

Erasure coding k=10 m=4 → 1.4× storage, survives 4 fragment losses · spread across failure domains · continuous scrubbing with fast repair. Durability is a repair-speed number. Tiering is the cost lever.

FlashWrite order between the two stores

Bytes first, metadata second. A crash then leaves an orphan blob (costs money, invisible) rather than a dangling row (breaks a file the user can see). Reconcile in both directions, forever.

FlashThe three numbers on the reconciliation report

Orphan blobs · dangling metadata rows · bytes in incomplete uploads. None of them appear in an error rate or a latency graph, so they need their own alarms.

Scenario Drill

DrillExtend the design into a sync client: files change on multiple devices, offline edits must merge, and bandwidth should be minimised. What carries over, what is added, and what new hard problem appears?

What carries over unchanged. The metadata and blob split, presigned transfers, content-hash deduplication, idempotent completion, and the reconciliation discipline. None of that is affected by files becoming mutable, which is a good sign that the original decomposition was sound.

What has to be added, in three pieces.

Block-level deduplication and delta sync. Hashing whole files is useless once files change, because editing one paragraph changes the whole hash. Instead split each file into content-defined chunks — chunk boundaries chosen by a rolling hash over the content rather than at fixed offsets, so that inserting a sentence near the beginning shifts one chunk boundary instead of every subsequent one — and store the file as an ordered list of chunk hashes. Editing a paragraph of a 50 MB document then uploads a few kilobytes. The blobs table becomes a chunk store with refcounts, and a file becomes a manifest that points at chunks. The bandwidth requirement is met structurally rather than by compression.

A per-user change journal. An ordered, monotonically numbered log of file events — created, modified, moved, deleted — per account. Clients hold a cursor and pull everything since their last position (10.8.1). This replaces "list everything and compare", which does not scale and cannot tell a move from a delete-plus-create, and it gives an offline device a precise resume point no matter how long it was away.

Local state on the client. A small database of known file states — path, chunk manifest, version, local modification time — so the client can detect what changed locally and compute exactly what to push.

The new hard problem: conflict resolution. Two devices editing the same file while both are offline is not an edge case, it is guaranteed to happen weekly. The options, honestly:

Last writer wins silently destroys work. For a user document that is unacceptable, and the reason is not technical — the user has no way to know it happened (10.3 explains why "last" is not even well defined across devices with independent clocks).

Version vectors let you detect true concurrency reliably: each device's counter for a file tells you whether one version descends from the other or whether they genuinely diverged. Detection is the part you can solve correctly. Resolution is not a technical question.

Materialise both versions. For opaque binary content, the only resolution that never loses data and that a non-technical user can understand is to keep both: report.docx and report (conflicted copy from Ana's laptop).docx. Every real sync product does this, and it is not a failure of the algorithm — it is the honest admission that the system cannot know which edit the user wanted. For genuinely mergeable formats, the manifest history gives you a common ancestor, so a three-way merge can be attempted and the conflicted copy kept as the fallback.

A second hard problem worth naming before the interviewer does: moves and deletes. A rename on one device plus an edit on another, or a delete racing an edit, produces situations where "the file" has no single correct outcome. Two rules keep it sane. Treat a move as a metadata operation on a stable file identity, never as delete-plus-create, or every rename becomes a full re-upload and every concurrent edit becomes an orphan. And use tombstones with a retention window rather than immediate deletion, so a late-arriving edit can resurrect the file rather than vanishing into a store that no longer has anywhere to put it.

The sentence for the design document: sync turns a storage system into a distributed-state system. Chunking solves bandwidth, the change journal solves resumption, version vectors solve detection — and everything hard that remains is conflict semantics, which is a product decision shown to the user rather than an algorithm that hides it from them.