13.4 — Object Storage, CDNs and the Edge

An upload works for every test file and fails on a 6 GB video:

EntityTooLarge: Your proposed upload exceeds the maximum allowed size

A single PUT to object storage is capped at 5 GB. The object itself can be five terabytes — you just cannot send it in one request. The fix is a multipart upload, which is also resumable, parallel and faster.

Object storage looks like a filesystem and is not one, and most of the surprises come from that gap. This page is what it actually is, and then how bytes get from it to a user on the other side of the world.

1. Object storage is a flat key-value store

There are no directories. A bucket holds objects, and an object has a key — a string. photos/2026/08/cat.jpg is one key with slashes in it, not three nested folders.

The consoles show folders by splitting on / and grouping, which is a display convenience. Several consequences follow directly:

There is no rename. Renaming means copying to a new key and deleting the old one. "Renaming a folder" is a copy and delete of every object under that prefix, which on a million objects is a job, not an operation.

There is no append, and no partial write. You replace an object entirely. A log file cannot be appended to.

Listing is a scan of a prefix, not a directory read, and it is paginated. Listing a bucket with ten million objects to find one is a mistake — derive the key or keep an index in a database.

Consistency has improved and is worth stating precisely. Object reads have been strongly consistent since 2020 on the major platforms: write an object and the next read returns it. Listing may still lag, so a workflow that writes then immediately lists to discover what it wrote can miss objects. Pass the keys forward rather than re-listing.

2. Durability, availability, and what neither protects

Durability — will the bytes still be there. Providers quote eleven nines, achieved by erasure coding across multiple facilities: the object is split into fragments with parity, spread across zones, so several failures lose nothing.

Availability — can you reach it right now. Quoted around three to four nines, and it is a much smaller number than durability. They are different questions and the marketing conflates them.

And neither protects you from yourself. Deletion, overwriting, ransomware and a bad script are all perfectly durable operations. Three controls address that:

• Versioning — every overwrite and delete keeps the previous version. A delete becomes a marker, and recovery is possible.
• Object lock — write-once-read-many for a retention period, so not even an administrator can delete it. This is the control that survives a compromised account, and it is the answer to ransomware on backups.
• Cross-account or cross-region replication, so a compromised account does not hold the only copy.

Versioning has a cost people forget: old versions are charged. Without a lifecycle rule expiring them, a bucket with frequently rewritten objects grows without limit, and the bill is invisible because the current listing looks small.

3. Storage classes

Class	For	Retrieval
Standard	Frequently accessed	Immediate
Infrequent access	Monthly-ish	Immediate, per-GB retrieval fee
One-zone infrequent	Reproducible data	Immediate, lost if the zone is lost
Archive / Glacier	Rare access	Minutes to hours
Deep archive	Compliance, years	Up to 12 hours

The trap is the retrieval fee and the retrieval time. Archive tiers are extremely cheap to store and expensive and slow to read. A dataset moved to deep archive by a lifecycle rule and then needed for a project costs far more to retrieve than it saved, and the restore takes half a day. Move data down a tier based on measured access, not on age alone.

Intelligent tiering moves objects between tiers automatically based on observed access, for a small monitoring fee per object. It is the right default when access patterns are unknown, and the per-object fee makes it wrong for very many tiny objects.

Lifecycle rules should always include one that most people omit: expire incomplete multipart uploads. A failed 6 GB upload leaves its parts behind, charged, invisible in a normal listing, forever. This is one of the most common silent cloud costs, and it is a three-line rule.

4. Performance

Request rate scales with key prefix. The partitioning is by key, so thousands of requests per second per prefix are supported, and spreading keys across prefixes multiplies it. Sequential keys — a timestamp prefix — concentrate load on one partition, which is the same hot-partition problem as Chapter 7.5.2. Put the varying part early: a7f3/2026-08-02/... rather than 2026-08-02/a7f3/....

Multipart upload splits an object into parts uploaded in parallel and reassembled server-side. Required above 5 GB, and worth using above about 100 MB for the parallelism and the ability to retry one failed part rather than the whole transfer.

Byte-range requests fetch part of an object — how a video player seeks, and how a Parquet reader fetches only the column chunks it needs (Chapter 7.8.1) without downloading the file.

Server-side filtering — running a SELECT against an object and returning only matching rows — moves the work to the storage layer and cuts transfer dramatically for analytical access.

Transfer acceleration routes uploads over the provider's backbone from a nearby edge rather than across the public internet, which helps for large uploads from distant regions.

5. Pre-signed URLs

The pattern that should be the default for user uploads and downloads.

Instead of proxying bytes through your application, generate a time-limited signed URL and let the client talk to storage directly.

const url = await getSignedUrl(s3, new PutObjectCommand({
  Bucket: 'uploads', Key: `u/${userId}/${crypto.randomUUID()}`,   // (1)
  ContentType: 'image/jpeg',
}), { expiresIn: 300 });                                          // (2)

(1) Your server chooses the key, so the client cannot write outside its own space — the path-traversal defence from Chapter 8.5.1 applied to storage. (2) Five minutes, because a signed URL is a bearer credential: anyone who obtains it has that access until it expires.

Why this matters: your application never handles the bytes, so it needs no bandwidth, no memory and no timeout for a 5 GB file. Chapter 6.8.2's chunked resumable upload builds on exactly this.

Two constraints worth knowing. A signed PUT cannot limit the size — a client can upload a terabyte — so use a signed POST policy where you can set a content-length range, or check the size afterwards from the storage event. And enforce the content type, or you get an HTML file where you expected an image (Chapter 8.5.3).

6. Security

Block public access at the account level. Every headline object-storage leak is a bucket someone made public. The account-level block overrides per-bucket settings, which is what makes it effective, and it should be on by default with exceptions granted deliberately.

Bucket policies over ACLs. Object ACLs are legacy and now disabled by default; policies are clearer and auditable.

Encryption — server-side with provider-managed keys is on by default; with your own key management keys it adds an audit trail of every decryption and lets you revoke access by revoking key permissions (Chapter 8.6.1). Client-side encryption is the only option that keeps the provider from being able to read the data, at the cost of losing server-side features.

VPC endpoints so traffic never traverses the internet, and a bucket policy that only accepts requests from that endpoint (Chapter 5.10).

Access logging to a separate bucket, in a separate account for anything sensitive.

7. CDN topology

Chapter 10.14.3 covers caching semantics — Cache-Control, cache keys, invalidation. This is the topology: why a CDN is fast, and why it is a hierarchy rather than a single layer.

Edge locations (points of presence) are hundreds of small facilities holding caches close to users. Anycast routing announces the same address from all of them, so the network delivers a request to the nearest one automatically (Chapter 5.3.2).

Regional edge caches sit between the edges and your origin — larger, fewer, with much more storage and longer retention.

The hierarchy is what makes the hit ratio work, and this answers the question people ask about content that "should" be far away:

Content published in Sydney is requested by a user in Los Angeles. The first request misses at the Los Angeles edge, misses at the regional cache, and fetches from Sydney — slow, once. It is now cached at the Los Angeles edge for the whole west coast. When it falls out of the small edge cache, the next request is served by the regional cache rather than crossing the Pacific again.

So the origin sees a tiny fraction of the traffic, and the second and every subsequent user gets an edge-speed response regardless of where the content originated. Distance to the origin costs you one request, not every request.

Origin shield adds one more tier: a single designated cache that all others fetch through, so a popular object is requested from your origin once rather than once per region — which is what protects an origin during a traffic spike.

What actually makes this fast is more than proximity: the connection is terminated at the edge, so the TLS handshake's round trips (Chapter 5.7) are local; the long-haul leg to the origin runs over the provider's optimised backbone with warm connections; and modern protocols are terminated close to the user.

Two operational rules. Cache keys decide the hit ratio — a key including a query string that varies per user, or a Vary on User-Agent, fragments the cache into uselessness. And use versioned URLs rather than invalidation: a content-hashed filename can be cached for a year and never needs purging, while invalidation is slow, rate-limited and often charged.

8. The Cloudflare story

Worth knowing because it explains a business model that changed how the internet is served.

It launched in 2010 out of Project Honey Pot, a system for tracking spammers and their sources — the founders' insight being that if you can see attacks across many sites, you can defend all of them better than any one site can defend itself.

The strategy was a genuinely free tier on an anycast network. The free tier looked like generosity and was a network effect: many sites meant many points of presence, which meant a better network, which attracted paying customers — and it meant visibility into attack traffic across a large share of the web.

The DDoS model follows from anycast. An attack against one customer is spread across every location announcing that address, so it is absorbed by the aggregate capacity rather than concentrated on one target. Capacity is shared, so the largest attacks became survivable for sites that could never have provisioned for them.

Workers put compute at the edge using isolates rather than containers (a V8 isolate per request, Chapter 3.6.9), which is why start-up is sub-millisecond where a container measured in hundreds.

And the outages are the more useful part, because each taught something general:

• 2019 — a regular expression deployed globally caused catastrophic backtracking (Chapter 3.6.10) and consumed CPU across the fleet, taking the network down. Lesson: a configuration change deployed everywhere at once is as dangerous as a code deploy, and needs the same staged rollout.
• 2020 — a backbone routing change caused a large traffic loss. Lesson: the network is software and inherits software's failure modes.
• 2022 — a change during a network upgrade took out data centres serving a large share of traffic. Lesson: the blast radius of a central control plane is the whole product.

The general point for an architect: a CDN is a dependency with your availability in its hands. That is usually a good trade — their availability is better than yours — and it is a decision to make consciously, with a plan for what happens when it fails.

9. Choosing a storage type

Need	Use
A disk for one machine	Block storage
A shared filesystem for several machines	File storage
Files served to users, backups, data lake	Object storage
Scratch space, fastest possible	Local instance storage (lost on stop)
Static assets and media to users	Object storage behind a CDN

And the default worth stating: put static content in object storage behind a CDN, and never serve it from your application. It is cheaper, faster, more available, and it removes an entire class of load from the service you actually have to operate.

What the interviewer will push on

"How is object storage different from a filesystem?" Flat key-value: no directories, no rename (copy plus delete), no append or partial write, and listing is a paginated prefix scan. The consequence that matters is that "rename a folder" is a job over every object, and that you should derive keys or index them rather than listing to find things.

"Eleven nines of durability — what does that protect against?" Hardware and facility loss, through erasure coding across zones. Not deletion, not overwriting, not ransomware, not a bad script. Those need versioning, object lock (which even an administrator cannot bypass) and cross-account replication.

"How would you handle large user uploads?" Pre-signed URL direct to storage, with your server choosing the key so the client cannot write outside its space, a short expiry, multipart above ~100 MB, and a storage event to trigger processing. Then the two constraints: a signed PUT cannot limit size, and content type must be enforced.

"Why is content from a distant origin still fast on a CDN?" Only the first request pays the distance. After that it is cached at the local edge, and when it ages out of the edge it is served by a regional cache rather than crossing the ocean again. The hierarchy is the answer, and origin shield reduces origin fetches to one globally.

"What silently costs money in object storage?" Incomplete multipart uploads (charged, invisible in a listing), old versions with no expiry rule, small-object request charges, and archive retrieval fees on data a lifecycle rule moved down. Naming incomplete multiparts is the tell — almost nobody mentions it and almost every large bucket has them.

"What do you lose by depending on a CDN?" Their availability becomes yours, and a global configuration change on their side is a global change to your service. It is usually a good trade, and it needs a conscious decision, an origin that can serve directly, and a plan for the day it fails.

One thing to volunteer: point out that a CDN configuration change is deployed globally in seconds, which makes it more dangerous than a code deploy — Cloudflare's 2019 regular-expression outage is the clearest case. Staged rollout for configuration, not just for code, is the lesson worth carrying into your own edge rules.

Recall

• Object storage is a flat key-value store: no directories, no rename (copy plus delete), no append, and listing is a paginated prefix scan. Reads are strongly consistent; listings can lag, so pass keys forward rather than re-listing.
• Durability ≠ availability, and neither protects against deletion or ransomware. Versioning, object lock (an administrator cannot delete), and cross-account replication do — and old versions are charged, so expire them.
• Storage classes trade cheap storage for retrieval fees and retrieval time — archive restores take hours. Move tiers on measured access, not age. Always add a lifecycle rule expiring incomplete multipart uploads — charged and invisible.
• Request rate scales per key prefix, so sequential timestamp prefixes create a hot partition. Multipart above 5 GB (required) and ~100 MB (worthwhile); byte-range requests power seeking and columnar reads.
• Pre-signed URLs keep bytes out of your application. Your server chooses the key; expiry is short because the URL is a bearer credential; a signed PUT cannot limit size — use a POST policy or check afterwards.
• Block public access at the account level, policies over ACLs, encryption with your own keys for an audit trail, VPC endpoints, and access logs in a separate account.
• CDN is a hierarchy: anycast to hundreds of edges, then regional edge caches, then optionally an origin shield so a popular object is fetched from origin once globally. Distance to origin costs one request, not every request. Cache keys decide hit ratio; prefer versioned URLs over invalidation.
• Cloudflare's model was a free tier on an anycast network, absorbing attacks with shared capacity. Its outages taught that a global configuration change is as dangerous as a code deploy and needs staged rollout.

Self-test: Why is renaming a prefix an expensive job? · What does eleven nines not protect you from? · Which lifecycle rule is almost always missing? · Why does a sequential key prefix hurt throughput? · Who chooses the key in a pre-signed upload, and why? · Why is only the first request slow for distant content?

Next: 13.5 goes into one provider in depth — Azure's identity-centred model, the resource hierarchy, the Graph API and its cost model, and why there are two keys for everything.