13.3 — Compute: Machines, Containers and Functions

A function generates a report and returns it. It works in development against test data. In production it fails:

Response payload size exceeded maximum allowed payload size (6291456 bytes)

Six megabytes. The report is 9 MB, and there is no configuration to raise it. The fix is architectural: write the report to object storage and return a pre-signed URL — which is a better design anyway, and it had to be discovered by hitting a wall.

Serverless limits are not tuning knobs, they are shape constraints, and the same is true one level down: an instance family, a container's memory limit and a scaling policy all decide what your application can be. This page is that ladder.

1. The ladder

Level	You manage	Scales in	Pay for
Virtual machine	OS, runtime, app	Minutes	Uptime
Container on your cluster	Cluster, image, app	Seconds–minutes	Node uptime
Managed container (Cloud Run, Fargate, Container Apps)	Image, app	Seconds	Container time, can be zero
Function (Lambda, Functions)	Code	Milliseconds	Execution time

Each step removes work and adds constraint. The right question is not "which is most modern" but which constraints you can live with, and whether the work removed is work you were doing badly.

2. Virtual machines

Instance families are the first choice, and picking wrongly is the most common waste.

• General purpose — balanced. The default when you do not know.
• Compute optimised — more CPU per unit of memory. Batch processing, encoding, web serving.
• Memory optimised — databases, caches, in-memory analytics.
• Storage optimised — high local IOPS, for data stores.
• Accelerated — GPUs and other accelerators (Chapter 12.9).

Right-size from measurement, not from the previous machine. The common pattern is a lift-and-shift that copies the on-premises specification, and the on-premises specification was sized for a three-year peak with headroom. Look at actual CPU and memory over a fortnight, then choose.

Burstable instances deserve their own warning. The cheap t-family accumulates CPU credits while idle and spends them when busy. Run above the baseline continuously and credits exhaust and the instance is throttled to a fraction of a core — a permanent, cliff-edged slowdown with no code change, exactly like the storage burst credits in Chapter 13.1. Excellent for genuinely spiky low-average workloads, wrong for anything with steady load.

Images and configuration. Two philosophies:

• A golden image — bake the application and dependencies into a machine image. Boots fast, identical everywhere, and rebuilding is a pipeline step.
• Configuration at boot — a plain image plus a configuration tool. Flexible, slower to start, and drift accumulates: machines built at different times end up different.

Prefer baked images and immutable infrastructure — never modify a running instance, replace it. This is what makes a fleet reproducible, and it is the same argument as containers, applied to virtual machines.

Auto-scaling groups maintain a count of healthy instances, replace failures, spread across zones, and scale on a metric. The parts people get wrong: scale-out should be fast and scale-in slow (a cooldown), health checks should test the application rather than the machine, and a lifecycle hook is needed for graceful shutdown — draining connections before termination (Chapter 9.9.7).

3. Serverless functions, precisely

The execution model: an event arrives, the platform finds a warm execution environment or creates one, runs your handler, and keeps the environment warm for a while in case another event arrives.

One environment handles one request at a time. This is the single most important property and it is repeatedly misunderstood: concurrency is achieved by creating more environments, not by threading inside one. So 100 simultaneous requests means 100 environments — and 100 database connections, which is section 5's problem.

Cold starts. When no warm environment exists, the platform must download your package, start the runtime and run your initialisation code. Typical costs: tens of milliseconds for a small package on a fast-starting runtime, hundreds of milliseconds to a few seconds for a large one on a JVM or .NET runtime.

What actually reduces them:

• Smaller deployment packages. Fewer dependencies, tree-shaken, no unnecessary assets.
• Faster runtimes for latency-sensitive paths — Node, Python, Go and Rust start much faster than JVM-based ones.
• Do expensive work outside the handler, in the initialisation phase, so it is reused across invocations on the same environment.
• Provisioned concurrency — pay to keep N environments warm. It works and it removes the main financial argument for functions, so use it on the latency-sensitive path only.
• Snapshot-based starts (SnapStart and equivalents) — take a memory snapshot after initialisation and restore it, which largely removes the JVM penalty. The caveat matters: anything unique per instance — a random seed, a cached credential, an open connection — is captured in the snapshot and must be re-initialised explicitly.

"Keep-warm" pings are folklore worth dropping. They keep one environment warm and do nothing for the eleventh concurrent request, which is where cold starts actually hurt.

4. The limits that decide the design

These are the constraints to check before choosing functions, not after:

• Execution time — 15 minutes maximum on the main platforms. Longer work needs a container or a step-function-style workflow.
• Payload — 6 MB synchronous, 256 KB asynchronous on Lambda. Return a reference, not a blob, which is the opening story.
• Package size — around 50 MB zipped, 250 MB unzipped; container images allow far more and start more slowly.
• Temporary disk — a few hundred megabytes in /tmp, and it may persist between invocations on the same environment, which is a data-leak risk if you write user data there and do not clean up.
• Memory — configurable, and on most platforms CPU is allocated proportionally to memory. So raising memory for a CPU-bound function can make it cheaper, because it finishes proportionally faster. This is genuinely counter-intuitive and worth measuring, not assuming.
• Concurrency — an account-level ceiling and a burst rate. A sudden spike beyond the burst rate is throttled, which is a real availability limit.
• No local state between invocations, and no guarantee the next request reaches the same environment.

5. Functions and databases

The classic incompatibility. Each environment opens its own database connection, and a spike to 500 concurrent executions opens 500 connections — which exhausts a database sized per Chapter 7.2.4's arithmetic, where the correct total is closer to twenty.

Three answers:

A connection proxy — RDS Proxy or PgBouncer (Chapter 7.2.4) — multiplexes many client connections onto few real ones. This is the standard fix.

An HTTP data API — the database exposes a request-per-query interface with no persistent connection. Higher per-query latency, no connection problem at all.

A different data store — a request-based service such as DynamoDB has no connection concept and fits the model natively. This is why serverless architectures gravitate to serverless databases, and it is a coupling worth noticing rather than stumbling into.

Also: initialise the connection outside the handler so it is reused across invocations on a warm environment, and handle the connection being dead after an idle period.

6. Serverless containers: the middle that usually wins

Cloud Run, Fargate and Container Apps run a container image, scale to zero, and bill per second of use.

What that combination gives you:

• No cold-start penalty from a proprietary packaging format — it is your container, with your runtime.
• No 15-minute limit, no 6 MB payload limit.
• Multiple concurrent requests per instance on some platforms, which fixes the connection problem directly.
• Portability — the same image runs on your machine, on Kubernetes, or here.

The cost is a slower scale from zero than a function, and a slightly higher operational floor.

For most teams building an HTTP service, this is the right default. It removes the cluster management that Kubernetes imposes (Chapter 13.6.8) while keeping the container's portability and avoiding the function model's constraints. Reach for functions when the workload is genuinely event-driven and bursty, and for Kubernetes when you need what Kubernetes actually provides.

7. When serverless is cheaper, and when it is not

The cost model is per request plus per gigabyte-second of execution. The crossover is about utilisation.

Serverless is dramatically cheaper for spiky, low-average traffic. An endpoint called 10,000 times a day costs pennies as a function and needs an always-on instance otherwise. This is the real win and it is large.

It is more expensive at sustained high volume. A service handling steady thousands of requests per second on an always-busy instance is cheaper on a reserved instance, and the multiple can be several times. Compute the crossover with your own numbers — requests per month, average duration, memory — because the answer varies by an order of magnitude between workloads.

And the costs that are not on the invoice:

• Local development and testing are harder; emulators are approximate.
• Observability is different — many short-lived executions rather than a process you can attach to. Distributed tracing (Chapter 10.10) stops being optional.
• Vendor coupling is high: the event shapes, the permission model and the deployment format are provider-specific.
• Debugging a distributed set of functions is genuinely harder than debugging one service.

The pragmatic position: functions for glue, events and spiky work; a container service for anything with a steady request rate.

8. Event-driven glue

Serverless architecture is mostly not functions — it is the services between them.

Queues for work that must not be lost, with a dead-letter queue for what fails repeatedly (Chapter 10.8.1).

Event buses for publish-subscribe with routing rules, so producers do not know their consumers.

Object storage and database change events as triggers, so a file upload or a row change starts work with no polling.

Managed workflow engines — Step Functions, Durable Functions, Logic Apps — for orchestration: sequences, branches, parallel work, retries with backoff, timeouts, and waiting for hours or days without holding a process open. This is the correct home for the saga pattern (Chapter 10.8.4) and for anything long-running, because it removes the 15-minute limit by making waiting a state rather than a process.

Two rules that keep event-driven systems sane. Make every consumer idempotent — at-least-once delivery is the norm, so duplicates will arrive (Chapter 10.4). And give every flow a dead-letter path with an alert; the characteristic failure of event-driven systems is silent loss, where a message fails, retries, and disappears with nobody watching.

9. Choosing

Situation	Choose
Steady HTTP service	Managed container
Spiky or event-driven work	Functions
Long-running (> 15 min)	Container or workflow engine
Needs specific OS, kernel or hardware	Virtual machine
Many services, complex networking, a platform team	Kubernetes (Chapter 13.6.8)
Legacy application, unchanged	Virtual machine — lift and shift
Batch, interruptible	Spot instances or spot containers

And the ordering advice: start at the highest level that fits, and move down only when a constraint forces you. Teams that start on Kubernetes because it is where they expect to end up spend their first year operating a cluster instead of shipping.

What the interviewer will push on

"When would you not use serverless functions?" Steady high volume (cost), work over 15 minutes, large payloads, latency-sensitive paths where cold starts matter, and anything needing a persistent connection pool. Then the strong version: the constraints are shape constraints, not knobs — a 9 MB response cannot be configured away, it must be redesigned as a reference.

"Why do functions cause database problems?" One environment serves one request, so concurrency means more environments and therefore more connections — 500 concurrent executions against a database sized for 20. Fix with a connection proxy, an HTTP data API, or a request-based store. Naming the one-request-per-environment model is what shows you understand it rather than repeating a warning.

"How do you reduce cold starts?" Smaller packages, faster runtimes, expensive work in initialisation, provisioned concurrency for the latency-sensitive path, and snapshot starts — with the caveat that a snapshot captures anything unique per instance, so seeds and credentials must be re-initialised. Then dismiss keep-warm pings: they warm one environment and do nothing for concurrent load.

"Increasing a function's memory made it cheaper. Why?" CPU is allocated proportionally to memory, so a CPU-bound function finishes proportionally faster and the gigabyte-seconds fall. It is counter-intuitive, it is real, and it is why you measure the cost curve rather than assuming more memory costs more.

"Kubernetes, managed containers or functions?" Start at the highest level that fits and move down when a constraint forces it. Managed containers are the right default for an HTTP service; functions for spiky event-driven glue; Kubernetes when you genuinely need it and have someone to operate it. Teams that start on Kubernetes spend their first year operating a cluster.

"Your instances got slow after a month with no deploy." Burstable instance credits exhausted, or storage burst credits, or the working set outgrowing memory. All three are gradual, none has a code change, and all three look like an application regression.

One thing to volunteer: mention that a function's /tmp can persist between invocations on the same environment, so user data written there is visible to the next request that lands on it. It is a real cross-request data-leak path that almost nobody accounts for, and the fix is simply to clean up.

Recall

• The ladder — virtual machine → container → managed container → function — removes work and adds constraint at each step. Start at the highest level that fits and move down only when a constraint forces it.
• Burstable instances throttle to a fraction of a core when credits run out, which is the "slow after a month with no deploy" failure. Right-size from two weeks of measurement, and prefer baked images and immutable infrastructure over configuring live machines.
• One function environment serves one request at a time — concurrency means more environments, hence more database connections. Fix with a connection proxy, an HTTP data API, or a request-based store.
• Cold starts: smaller packages, faster runtimes, work in initialisation, provisioned concurrency, and snapshot starts — which capture per-instance state such as seeds and credentials. Keep-warm pings warm one environment and do nothing for concurrency.
• Shape constraints, not knobs: 15 minutes, 6 MB synchronous payload, ~250 MB unzipped package, limited /tmp that may persist across invocations, and account-level concurrency with a burst ceiling.
• CPU scales with configured memory, so raising memory on a CPU-bound function can reduce cost. Measure the curve.
• Managed containers are the right default for a steady HTTP service: scale to zero, your image, no 15-minute or payload limits, and multiple concurrent requests per instance.
• Serverless is much cheaper for spiky low-average load and more expensive at sustained volume. Managed workflow engines are the correct home for long-running orchestration and sagas, and every consumer needs idempotency plus a dead-letter path with an alert.

Self-test: Why can't a 9 MB response be fixed with configuration? · What exactly makes functions open too many database connections? · What does a snapshot-based cold start silently capture? · Why might more memory be cheaper? · What does a managed container give you that a function does not? · Which failure looks like an application regression but is a credit balance?

Next: 13.4 covers where the data sits and how it reaches users — object storage internals, CDN mechanics, and why content served from an edge in Sydney is fast in Los Angeles.