10.27 — Capacity & Cost Modelling

Capacity planning and cost modeling are the physics of system design. You can build the most elegant distributed system in the world, but if it bankrupts the company or crumbles under unexpected load, it is a failure. Architecture is fundamentally about resource allocation under constraints.

Capacity Planning: Predicting the Future

At its core, capacity planning answers a simple question: How much hardware do we need?

The ELI5

Imagine you are hosting a dinner party. If you invite 10 people, you need 10 chairs, 10 plates, and enough food for 10. If 50 people show up, people will be standing, plates will run out, and the food will be gone. Capacity planning is figuring out how many chairs to rent and how many pizzas to order before the guests arrive.

The PhD

In distributed systems, capacity planning requires modeling system behavior under stochastic workloads. It involves analyzing resource saturation points across CPU, memory, Disk I/O, and Network I/O. We use historical metrics, projected growth rates, and queuing theory to provision resources that satisfy Service Level Objectives (SLOs) without excessive over-provisioning.

The Math of Capacity

To plan capacity, you must estimate three dimensions: Storage, Bandwidth, and Compute.

1. Storage Estimation Calculate total data generated per day and multiply by the retention period. Formula: Total Storage = (Write QPS) * (Average Size per Write) * (Seconds in a Day) * (Retention Days)

2. Bandwidth Estimation Calculate the data transfer rate in and out of the system. Formula: Ingress Bandwidth = (Write QPS) * (Average Payload Size)Formula: Egress Bandwidth = (Read QPS) * (Average Payload Size)

3. Compute Estimation Compute is harder. It depends on the complexity of the processing. A common heuristic is benchmarking a single node to find its maximum safe QPS (e.g., at 70% CPU utilization). Formula: Total Nodes = (Peak QPS) / (Safe QPS per Node)

Cost Models: The Cloud Bill

The shift from on-premise to cloud transformed hardware from a Capital Expenditure (CapEx - buying servers upfront) to an Operating Expenditure (OpEx - renting servers by the second).

Managed vs. Self-Hosted

When designing a system, you must choose between managed services (e.g., AWS DynamoDB, Confluent Cloud) and self-hosted open-source software (e.g., running Cassandra or Kafka on raw EC2 instances).

Managed vs Self-Hosted Database Decision

The Trade-off: Managed services cost more on the AWS bill but save money on payroll. Self-hosting requires hiring specialized infrastructure engineers. For most startups, managed services are the correct choice until scale dictates otherwise.

The Hidden Killer: Egress Costs

Cloud providers let data in for free but charge heavily to let it out. If your architecture spans multiple availability zones (AZs) or regions, inter-AZ and inter-region data transfer costs can quickly eclipse compute costs.

Finding the Bottleneck

A bottleneck is the component that limits the maximum throughput of the system. According to the Theory of Constraints, improving anything other than the bottleneck is an illusion.

Little's Law

Little's Law is a theorem that determines the average number of items in a stationary queuing system. L = λ * W Where:

• L = Average number of items in the system (Concurrency)
• λ = Average arrival rate (Throughput or QPS)
• W = Average time an item spends in the system (Latency)

If your server can handle 100 concurrent connections (L=100) and each request takes 0.1 seconds (W=0.1), your maximum throughput (λ) is 1000 QPS. If traffic spikes to 2000 QPS, requests will queue, latency will spike, and the system will eventually fail.

TypeScript Example: Bottleneck Simulation

Let's model an order processing pipeline where the database is the bottleneck.

// order-pipeline.ts

interface Order {
  id: string;
  items: string[];
}

class Database {
  private maxConcurrentConnections = 5;
  private currentConnections = 0;

  async saveOrder(order: Order): Promise<void> {
    if (this.currentConnections >= this.maxConcurrentConnections) {
      throw new Error("Database connection pool exhausted. System Bottleneck!");
    }

    this.currentConnections++;
    
    // Simulate DB latency (100ms)
    await new Promise(resolve => setTimeout(resolve, 100));
    
    this.currentConnections--;
  }
}

class OrderService {
  private db = new Database();

  async processOrder(order: Order): Promise<string> {
    try {
      // Fast operations (CPU bound)
      this.validateOrder(order);
      
      // Slow operation (I/O bound) - THE BOTTLENECK
      await this.db.saveOrder(order);
      
      return `Order ${order.id} processed successfully`;
    } catch (error) {
      return `Failed to process ${order.id}: ${(error as Error).message}`;
    }
  }

  private validateOrder(order: Order): void {
    if (order.items.length === 0) throw new Error("Empty order");
  }
}

// Simulation
async function runSimulation() {
  const service = new OrderService();
  const incomingTraffic = 10; // 10 concurrent orders
  
  const requests = Array.from({ length: incomingTraffic }).map((_, i) => 
    service.processOrder({ id: `ORD-${i}`, items: ["apple"] })
  );

  const results = await Promise.allSettled(requests);
  
  let success = 0;
  let failed = 0;
  
  results.forEach(res => {
    if (res.status === "fulfilled" && res.value.includes("successfully")) success++;
    else failed++;
  });
  
  console.log(`Throughput: ${success} successes, ${failed} failures.`);
  // Output: Throughput: 5 successes, 5 failures.
}

runSimulation();

In this scenario, the stateless compute (the Node.js process running the simulation) can scale infinitely, but the stateful layer (the Database) has a hard concurrency limit. The bottleneck dictates the system's absolute capacity. One honest difference from a real system: a real connection pool does not throw when it is full, it makes the sixth request wait for a free connection. That turns the five failures into five slow responses, and under sustained overload the waiting queue grows until requests time out, which is the same failure arriving later. Little's Law above tells you when that queue will start to grow.

What the interviewer will push on

1. "How do you handle a 10x traffic spike?" Do not just say "auto-scaling." The interviewer wants to hear about load shedding, rate limiting, and degrading gracefully. Auto-scaling takes time (minutes to boot VMs); spikes happen in seconds.

2. "We have a strict budget. Where do you cut costs?" They want you to identify tiering data (moving cold data to S3 instead of keeping it in an expensive database), optimizing payload sizes to reduce bandwidth, and using spot instances for stateless, fault-tolerant asynchronous workers.

3. "Where is the bottleneck in the design you just drew?" Always look at the stateful components. Databases, message brokers, or single-point-of-failure leaders are typically the bottlenecks. Calculate the QPS at that specific component to prove it can handle the load.

Recall

• Capacity Planning: Estimating hardware needs based on expected QPS, payload size, and retention periods.
• Cost Models: OpEx (Cloud) vs CapEx (On-prem). Managed services trade higher AWS bills for lower engineering costs.
• Bottlenecks: The component that limits total system throughput. Governed by Little's Law (L = λ * W).
• Network Costs: Egress data and cross-AZ traffic are significant hidden costs in cloud architectures.

Quiz

1. If your system receives 500 QPS, and the average request takes 200ms to process, how many concurrent requests are in the system on average according to Little's Law?
2. Why is auto-scaling not a complete solution for sudden traffic spikes?
3. When deciding between a managed database and hosting it yourself on EC2, what are the primary trade-offs?

Next: Part 11 puts every idea in Part 10 to work, one complete system at a time, starting with the method in 11.0.