13.8 — Infrastructure as Code

A pull request renames a database resource from main to primary. The plan output is fourteen screens long. Someone scrolls to the bottom, sees "1 to add, 1 to destroy", and approves.

# aws_db_instance.main must be replaced
-/+ resource "aws_db_instance" "primary" {

Terraform does not know the resource was renamed. It sees one resource gone from the configuration and a new one appearing, so it destroys the database and creates an empty one.

The tool did exactly what the configuration said. This chapter is about the mental model that makes plans readable, and the small number of guards that make that outcome impossible.

1. Why, and what the alternatives are

Reproducibility — the same configuration produces the same environment, so staging genuinely resembles production.

Review — an infrastructure change goes through a pull request like any other change.

Disaster recovery — rebuilding a region is running an apply, not remembering what was clicked.

Documentation that cannot go stale, because it is the thing that built it.

The categories, and the distinction people blur:

Provisioning creates infrastructure: Terraform, OpenTofu, CloudFormation, Bicep, Pulumi, CDK.

Configuration management configures machines that already exist: Ansible, Chef, Puppet. With immutable infrastructure (Chapter 13.3), this category mostly disappears — you rebuild the image rather than converging a live machine.

Declarative versus imperative is the other axis. Terraform, CloudFormation and Bicep are declarative: describe the end state. Pulumi and the CDKs let you write a real programming language that generates a declarative plan — so you get loops, types and testing, at the cost of an abstraction that can hide what will actually happen.

Terraform's position: multi-cloud, a large provider ecosystem covering far more than cloud infrastructure, and the widest hiring pool. Its licence changed in 2023 to the Business Source License, which prompted the OpenTofu fork under a foundation. They remain compatible in practice, and which you choose is now a governance decision more than a technical one.

2. The workflow

terraform init      # download providers, configure the backend
terraform plan      # compare configuration against state and reality
terraform apply     # execute the plan
terraform destroy   # remove everything in this state

plan is the product. It reads the configuration, reads state, refreshes real resources, and prints the difference. Everything about safe operation reduces to producing a readable plan and actually reading it.

Terraform builds a dependency graph from references — using aws_vpc.main.id inside a subnet creates the edge — and applies in dependency order with independent resources in parallel. Explicit depends_on is only needed for dependencies the references do not express, such as an IAM policy that must exist before a service will accept a resource that uses it.

3. The language, and the two decisions that matter

terraform {
  required_version = "~> 1.9"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.60" }   # (1)
  }
  backend "s3" {                                              # (2)
    bucket = "tfstate-prod"
    key    = "network/terraform.tfstate"
    region = "eu-west-2"
    use_lockfile = true
  }
}

variable "environment" {
  type = string
  validation {                                                # (3)
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging or prod."
  }
}

locals {
  tags = { Environment = var.environment, ManagedBy = "terraform" }   # (4)
}

resource "aws_db_instance" "primary" {
  identifier     = "app-${var.environment}"
  instance_class = var.db_class
  tags           = local.tags

  lifecycle {
    prevent_destroy = true                                    # (5)
    ignore_changes  = [engine_version]                        # (6)
  }
}

(1) Pin provider versions. A provider minor release changing a default has broken more applies than any other single cause. (2) Remote state, section 4. (3) Validation gives a clear error instead of a confusing failure three resources later. (4) Locals for values used repeatedly. (5) prevent_destroy makes the opening incident impossible — the plan errors instead of destroying. (6) ignore_changes for fields another system legitimately manages, such as an auto-applied minor version.

for_each versus count, which is the single most consequential language choice:

count    = length(var.names)                # ✗ indexed: [0], [1], [2]
for_each = toset(var.names)                 # ✓ keyed:   ["api"], ["web"]

With count, resources are addressed by position. Remove the middle name from the list and everything after it shifts down by one — so Terraform plans to destroy and recreate every subsequent resource, because [1] now describes a different thing.

With for_each, resources are addressed by key. Removing one destroys exactly that one and leaves the others untouched.

Use for_each for anything that is a set of distinct things, and count only for "should this exist at all" (count = var.enabled ? 1 : 0). This one rule prevents a large share of accidental destruction.

create_before_destroy builds the replacement before removing the original, which is what you want for anything serving traffic. It requires names to be unique, so pair it with a name_prefix or a random suffix.

Avoid provisioners (remote-exec, local-exec). They run once at creation, are not tracked in state, do not re-run when the script changes, and turn a declarative system into a fragile imperative one. Bake the image (Chapter 13.6.5) or use a configuration tool.

4. State

State is a JSON file mapping your configuration to real resource identifiers, plus attributes read from the provider and metadata.

It exists because a resource has no back-reference to your code. Terraform must remember that aws_db_instance.primary is db-abc123, and it uses the stored attributes to compute a difference without querying everything.

Three properties with real consequences:

State contains secrets in plain text. A generated password, a private key, a connection string — whatever the provider returned. Marking a variable sensitive hides it from output, not from state. So state must be encrypted at rest and access-controlled like a secret store (Chapter 8.6.1).

State must be shared and locked. Local state means one person can apply. Remote state with locking — an S3 bucket with a lock file, an Azure blob lease, or a managed backend — is mandatory for a team, because two concurrent applies corrupt state.

State is authoritative to Terraform. If it says a resource exists and it does not, the plan tries to modify something absent. If a resource exists and state does not know it, Terraform will try to create a duplicate.

The commands for when reality and state diverge:

import {                                   # (1)
  to = aws_db_instance.primary
  id = "db-abc123"
}

moved {                                    # (2)
  from = aws_db_instance.main
  to   = aws_db_instance.primary
}

(1) Adopt an existing resource created by hand or by another tool, without recreating it. (2) moved is the fix for the opening incident: it tells Terraform the resource was renamed, so the plan shows a move rather than a destroy and create. Any rename or refactor of a resource address should come with a moved block.

terraform state rm and state mv edit state directly and are dangerous — rm makes Terraform forget a resource that still exists, orphaning it. Back up the state file first, every time.

5. Structuring environments

Three approaches, and the recommendation is clear.

Workspaces — one configuration, several states. Tempting, and it means production and development share a code path with no isolation, so a mistake affects both, and per-environment differences accumulate as conditionals.

Separate directories per environment, each with its own state and its own backend, sharing modules.

environments/
  prod/     main.tf  terraform.tfvars  backend.tf
  staging/  main.tf  terraform.tfvars  backend.tf
modules/
  network/  database/  service/

This is the recommended default: explicit, isolated blast radius, and a production apply cannot touch staging. The duplication is a few files and is worth it.

Terragrunt reduces that duplication and adds dependency ordering across state files. Useful at scale, and another tool to learn.

Split state by blast radius and by change rate. Networking, data stores and applications in separate states means an application deploy cannot plan a change to the VPC, and a corrupted state file loses one layer rather than everything. A single state file for an entire organisation is slow to plan and terrifying to apply.

6. Modules

A module is a directory of Terraform files with inputs and outputs. Everything is already a module — the root is one.

Write a module when a pattern repeats across environments or teams. Do not write one for a single resource: a module that wraps one resource and passes through its arguments adds a layer and removes flexibility. This "thin wrapper" is the most common module anti-pattern.

Version modules and pin the version. source = "git::...?ref=v1.4.0" or a registry version constraint. An unpinned module is an unannounced change to every consumer.

Design inputs as a small set of meaningful decisions, not a passthrough of every argument, with validation and sensible defaults. A module with sixty variables has not abstracted anything.

Public registry modules are useful and are third-party code with your cloud permissions (Chapter 8.6.2). Read what they create, and pin.

7. Testing and policy

Fast checks in order: terraform fmt -check, terraform validate, then tflint for provider-specific mistakes.

Security scanning — Checkov, tfsec, Trivy — catches public buckets, unencrypted volumes and open security groups before they exist, which is the whole argument for infrastructure as code as a security control.

Policy as code — OPA or Sentinel — enforces organisational rules against the plan: no untagged resources, no instance types above a size, no public ingress on port 22. Evaluating the plan rather than the code is what makes it reliable, because it sees the resolved values.

terraform test runs real applies against a scratch environment and asserts outputs, which is the only way to test a module's actual behaviour. Slow, and worth it for modules many teams depend on.

8. Running it in a pipeline

On a pull request: fmt, validate, lint, security scan, then plan with the output posted as a comment. The plan is the review artefact — reviewing HCL without the plan is reviewing intent, not effect.

On merge: apply, using the saved plan file from the pull request so what was reviewed is what runs. An apply that re-plans can execute something nobody reviewed.

Require approval for production, with an environment gate (Chapter 13.7).

Run a scheduled plan against every environment to detect drift. A non-empty plan means someone changed something in the console — and finding that on a Monday is far better than discovering it inside an unrelated apply.

Never auto-apply to production from a merge without a human, unless your policy checks and test coverage genuinely justify it. The cost of a wrong infrastructure apply is not comparable to a wrong application deploy, because there is often nothing to roll back to.

Credentials via OIDC federation (Chapter 13.7), never stored keys.

9. Reading a plan

The skill this chapter exists to build.

  + create        new resource
  - destroy       removed — the one to read carefully
  ~ update        changed in place, no interruption
-/+ replace       DESTROY and recreate — read the reason

Terraform prints why a replacement is happening — # forces replacement next to the attribute. That line is the most important text in any plan, and it is what distinguishes a rename from a rebuild.

Read the summary and then find every destroy and every replace. For each, ask: is this intended, does it hold data, and does it interrupt traffic. Then read the plan header for changes outside Terraform, which is how you learn someone edited something by hand.

And the rule that prevents the opening incident, stated plainly: prevent_destroy on every resource that holds data. A database, an object storage bucket, a key vault. It converts a catastrophic apply into an error message, and the error message costs a minute.

What the interviewer will push on

"What is Terraform state and why does it matter?" A map from configuration addresses to real resource identifiers, plus cached attributes. It matters because it contains secrets in plain text, it must be remote and locked for a team, and it is authoritative — Terraform will recreate a resource it has forgotten and try to modify one that no longer exists.

"count or for_each?" for_each for a set of distinct things, because resources are keyed rather than indexed — removing one from a count list shifts every subsequent index and plans to destroy and recreate them all. count only for conditional existence.

"A rename is planning to destroy a database. What do you do?" Add a moved block so Terraform knows the address changed, and put prevent_destroy on the resource so this class of mistake errors instead of executing. Then generalise: any resource address refactor ships with a moved block.

"How do you structure environments?" Separate directories with separate state and shared modules, not workspaces — because workspaces share a code path with no isolation. Split state by blast radius and change rate, so an application apply cannot alter the network and one corrupt state file loses one layer.

"How do you run Terraform in CI safely?" Plan on the pull request and post it as the review artefact, apply the saved plan file so what ran is what was reviewed, environment approval for production, OIDC for credentials, and a scheduled drift-detection plan. Naming the saved-plan detail is the tell — re-planning at apply time can execute something unreviewed.

"What does infrastructure as code give you that clicking does not?" Review, reproducibility, disaster recovery, and security checks before the resource exists — a public bucket caught in a pull request never existed. That last one is the strongest argument and the one people usually miss.

One thing to volunteer: point out that sensitive = true hides a value from console output and does not remove it from state, so state files must be encrypted and access-controlled like a secret store. It is a widespread misunderstanding, and it means a state bucket with loose permissions is a credential store with loose permissions.

Recall

• The plan is the product. Read every - destroy and -/+ replace, and the # forces replacement line is the most important text in any plan.
• prevent_destroy on everything that holds data, and a moved block for every resource rename — together they make "the rename destroyed the database" impossible.
• for_each keys resources; count indexes them. Removing an item from a count list shifts every later index and plans to recreate them all. count is for conditional existence only.
• State maps configuration to real resources and contains secrets in plain text. sensitive = true hides output, not state. Remote backend with locking is mandatory for a team, and state is authoritative — import adopts, moved renames, state rm orphans.
• Separate directories per environment with separate state, not workspaces. Split state by blast radius and change rate; one state for an organisation is slow and terrifying.
• Pin provider and module versions. Write a module when a pattern repeats; a module wrapping one resource is the common anti-pattern. Registry modules are third-party code with your cloud permissions.
• Policy as code evaluates the plan, not the source, so it sees resolved values. Security scanning catches a public bucket before it exists — the strongest argument for infrastructure as code.
• Pipeline: plan on the pull request as the review artefact, apply the saved plan file, environment approval for production, OIDC credentials, and a scheduled drift-detection plan.

Self-test: Why does removing one item from a count list recreate several resources? · What exactly is in state, and why must it be treated as a secret? · Which two settings would have prevented the opening incident? · Why apply a saved plan rather than re-planning? · Why are workspaces the wrong way to separate production? · What does a scheduled plan detect?

Next: 13.9 covers the estate most large organisations actually run — the on-premises stack layer by layer, integration middleware, and the Microsoft data platform that a great deal of the world's reporting still runs on.