13.7 — Pipelines: From Commit to Production

A release fails in production. The team rebuilds from the same commit to investigate, and the new build behaves differently — a transitive dependency published a patch release in between.

The artefact that was tested is not the artefact that was deployed, and that is the failure this whole chapter is organised around: build once, then promote that exact artefact through every environment.

1. The words, used precisely

Continuous integration — every commit is merged to the main branch frequently and verified automatically. The "integration" is the point: the alternative is long-lived branches that diverge and merge painfully.

Continuous delivery — every commit that passes is ready to deploy, and deploying is a decision someone makes.

Continuous deployment — every commit that passes is deployed, with no human step.

Most organisations do continuous delivery and call it deployment. The distinction matters because continuous deployment requires a level of automated confidence — tests, canaries, automated rollback — that has to be built first.

2. Build once, promote many

Two pipelines with different jobs, and conflating them is the most common structural mistake.

The build pipeline runs on every commit: compile, verify, package, publish an immutable versioned artefact. It knows nothing about environments.

The release pipeline takes an existing artefact and deploys it to an environment, with approvals and environment-specific configuration.

commit ──▶ build ──▶ artefact:sha-0e1b ──▶ deploy dev ──▶ deploy staging ──▶ deploy prod
                     (built once)            (same bytes, every time)

Why this ordering is not negotiable: if you rebuild per environment you have not tested what you shipped — different dependency resolutions, different build machines, different timestamps. The artefact is the unit of promotion.

So environment differences must be configuration, not build inputs. One container image, configured at runtime (Chapter 13.6.5). A build argument that differs per environment produces per-environment artefacts, which is the same mistake wearing a disguise.

An artefact repository stores them immutably — a container registry, a package feed — with retention rules and, ideally, signatures verified at deploy (Chapter 8.6.2).

3. Pipeline anatomy

trigger → checkout → restore cache → install → build
        → unit tests → lint & types → security scan
        → package → publish artefact
        → deploy → smoke test → (approval) → promote

Order the pipeline to fail fast and cheap. Lint and type-check take seconds and catch a large share of mistakes; integration tests take minutes. Cheap and broad first, slow and narrow last — a pipeline that runs a twenty-minute suite before a formatting check wastes twenty minutes on a misplaced comma.

Speed is a feature, and the target worth aiming at is ten minutes to feedback. Beyond that, people stop waiting, start context-switching, and batch their commits — which reverses the benefit of continuous integration.

What actually makes pipelines fast:

• Cache dependencies, keyed on the lockfile hash so the key changes exactly when the dependencies do.
• Parallelise by splitting tests across runners — most frameworks support sharding.
• Run only what changed in a monorepo, using the dependency graph.
• Reuse container layers (Chapter 13.6.4) and build with a shared cache.
• Right-size runners. A larger runner that halves a twenty-minute build is usually cheaper than the engineering time it returns.

Ephemeral runners are the right default for security — a fresh environment per job, so nothing leaks between builds. Persistent runners are faster because caches are warm, and they accumulate state and become a shared attack surface.

4. Branching and merging

Trunk-based development — short-lived branches, merged to main within a day or two, with incomplete work hidden behind feature flags. This is what makes continuous integration real, because there is nothing to integrate later.

GitFlow — long-lived develop, release and hotfix branches. Designed for versioned software with scheduled releases, and a poor fit for a continuously deployed service, where it produces large risky merges. Chapter 14.1 covers the mechanics.

Feature flags decouple deploying from releasing. Code ships dark and is enabled per user, per percentage, or by rule. The discipline that keeps this healthy: every flag has an owner and a removal date, because a codebase with two hundred stale flags has an unreasoned number of possible states.

A merge queue solves the semantic conflict: two pull requests each pass against main independently and break when both land. The queue tests them together in order before merging. Worth it as soon as a busy repository starts breaking main.

Required checks and a required review are the enforcement points, and they should be branch protection rules rather than convention.

5. Deploying

Strategy	Downtime	Cost	Rollback
Recreate	Yes	Low	Redeploy
Rolling	No	Low	Roll forward or back
Blue-green	No	2× capacity	Instant — flip back
Canary	No	Slightly more	Shift traffic back
Feature flag	No	None	Toggle

Rolling is the default and what Kubernetes does natively (Chapter 13.6.11).

Blue-green runs a full second environment, switches traffic, and keeps the old one warm. Instant rollback is the reason to pay for it, and it is the right choice when a bad release is expensive and you cannot risk a slow recovery.

Canary sends a small percentage to the new version and watches error rate and latency before proceeding. Progressive delivery is the automated form: shift 5%, evaluate against objectives, shift more, and roll back automatically if the objectives are breached — which is what makes continuous deployment safe enough to do.

Two rules that apply to all of them.

Every deploy must be backward compatible with the version it replaces, because both run simultaneously during a rollout. That includes API shapes, message formats and database schemas.

Database migrations follow expand-and-contract (Chapter 7.2.4) and are deployed separately from and ahead of the code that needs them. A migration bundled into a release cannot be rolled back with the release, and that is how a rollback becomes an outage.

6. Security in the pipeline

Chapter 8.6.2 covers the reasoning; this is the checklist.

Use OIDC federation for cloud credentials. The pipeline presents a signed token proving which repository, branch and workflow it is, and exchanges it for short-lived credentials. No cloud key is stored in CI at all, which removes the highest-value secret in most organisations.

Never expose secrets to pull requests from forks. A contributor's branch can otherwise run arbitrary code with your credentials.

Pin third-party actions by commit SHA, not by tag. Tags are mutable and can be repointed.

Least privilege on the pipeline's own token. Default it to read-only and grant write where needed, per job.

Separate build from deploy privileges, and require approval for changes to deployment workflows.

Sign artefacts at build and verify at deploy, so a compromised registry is not a compromised production.

7. GitHub Actions

name: ci
on:
  push: { branches: [main] }
  pull_request:
permissions:
  contents: read                      # (1)
  id-token: write                     # (2)
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true            # (3)

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix: { node: [20, 22] }      # (4)
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683   # (5)
      - uses: actions/setup-node@v4
        with: { node-version: '${{ matrix.node }}', cache: npm }
      - run: npm ci
      - run: npm run lint && npm test

  deploy:
    needs: test
    if: github.ref == 'refs/heads/main'
    environment: production           # (6)
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy   # (7)
          aws-region: eu-west-2

(1) Least privilege by default, granting only what is needed. (2) Required for OIDC. (3) Cancel superseded runs on the same branch — a large and free saving on a busy repository. (4) A matrix runs the job once per combination in parallel. (5) Pinned by SHA. (6) An environment carries protection rules — required reviewers, wait timers, and environment-scoped secrets. This is where the deployment approval lives. (7) No stored key — the OIDC token is exchanged for a role.

Reusable workflows and composite actions remove duplication across repositories, which matters as soon as you have more than a handful.

Self-hosted runners are needed for private network access or specialised hardware, and they are a security boundary: never run untrusted pull requests on a self-hosted runner with network access to anything you care about.

8. Jenkins

Jenkins predates all of this, and it is still everywhere — because it runs anywhere, has a plugin for everything, and a great deal of institutional pipeline logic lives in it.

Architecture: a controller schedules and holds configuration; agents execute. Never run builds on the controller — a build with controller access is a full compromise of the automation system.

pipeline {
  agent { label 'linux' }                      // (1)
  options { timeout(time: 30, unit: 'MINUTES') }
  environment { REGISTRY = 'registry.example.com' }
  stages {
    stage('Checkout') { steps { checkout scm } }   // (2)
    stage('Build')    { steps { sh 'make build' } }
    stage('Test') {
      steps { sh 'make test' }
      post { always { junit 'reports/*.xml' } }    // (3)
    }
    stage('Deploy') {
      when { branch 'main' }                       // (4)
      steps {
        withCredentials([string(credentialsId: 'deploy-token', variable: 'TOKEN')]) {
          sh 'make deploy'                         // (5)
        }
      }
    }
  }
  post { failure { slackSend(message: "Build failed: ${env.BUILD_URL}") } }
}

(1) Which agent pool. (2) checkout scm is the idiom worth understanding. In a multibranch pipeline, Jenkins discovers branches and pull requests in a repository and creates a job per branch, injecting the source configuration into the scm variable automatically. So checkout scm means "check out the exact repository, branch and commit that triggered this job" — with no URL or branch hard-coded, which is what makes one Jenkinsfile work for every branch. (3) post blocks run regardless of outcome — the place for test reports and cleanup. (4) Conditional stages. (5) Credentials are injected as variables scoped to a block and masked in logs, rather than being in the environment for the whole build.

Shared libraries put common pipeline code in a separate repository, loaded with @Library. This is what stops fifty Jenkinsfiles duplicating the same forty lines, and it is the main thing separating a maintained Jenkins estate from an unmaintainable one.

The honest assessment. Jenkins is powerful, self-hosted, and it requires ongoing care: plugin updates that break, a controller that becomes a single point of failure, and Groovy pipeline code that few people want to maintain. New projects generally choose a hosted service; existing Jenkins estates are usually cheaper to maintain than to migrate.

9. Azure DevOps and GitLab

Azure DevOps Pipelines — YAML with stages → jobs → steps, agent pools (Microsoft-hosted or self-hosted), service connections holding credentials for external systems (and workload identity federation so those are not stored secrets), environments with approvals and gates, and Artifacts feeds for packages. Classic drag-and-drop pipelines still exist; YAML in the repository is the right choice because the pipeline is versioned with the code.

GitLab CI — .gitlab-ci.yml with stages and jobs, an integrated container registry, environments and review apps. Review apps are the notable idea: a full ephemeral deployment per merge request, with a URL, torn down on merge.

The concepts transfer completely. Every one of these has: a trigger, a unit of work on a runner, caching, artefacts, environments with approvals, and secret injection. Learn the model once and the syntax is a lookup.

10. GitOps

The repository is the desired state, and an agent inside the cluster pulls it.

Instead of a pipeline holding cluster credentials and pushing changes, Argo CD or Flux runs in the cluster, watches a Git repository, and reconciles — the same control-loop idea as Kubernetes itself (Chapter 13.6.14).

What it buys:

• No cluster credentials in CI, which removes a whole class of risk.
• Git is the audit log: who changed what, when, reviewed by whom.
• Drift detection — a manual change is reverted or flagged.
• Rollback is a revert, and recovery from a lost cluster is pointing the agent at the repository.

What it costs:

• Two repositories — application code and deployment manifests — and a step that updates the second when the first builds. Rendering the manifests and committing them is the usual approach.
• Debugging gains a layer: is it the pipeline, the manifest repository, or the agent?
• Secrets need a solution — sealed secrets, an external store with an operator, or a provider's secret store — because plaintext secrets cannot go in Git.

Worth adopting when you have several clusters or environments. For a single cluster and a small team, a push-based pipeline is simpler and fine.

11. A complete free lab on one machine

Everything above, running locally, with no cloud account. This is the fastest way to understand the whole chain, and it deploys the FastAPI plus RabbitMQ microservice from Chapter 10.8.3.

The pieces:

1. Source and CI — Gitea with Gitea Actions (workflow syntax compatible with GitHub Actions), or a GitHub repository with a self-hosted runner in a container.
2. Registry — a local registry container, or Harbor for scanning and signing.
3. Cluster — k3d or kind, which runs a real Kubernetes cluster inside Docker in about thirty seconds.
4. Delivery — Argo CD installed in the cluster, watching a local manifests repository.
5. Observability — Prometheus and Grafana via a Helm chart.

The flow to build, in order:

• Commit to the service repository.
• The pipeline runs tests, builds a multi-stage image tagged with the commit SHA, pushes it to the local registry.
• The pipeline updates the image tag in the manifests repository and commits.
• Argo CD sees the change and applies it to the cluster.
• A rolling update replaces the pods; readiness probes gate the traffic.
• Grafana shows request rate, latency and queue depth.

Then deliberately break things, because that is where the understanding is: push an image that fails its readiness probe and watch the rollout stall rather than take the service down; delete a deployment by hand and watch Argo CD restore it; scale RabbitMQ to zero and watch the consumer's error handling and dead-letter path; apply a migration out of order and see why expand-and-contract exists.

The whole lab runs on a laptop with 8 GB of memory, uses only open-source components, and covers the same concepts as a managed enterprise setup.

What the interviewer will push on

"Why build once and promote?" Because rebuilding per environment means you did not test what you shipped — dependency resolution and build environments drift. The artefact is the unit of promotion, and environment differences must be runtime configuration, not build inputs, or you have per-environment artefacts by another name.

"How do you deploy a database migration safely?" Expand and contract, deployed ahead of and separately from the code, so both old and new versions work against the intermediate schema. Then the reason: a migration bundled into a release cannot be rolled back with it, which is how a rollback becomes an outage.

"Blue-green or canary?" Blue-green for instant rollback at the cost of double capacity; canary to limit blast radius and catch what tests missed, ideally progressive with automated rollback on objective breach. Feature flags are the cheapest rollback of all — a toggle rather than a deploy.

"How does your pipeline get cloud credentials?" OIDC federation — a short-lived token proving the repository and branch, exchanged for a role. No stored key. Then the fork rule: never expose secrets to pull requests from forks, and never run untrusted code on a self-hosted runner with network access.

"What does checkout scm mean in a Jenkinsfile?" In a multibranch pipeline, Jenkins injects the triggering repository, branch and commit into the scm variable, so one Jenkinsfile works for every branch and pull request with nothing hard-coded. It is the mechanism that makes multibranch pipelines work.

"What does GitOps actually give you?" No cluster credentials in CI, Git as the audit log, drift detection, and rollback as a revert. Then the honest costs: a second repository and an update step, an extra layer when debugging, and a real secrets problem because plaintext cannot go in Git. Worth it at several clusters; unnecessary for one.

One thing to volunteer: point out that pipeline feedback time is a productivity metric, not a technical detail. Past about ten minutes people stop waiting and start batching commits, which reverses the point of continuous integration — so cancelling superseded runs, sharding tests and caching on the lockfile hash are engineering-throughput work, not tidying.

Recall

• Build once, promote the same artefact. Rebuilding per environment means you did not test what you shipped. Environment differences are runtime configuration, never build inputs.
• Build pipeline (per commit, environment-agnostic) and release pipeline (per environment, approvals) are different jobs.
• Order the pipeline cheap and broad first, and aim for ten minutes to feedback — past that, people batch commits and continuous integration stops working. Cache on the lockfile hash, shard tests, cancel superseded runs, and prefer ephemeral runners.
• Trunk-based plus feature flags is what makes continuous integration real; every flag needs an owner and a removal date. A merge queue fixes semantic conflicts between independently-passing pull requests.
• Every deploy must be backward compatible with the version it replaces, and migrations go ahead of and separately from the code — expand and contract — or a rollback becomes an outage.
• Security: OIDC federation instead of stored cloud keys, no secrets to fork pull requests, actions pinned by SHA, least-privilege pipeline token, build and deploy privileges separated, artefacts signed and verified.
• checkout scm works because a multibranch pipeline injects the triggering repository, branch and commit — one Jenkinsfile for every branch. Never build on the Jenkins controller, and use shared libraries.
• GitOps removes cluster credentials from CI and makes Git the audit log with drift detection and revert-based rollback, at the cost of a second repository, an extra debugging layer and a real secrets problem.

Self-test: What breaks if you rebuild per environment? · Why must a migration deploy separately from the code that uses it? · What does an "environment" object give you in a pipeline? · Which credential mechanism removes the highest-value secret from CI? · What does scm contain in a multibranch Jenkins job? · What is the productivity cost of a twenty-minute pipeline?

Next: 13.8 covers the other half of reproducibility — describing infrastructure as code, what state actually is, and why the most dangerous line in Terraform is a plan nobody read.