14.2 — The Engineer's Toolbox

A release checklist has eleven steps. Someone runs it about forty times a month, it takes twelve minutes, and roughly once a quarter a step gets skipped and causes an incident.

Eight hours a month, plus four incidents a year. A script that does it takes an afternoon. Most engineers know this and do not write the script, because the twelve minutes never feel like the right moment.

This page is the tooling that makes that afternoon short: how command-line programs actually work, how to build one worth using, how to draw architecture that stays current, and where automation genuinely pays.

1. What a command-line program actually is

It is a process with three streams, some arguments and an exit code (Chapter 2.8).

Arguments arrive as argv, an array. The shell splits on whitespace before your program sees anything, which is why quoting matters: rm $file with a filename containing a space becomes two arguments.

Three streams: stdin for input, stdout for the result, stderr for everything else. This split is what makes pipelines work. Progress bars, warnings and logs go to stderr, so mytool | jq receives only data. A tool that prints progress to stdout cannot be piped, and it is the most common mistake in a first CLI.

The exit code is the API for scripts. 0 is success, anything else is failure, and &&, || and CI steps all depend on it. Use distinct non-zero codes for distinct failures — 1 general, 2 usage error — and never exit 0 on failure, which silently breaks every caller.

Detect whether you are talking to a terminal. isatty tells you, and the convention is: colour, progress and interactive prompts only when interactive. Piped or redirected output should be plain. Respect NO_COLOR, and offer --no-color.

How the command gets found. The shell searches PATH for an executable, reads the shebang (#!/usr/bin/env python3), and runs the interpreter with your file as an argument. Package managers put commands on PATH by declaring an entry point — npm's bin field, Python's console_scripts — which creates a small wrapper in a directory already on PATH. That is the whole mechanism.

Signals matter for anything long-running. SIGINT from Ctrl+C should clean up and exit, not leave a half-written file. SIGPIPE is what happens when you pipe to head and it exits — an unhandled SIGPIPE produces the "broken pipe" error people see and ignore.

2. Designing one people will use

Nouns and verbs: tool resource action — deploy service restart, db migrate up. Consistent, discoverable, and it scales past ten commands where a flat list does not.

--help on everything, with examples. The examples are what people actually read.

Sensible defaults, overridable. A tool that requires six flags for the common case will be wrapped in a shell alias by everyone.

Machine-readable output on request. --json turns your tool into something composable. This single flag is the difference between a tool and a component.

--dry-run for anything destructive, showing exactly what would happen. And a confirmation prompt for destruction — which must be skippable with --yes for automation, or people script around your safety.

Idempotent where possible. Running it twice should be safe.

Configuration precedence, in this order: command-line flags → environment variables → a project config file → a user config file → defaults. Follow the platform's config locations rather than dropping a dotfile in the home directory.

Exit non-zero on failure, print errors to stderr, and say what to do next. Error: config not found. Run 'tool init' to create one. is a good error; Error: ENOENT is not.

3. Building one

Python — the default for internal tooling, because it is everywhere and the ecosystem is deep.

import typer, json
from pathlib import Path

app = typer.Typer(help="Manage deployments.")          # (1)

@app.command()
def deploy(
    service: str,                                       # (2)
    env: str = typer.Option("staging", "--env", "-e"),
    dry_run: bool = typer.Option(False, "--dry-run"),
    output_json: bool = typer.Option(False, "--json"),
):
    """Deploy SERVICE to an environment."""            # (3)
    plan = build_plan(service, env)
    if dry_run:
        typer.echo(json.dumps(plan) if output_json else render(plan))
        raise typer.Exit(0)                             # (4)
    try:
        result = run(plan)
    except DeployError as e:
        typer.echo(f"Error: {e}", err=True)             # (5)
        raise typer.Exit(2)
    typer.echo(json.dumps(result) if output_json else f"Deployed {service}")

(1) Typer builds the parser from type hints, so the signature is the interface. (2) A required positional argument; Option makes it a flag. (3) The docstring becomes the help text, so documentation cannot drift from the code. (4) An explicit exit code. (5) Errors to stderr, and a distinct code.

argparse is in the standard library and needs no dependency — the right choice for a script that must run anywhere. Click is Typer's underlying library and is equally good. Rich adds tables and progress, and should be switched off when not interactive.

Node — commander or oclif, distributed via npx so users need no install step.

Go or Rust — when you need a single static binary with no runtime. cobra in Go is the standard, and it is why so many infrastructure tools are Go: one file to download, no version conflicts, cross-compiled for every platform.

Distribution decides adoption. pipx install for a Python tool (isolated, on PATH), npx for Node, a single binary in a release for Go or Rust, and a package manager formula if it is public. A tool that requires "clone the repo and set up a virtual environment" will not be used.

4. Shell scripts, and when to stop

Bash is right for gluing commands together, and it stops being right sooner than people think.

Always start with:

#!/usr/bin/env bash
set -euo pipefail        # (1)
IFS=$'\n\t'              # (2)

(1) -e exit on error, -u error on undefined variables, -o pipefail fail if any command in a pipeline fails. Without pipefail, false | true succeeds — which is how a broken step in a pipeline passes silently. (2) Splits on newlines and tabs only, so filenames with spaces stop breaking loops.

Quote every variable. "$var", not $var. Almost every shell bug is an unquoted variable.

Run ShellCheck. It catches the entire class of mistakes above and should be in CI.

Switch to Python when the script passes about a hundred lines, needs data structures, needs error handling beyond exit codes, needs to parse JSON or XML properly, or must run on Windows. A 400-line Bash script is a Python script that has not been written yet, and the rewrite is usually shorter.

5. Diagrams as code

A diagram in a drawing tool is out of date within a month, because updating it requires opening a separate application. A diagram in a text file next to the code is reviewed in the same pull request as the change.

Mermaid — renders natively in GitHub, GitLab, Notion, and this book. The right default, because it needs no toolchain.

```mermaid
sequenceDiagram
    Client->>API: POST /orders
    API->>DB: INSERT order
    API-->>Queue: OrderPlaced
    Queue-->>Worker: OrderPlaced
    Worker->>Payments: charge()
```

It covers flowcharts, sequence, state, entity-relationship, class and Gantt diagrams. Its weakness is layout control — you get what its engine decides, and on a large graph that is often a mess.

Graphviz / DOT — you describe nodes and edges and a layout engine positions them. This is what wins on large or generated graphs: dot for hierarchies, neato and fdp for undirected networks, circo for circular layouts.

digraph services {
  rankdir=LR;
  api -> postgres [label="reads"];
  api -> queue;
  queue -> worker;
  worker -> payments [label="HTTPS"];
}

PlantUML is stronger for UML specifically (Chapter 9.2.7); D2 is a newer alternative with better layouts and a modern syntax.

And the highest-value use, which most people miss: generate diagrams from real data.

import pydot
g = pydot.Dot("infra", graph_type="digraph", rankdir="LR")
for r in terraform_state_resources():                     # (1)
    g.add_node(pydot.Node(r.name, shape="box"))
for src, dst in r.dependencies:
    g.add_edge(pydot.Edge(src, dst))
g.write_png("infra.png")                                  # (2)

(1) The source is the real system — Terraform state, a service registry, a database schema, an OpenAPI specification. (2) Regenerate in CI on every change.

A generated diagram cannot be wrong. That is a categorical difference from a hand-drawn one, and it is worth an afternoon for any architecture people argue about.

6. Scraping and document automation

Check for an API first. An API is stable, documented, rate-limited politely and permitted. Scraping is what you do when there is no API, and it breaks when the page changes.

The ethics and the rules are not optional: respect robots.txt (Chapter 5.6.1), read the terms of service, identify yourself in the user agent, rate-limit yourself, cache what you fetch so re-running does not re-hit, and do not scrape personal data without a lawful basis (Chapter 8.7).

import requests
from bs4 import BeautifulSoup

r = requests.get(url, headers={"User-Agent": "acme-reports/1.0 (ops@acme.com)"},
                 timeout=10)                              # (1)
r.raise_for_status()                                      # (2)
soup = BeautifulSoup(r.text, "lxml")                      # (3)

for row in soup.select("table#prices tbody tr"):          # (4)
    cells = [c.get_text(strip=True) for c in row.select("td")]

(1) Always a timeout — a request with none can hang forever. (2) Raise on an HTTP error rather than parsing an error page. (3) lxml is much faster than the built-in parser. (4) CSS selectors are the readable interface, and selecting by structure (table#prices) is more durable than by class names, which change with styling.

For pages rendered by JavaScript, a parser sees an empty shell. Playwright drives a real browser, and it is far heavier — so check for a JSON endpoint in the network tab first, which the page itself is calling and which is usually easier to consume than the HTML.

Document generation is the other half, and it removes a genuinely tedious category of work:

• python-docx — Word documents. The pattern that works is a template with placeholders, filled programmatically, so the formatting stays with the people who care about formatting.
• openpyxl — Excel, including formulas and charts. Business stakeholders want a spreadsheet, and delivering one rather than a CSV is a small effort with a large reception.
• PDF — render HTML with a headless browser, which gives you CSS for layout instead of a drawing API.

7. What to automate

Automate when frequency × time × error-proneness is high: anything done weekly that takes more than a few minutes, anything with a checklist, anything that has caused an incident by being skipped, and anything that blocks someone else while they wait.

Do not automate a one-off, something whose rules change every time, or something where the automation is harder to maintain than the task. And do not automate a broken process — you get the same wrong outcome faster and with fewer people watching.

The most valuable target is usually the thing nobody complains about, because it has been absorbed as normal. The eleven-step release checklist is invisible until someone measures it.

8. Editor and environment

Language servers are why editors got good. A single protocol gives every editor real completion, go-to-definition, find-references, rename-symbol, inline errors and formatting from a language-aware server (Chapter 3.1's front end doing double duty). The editor is now a preference; the language server is the capability.

The skills that transfer everywhere:

• Go to definition and find references — fluency here is the largest single reading-speed gain.
• Rename symbol — a real refactor, not find-and-replace, so it does not touch a string that happens to match.
• Multi-cursor and structural selection.
• Fuzzy file and symbol search — stop navigating a tree.
• A three-way diff view for conflicts (Chapter 14.1.2).
• The integrated debugger.

That last one deserves emphasis. Most engineers debug with print statements for their whole career. A debugger — breakpoints, conditional breakpoints, watch expressions, stepping, inspecting the whole stack — is a permanent multiplier, and the investment is an afternoon. Conditional breakpoints alone ("stop when orderId == 'ord_991'") replace a dozen print-and-rerun cycles. Chapter 14.6 puts it in the wider method.

A terminal multiplexer (tmux, or the equivalent) keeps sessions alive across disconnections — which is not optional when working on a remote machine over an unreliable link.

Put your dotfiles in version control. Shell configuration, editor settings, Git configuration, aliases. A new machine becomes a clone and a script, and — the underrated part — you can see what you changed and why.

Recall

• A CLI is arguments, three streams, and an exit code. Results to stdout, everything else to stderr — a tool that prints progress to stdout cannot be piped. Never exit 0 on failure. Use isatty so colour and progress appear only when interactive.
• Design: noun-verb commands, --help with examples, sensible defaults, --json (which turns a tool into a component), --dry-run plus a confirmation that --yes can skip, idempotency, and config precedence flags → environment → project file → user file → defaults.
• Build with Typer/Click (help text comes from the docstring), argparse when no dependency is allowed, Go or Rust for a single static binary. Distribution decides adoption — pipx, npx, or one downloadable file.
• Shell: set -euo pipefail (without pipefail, a failing step in a pipeline passes silently), quote every variable, run ShellCheck — and switch to Python past about a hundred lines.
• Diagrams as code are reviewed with the change instead of rotting. Mermaid by default (renders everywhere), Graphviz/DOT for large or generated graphs. The best use is generating them from real data — Terraform state, a service registry, a schema — because a generated diagram cannot be wrong.
• Scraping: look for an API first, respect robots.txt and terms, identify yourself, rate-limit, cache, always set a timeout, and select by structure rather than class names. Check the network tab for a JSON endpoint before reaching for a headless browser.
• Automate on frequency × time × error-proneness; do not automate a broken process. The best target is usually the task nobody complains about.
• Language servers are why editors got good. Learn go-to-definition, rename-symbol, and — the big one — the debugger, because conditional breakpoints replace a dozen print-and-rerun cycles. Keep dotfiles in version control.

Self-test: Why must progress output go to stderr? · Which single flag turns a tool into a composable component? · What does pipefail prevent? · What makes a generated diagram categorically different from a drawn one? · What should you check before writing a scraper? · Which editor skill is the largest permanent multiplier?

Next: 14.4 covers how the work is organised around all this — Scrum and its alternatives assessed honestly, what story points measure, and what changed about estimation once assistants started writing code.