10.10 — Observability: Logs, Metrics, Traces & Alerts That Work

You cannot operate what you cannot see, and in distributed systems "seeing" stops being free: a request touches six services, the failure is in the seventh's dependency, and no single machine holds the story. Observability is the property of being able to answer new questions about a system's behavior without shipping new code — which is more than "we have dashboards." This page builds the three pillars (logs, metrics, traces), the correlation that makes them one system, OpenTelemetry, and the alerting discipline that decides whether your on-call rotation is sustainable.

1. Monitoring vs observability

Monitoring answers known questions: is CPU high, is the error rate above 1%, is the queue growing. You decide the questions in advance and build dashboards for them. Observability is the ability to answer unknown questions after the fact: "why are Android users in Brazil on the checkout page seeing 3-second latencies since Tuesday?" — a query nobody anticipated, answerable only if the telemetry carries enough dimensions (user segment, region, platform, version, endpoint) to slice by.

The practical difference is cardinality: monitoring aggregates aggressively (average latency per service); observability preserves detail (per-request records with many attributes). Both are needed — dashboards and alerts for known failure modes, high-dimensional data for investigation — and the industry's shift is toward preserving more dimensions because the interesting failures are always the unanticipated ones.

2. The three pillars

Logs — discrete events with context. The requirements from 9.7.31/9.9.4, now at fleet scale: structured JSON (queryable, not grep-able), leveled (with dynamic level control per module), sampled at high volume (log 100% of errors, 1% of successes — full-fidelity logs at scale cost more than the system they observe), redacted by allow-list, and — decisively — carrying the trace id so a log line joins its request. Logs answer what exactly happened here; they're the highest-detail, highest-cost pillar.

Metrics — numeric time series, aggregated. Cheap to store and fast to query, so they're what you alert on. The four types worth knowing: counters (monotonic — requests, errors), gauges (point-in-time — queue depth, connections), histograms (distributions — latency buckets, which is how percentiles are computed), and summaries (client-computed quantiles; usually prefer histograms because they aggregate across instances correctly). Cardinality is the trap: a label with unbounded values (user id, URL with ids embedded) multiplies series count and can bankrupt a metrics backend — label by bounded dimensions (endpoint template, status class, region), and put unbounded detail in logs or traces.

Traces — the causal path of one request across services. Each unit of work is a span (name, start, duration, attributes, status), spans nest into a trace identified by a trace id, and context propagation (W3C traceparent header) carries the ids across process boundaries so the pieces assemble. Traces answer where the time went and what called what — the questions that are nearly unanswerable from logs alone once a request spans more than two services (3.8.7's AsyncLocalStorage is how a Node service keeps the context without threading it through every function).

Figure 1 — A distributed trace. Nested spans show not only total latency but where it accumulated and what ran in parallel. Without tracing, this investigation requires correlating timestamps across four services' logs by hand.

3. Correlation, OpenTelemetry, and the fourth signal

The pillars are only powerful together, and correlation is what fuses them: a trace id on every log line and as an exemplar on metrics lets you move from "the p99 spiked at 14:02" → "here are traces from that spike" → "here are the logs of the slow span." Any observability stack that can't make those two hops leaves investigators correlating timestamps by hand.

OpenTelemetry (OTel) is the vendor-neutral standard that made this practical: one set of SDKs and one wire protocol (OTLP) for traces, metrics, and logs, with automatic instrumentation for common libraries (HTTP servers/clients, database drivers, message brokers) and a collector that receives, processes (sampling, redaction, enrichment), and exports to whatever backend you use. Its strategic value is avoiding lock-in: instrument once, switch vendors by changing the collector's exporter.

Sampling is the cost lever, and the choice matters: head-based (decide at the trace's start — simple, but you often discard the slow/errored traces you most want) versus tail-based (buffer spans, decide after seeing the whole trace — keep 100% of errors and slow traces plus a sample of the rest; more infrastructure, far better signal). Tail-based sampling with error/latency rules is the setup mature teams converge on.

The fourth signal worth naming: profiling (continuous CPU/memory profiling — 3.6.9/[14.5]) is increasingly part of the same stack, answering "which code is slow" after the trace has told you which service is.

4. What to instrument

A short, high-yield list — the golden signals plus their distributed extensions:

• The four golden signals (Google SRE): latency (distributions, split by success/failure — a fast error is not a fast request), traffic (requests/sec by endpoint), errors (rate by class), saturation (how full the constrained resource is: CPU, memory, connection pool, queue depth, event-loop lag — 3.8.2).
• The RED method for services (Rate, Errors, Duration) and USE for resources (Utilization, Saturation, Errors) — two mnemonics covering the same ground from different sides.
• Distributed extras that pay for themselves: consumer lag per group (10.8.1), DLQ depth, circuit-breaker state transitions (10.9), replication lag (10.5), retry rates (rising retries precede outages), cache hit ratio, and queue/pool wait times (the invisible-pool lesson — 3.8.2).
• Business metrics beside technical ones: orders/minute, signups, payment success rate — because they detect failures no technical metric catches (a deploy that silently drops a UI button produces perfect infrastructure metrics and zero orders).

5. Alerting that people can live with

The failure mode of observability isn't missing data — it's alert fatigue: too many alerts, too little signal, until humans stop reading them and the one that mattered is ignored. The discipline:

• Alert on symptoms, not causes. "Checkout success rate below SLO" pages someone; "CPU is 85%" does not (high CPU with healthy users is not an incident). Cause-based alerts multiply with your architecture; symptom-based alerts stay constant and match what users feel.
• Page only for human-actionable urgency. If the response is "acknowledge and look tomorrow," it's a ticket, not a page. Every page should have a runbook and a plausible action.
• Alert on SLO burn rate, not raw thresholds: a multi-window, multi-burn-rate policy (fast burn → page; slow burn → ticket) ties urgency to how quickly the error budget is being consumed (10.9) — the single biggest reduction in false pages most teams can make.
• Every alert names its impact and its first step. "Order-placement success 97% (SLO 99.9%), ~40 orders/min failing, runbook: …, dashboard: …, recent deploys: …" — an alert that requires investigation to determine whether it matters is already a failure.
• Review alerts like code: track page volume per rotation, delete alerts that never led to action, and treat a noisy alert as a defect with an owner.

6. The expert lens

Instrument for the questions you'll ask at 3 a.m., not for the dashboard demo. The predictable investigation path is: is it us or a dependency → which service → which operation → which code → which users are affected. That maps to: SLO/business dashboards → traces → span attributes → profiles → high-cardinality slicing. Teams that build in that order have short incidents; teams that build pretty CPU dashboards discover during the outage that they cannot answer question two.

High cardinality is the difference between observing and guessing. The failures that matter are usually specific — one tenant, one API version, one region, one device class — and aggregates hide them by construction (10.6's hot-shard lesson, generalized: an average of a bimodal distribution describes nobody). Preserve dimensions in traces and logs, keep metrics low-cardinality for alerting, and accept the storage cost as the price of answering unanticipated questions.

Observability is a feature with a budget, and it competes with the product. Full-fidelity logs at scale can cost more than the compute they describe. The mature posture: sample aggressively but keep 100% of errors and slow traces (tail-based sampling), keep raw data hot for days and aggregated for months, delete what nobody has queried, and review telemetry spend like any other bill — because an observability platform that gets switched off for cost is worse than a cheaper one that survives.

Next: 10.11 — changing systems safely: deployment strategies, schema migrations, backward compatibility, and the performance/cost levers that dominate real bills.

Recall

• Monitoring answers known questions (dashboards, thresholds); observability answers unknown ones after the fact — the difference is preserved dimensions/cardinality.
• Three pillars. Logs: structured, leveled, sampled (100% errors, ~1% successes), redacted, carrying the trace id — highest detail, highest cost. Metrics: counters/gauges/histograms (percentiles)/summaries — cheap, fast, what you alert on; cardinality is the trap (never label by user id or raw URL). Traces: spans nested under a trace id, propagated via W3C traceparent — answers where the time went across services.
• Correlation fuses them: trace ids on logs and as metric exemplars enable spike → traces → logs in two hops. OpenTelemetry = one SDK + OTLP + auto-instrumentation + a collector (sampling, redaction, export) ⇒ no vendor lock-in. Tail-based sampling (keep all errors/slow traces) beats head-based. Fourth signal: continuous profiling.
• Instrument: four golden signals (latency split by outcome, traffic, errors, saturation), RED/USE, plus distributed extras — consumer lag, DLQ depth, breaker transitions, replication lag, retry rate, cache hit ratio, pool wait times — and business metrics (orders/min catches failures infrastructure metrics miss).
• Alerting: symptoms not causes; page only for human-actionable urgency; SLO burn-rate (multi-window, multi-burn) instead of raw thresholds; every alert states impact + first step + runbook; review alerts like code and delete the ones that never lead to action.
• Lens: instrument for the 3 a.m. path (us-or-them → which service → which operation → which code → which users); high cardinality separates observing from guessing; telemetry is a budget — sample smart or it gets switched off.

Self-test: Give the monitoring/observability distinction in terms of cardinality. Name the metric types and the labeling trap. What two hops does trace-id correlation enable? Why is tail-based sampling better for investigation? State the alerting rule that most reduces false pages.

Quiz Bank

FoundationalWhat are the three pillars, what does each answer, and what is each one's characteristic failure?

Logs — discrete, timestamped events with context; they answer exactly what happened at this point in the code. Characteristic failure: unstructured strings that can't be queried, unbounded volume that costs more than the system, and missing correlation ids that make a line unattachable to its request. Requirements: structured JSON, leveled with dynamic control, sampled (100% of errors, a small fraction of successes), redacted by allow-list, and always carrying the trace id.

Metrics — aggregated numeric time series; they answer how much, how often, how full and are what you alert on because they're cheap to store and fast to query. Types: counters (monotonic), gauges (point-in-time), histograms (distributions — the correct way to get percentiles that aggregate across instances), summaries. Characteristic failure:

cardinality explosion from labeling with unbounded values (user id, raw URLs), which can bankrupt the backend. Traces — the causal path of one request across services, as nested spans sharing a trace id propagated by header; they answer where the time went and what called what. Characteristic failure: broken propagation (a service that doesn't forward context creates orphan traces) and head-based sampling that discards precisely the slow and failing traces you needed. Together, and only together, they support the full investigation: metrics detect, traces localize, logs explain.

FoundationalWhat is OpenTelemetry and what problem does it solve?

OpenTelemetry is the vendor-neutral standard for generating and exporting telemetry: one set of SDKs across languages, one wire protocol (OTLP), and a common data model for traces, metrics, and logs, plus automatic instrumentation for common libraries (HTTP servers and clients, database drivers, brokers) and a collector process that receives, processes (sampling, filtering, redaction, enrichment), and exports to any backend.

The problems it solves: (1) vendor lock-in — previously, instrumenting meant embedding a vendor's agent throughout your code, so changing vendors meant re-instrumenting everything; with OTel you instrument once and switch exporters in the collector's config. (2)

Polyglot inconsistency — every language and framework had its own conventions; OTel provides shared semantic conventions so http.route means the same thing everywhere, which is what makes cross-service queries possible. (3) Correlation — a single context propagation standard (W3C traceparent) means traces assemble across services written by different teams in different languages. (4)

Processing before egress — the collector is where sampling policy, PII redaction, and cost control live, outside application code. Practically: adopt auto-instrumentation first (large coverage for little effort), add manual spans and attributes for domain-meaningful operations, and treat the collector's configuration as the place where telemetry policy is centrally enforced.

AppliedDesign the alerting policy for a service with a 99.9% availability SLO. What pages, what tickets, what neither?

Base the policy on error-budget burn rate, not raw thresholds. 99.9% over 30 days ≈ 43 minutes of budget (10.9). Use multi-window, multi-burn-rate rules: a fast burn (e.g. consuming 2% of the monthly budget in one hour — roughly a 14× burn rate, sustained over both a 1-hour and a 5-minute window to avoid flapping) → page, because at that rate the month's budget is gone in days. A slow burn (e.g. 10% of budget over 6 hours) → ticket for the next working day, because it's real but not urgent. Anything below → dashboard only.

What pages: symptom-level, user-facing indicators — checkout success rate, p99 latency of the critical path, and business metrics with obvious impact (orders/minute at zero). What tickets: slow burns, rising retry rates, DLQ arrivals, replication lag above target, a circuit breaker that opened and closed, certificate expiry in 30 days, capacity trending toward a limit.

What neither: cause metrics with no user impact — CPU at 85%, a single instance restarting, a pod evicted, disk at 70%. Those belong on dashboards and in capacity reviews; alerting on them is the primary source of fatigue and trains people to ignore pages.

Required alert content: the user-visible impact, the current vs target number, a link to the dashboard and the runbook, and recent deploys — an alert that requires investigation to determine whether it matters has already failed. Hygiene: track pages per rotation, review every page monthly for "did this lead to action?", and delete or downgrade the ones that didn't.

InterviewA user reports 'the app is slow.' Walk your investigation with a modern observability stack.

Step 1 — is it real and how broad? Check the SLO dashboard for the affected journey: is p99 elevated, for whom, since when? This immediately separates "one user's device/network" from "a real regression," and the shape (sudden step vs gradual creep) hints at cause (deploy vs growth).

Step 2 — is it us or a dependency? Service-level RED metrics plus circuit-breaker and dependency-latency panels: if our latency rose in lockstep with a downstream's, the investigation moves there (10.9). Step 3 — which operation? Slice by endpoint, region, tenant, client version — high-cardinality dimensions are what make this possible; an aggregate p99 would tell you nothing beyond "something is slow."

Step 4 — where does the time go? Pull traces from the slow bucket (tail-based sampling means the slow ones were kept): the span waterfall shows exactly which call consumed the latency and whether calls that should be parallel are serialized — the single most valuable artifact in the investigation (section 2's figure).

Step 5 — why is that span slow? Span attributes (query text, cache hit/miss, retry count, payload size) plus logs filtered by the trace id, plus continuous profiling if the answer is "our code" rather than "a dependency." Step 6 — who is affected and what do we do? Slice by dimension to size the blast radius, then act: roll back if it correlates with a deploy (10.11), shed or degrade if it's load (10.9), or fix forward.

The check on your stack: if any step requires SSH-ing into a box or correlating timestamps across logs by hand, that's the gap to close before the next incident.

StaffYour observability bill has grown to 30% of infrastructure spend, yet mean time to resolution is getting worse. Diagnose and fix.

The pattern — paying for volume rather than signal. Likely contributors: (1) logging everything at info level including full request/response bodies, so the largest cost centre is data nobody queries (9.9.4); (2)

head-based trace sampling at a low rate, so the traces retained are mostly healthy ones and investigators can't find the slow requests — expensive and useless, which is exactly the combination that raises MTTR while raising the bill; (3) metrics cardinality explosion from labels like user id or raw URLs, multiplying series counts; (4)

no lifecycle — everything retained at full fidelity for a year; (5) alert fatigue meaning incidents are noticed late, which inflates MTTR independently of tooling. Fixes, ordered by benefit/effort: (a) Tail-based sampling — keep 100% of errored and slow traces plus 1–5% of the rest: usually cuts trace spend substantially while improving investigation, because the retained traces are the interesting ones.

(b) Log discipline — drop or sample info-level success logs (traces already cover the request path), keep 100% of warn/error, remove body logging, enforce redaction; measure spend per service and give teams their number. (c) Cardinality audit — find the top series-producing labels and bound them (endpoint templates, not raw paths; tenant tier, not tenant id — put the unbounded detail on traces/logs where it belongs).

(d) Retention tiers — hot for 7–14 days, aggregated/rolled-up for months, raw archived cheaply to object storage for compliance. (e) Fix MTTR directly — trace-id correlation everywhere, an SLO/burn-rate alerting policy replacing threshold noise (section 5), runbooks linked from alerts, and a quarterly review of "which signal actually resolved each incident?" — that review is what tells you which 30% of the bill to cut and which 5% to increase.

The framing for leadership: observability spend should be justified by resolution speed, not by data volume — we are currently buying storage instead of answers, and the fix both lowers the bill and shortens incidents.

Flashcards

FlashMonitoring vs observability

Monitoring: known questions, aggregated. Observability: unknown questions, preserved dimensions. The difference is cardinality.

FlashThree pillars

Logs = what happened here (structured, sampled, trace id). Metrics = how much/how full (histograms; cardinality trap). Traces = where time went across services (spans + traceparent).

FlashOpenTelemetry

One SDK + OTLP + semantic conventions + auto-instrumentation + collector (sampling/redaction/export). Instrument once, switch vendors in config.

FlashSampling

Head-based decides at the start (drops the slow/failed ones you need). Tail-based buffers and keeps 100% of errors + slow traces — better signal, more infra.

FlashWhat to instrument

Golden signals (latency by outcome, traffic, errors, saturation) + lag, DLQ depth, breaker transitions, replication lag, retry rate, pool waits + business metrics (orders/min).

FlashAlerting rules

Symptoms not causes · page only if human-actionable now · SLO burn-rate (multi-window) not thresholds · every alert carries impact + first step + runbook · delete alerts that never lead to action.

Scenario Drill

DrillYou join a company with 20 services, no tracing, logs in files on each host, Nagios-style CPU alerts, and a 4-hour average incident resolution time. Design the observability rollout in priority order with the business case for each step, and name the metric that proves it worked.

Priority 1 — correlation ids and structured logs to a central store (weeks 1–3). Without a trace id and centralized search, every investigation begins with SSH and timestamp arithmetic; this single change collapses the "find the relevant logs" phase from hours to minutes. Implementation: a shared middleware/library that mints or accepts a request id, puts it in AsyncLocalStorage-equivalent context (3.8.7), stamps every log line, propagates it on outbound calls, and returns it in error responses (9.9.3) so users can quote it. Logs go to stdout and are shipped by the platform (9.7.31) — no application-level shipping.

Priority 2 — distributed tracing via OpenTelemetry auto-instrumentation (weeks 3–8). With 20 services, "which service is slow" is the dominant question and it is currently unanswerable; auto-instrumentation gives HTTP, database, and broker spans for near-zero code change, and the same propagation from priority 1 is what makes traces assemble. Start with the two highest-traffic user journeys, tail-based sampling from the beginning (keep all errors and slow traces).

Priority 3 — SLOs and burn-rate alerting on the top three journeys (weeks 6–12), replacing CPU alerts. The current alerting both misses user-visible failures and pages for non-events; defining checkout/search/login SLOs measured at the edge, then paging only on fast burn (section 5), simultaneously reduces noise and catches the failures that matter. This is also the step that gives the organization a shared language for reliability trade-offs (10.9).

Priority 4 — the distributed-systems signals (weeks 8–14): consumer lag, DLQ depth, breaker transitions, retry rates, replication lag, pool waits — each of which turns a class of silent degradation into a visible one. Priority 5 — business metrics beside technical ones, so a deploy that breaks a button is detected by orders/minute rather than by a customer email.

The business case, stated once: at 4 hours average resolution and (say) 2 incidents a month, the company loses ~8 hours of degraded service monthly plus the engineering time; tracing and correlation typically cut localization time by more than half, and burn-rate alerting cuts detection time from customer-report to minutes.

The metric that proves it worked: MTTR, decomposed — time-to-detect, time-to-localize, time-to-mitigate, tracked per incident. If detect falls after priority 3 and localize falls after priority 2, the investment is demonstrably working; if localize doesn't fall, the traces aren't covering the right paths and the fix is more instrumentation, not more tools. Track alert volume per rotation alongside it, because an MTTR improvement bought with alert fatigue is temporary.