Appearance
11.19 — Metrics & Monitoring System
Someone adds one label to one metric. The label is the user's identifier. The deploy goes out at 14:02, and by 14:11 the metrics cluster is out of memory and every dashboard in the company is blank — including the dashboards you would use to work out what happened.
That is the characteristic failure of this system, and it is unusual in two ways. It is caused by a change that looks completely harmless in review. And when it happens, it removes your ability to see every other system at the same time.
This is the study that builds the thing watching everything else. It is time-series at scale: write-dominated, append-only, and ruthlessly compressible — and the compression is not an optimisation, it is the difference between a system that exists and one that does not.
1. Requirements
Functional. Ingest metrics — counters, gauges, histograms — with labels. Query with aggregation over time and across dimensions. Alerting rules that produce notifications. Dashboards. Retention with automatic reduction in resolution.
Non-functional, with numbers.
- Ingest 10 million data points a second.
- Query p99 under 1 second over a month of data.
- Under 30 seconds from a breach to a notification.
- The monitoring system must not fail alongside the systems it monitors. This is an obligation with a design consequence, not a nice sentiment.
- 13-month retention at reduced resolution.
Out of scope today: trace storage, log aggregation, and incident management tooling.
The clarifying questions, and what each answer changes
"How many distinct series, not how many data points?" This is the question that matters and the one people skip. Memory in a time-series database scales with the number of series, not with the volume of data, so ten million points a second across ten thousand series is easy and ten million series receiving one point each is fatal.
"What is the longest query anyone will run?" A year-long dashboard over a million series is a completely different cost from an hour-long one over ten, and it decides whether reduced-resolution tiers are optional or structural.
"Is data loss acceptable?" For metrics, yes — and getting that agreed early is what keeps the system affordable. This is not a ledger. A thirty-second gap during a restart is a gap in a graph, not a missing payment.
"Who is allowed to define new metrics and labels?" If the answer is "any engineer, at deploy time", then the system needs hard limits that reject, because the incident above will happen and it will happen more than once.
"How does a page reach a human, and does that path share anything with production?" Ask it explicitly. If paging goes through infrastructure you also monitor, an outage of that infrastructure is invisible.
2. Estimation
Series count. 10,000 hosts × ~1,000 series each = 10 million active series. What that forces: the index and the in-memory head block are sized by this number, not by the data rate, and section 7.2 is entirely about defending it.
Data points. At a 10-second collection interval that is 1 million points a second, and with key metrics at 1-second resolution, around 10 million a second.
Raw storage, which is the number that kills the naive design. Each point is an 8-byte timestamp plus an 8-byte value plus series identity — call it 16 bytes — so 10 million a second is 160 MB a second, about 13.8 TB a day. What that forces: rejection of the whole approach. Fourteen terabytes a day is not a budget line, it is a different company.
Compressed storage, which is the number that makes it exist. With the two encodings in section 7.1, a data point averages about 1.3 bytes. That is 13 MB a second, roughly 1.1 TB a day. What that forces: everything. A twelve-fold reduction turns an impossible system into an ordinary one, and it is the first thing to say when someone asks how this works.
Query cost, and why reduced resolution is structural. A year of one series at 10-second resolution is 3.15 million points. At one-hour resolution it is 8,760. What that forces: a dashboard showing a year across a hundred series reads 876,000 points instead of 315 million. That is the difference between a second and a timeout, and it is why the tiering in section 7.3 is not a storage-cost decision but a query-latency one.
Memory per series, which is the real limit. Each active series carries an index entry, its label set, and an open compression stream — on the order of a few kilobytes. Ten million series is therefore tens of gigabytes of memory before any data. What that forces: the arithmetic behind the cardinality catastrophe. Adding one unbounded label does not increase the data rate at all; it multiplies the series count, and memory follows series count.
3. API
# ingest — the collector scrapes, or the client pushes
http_requests_total{service="checkout", method="POST", status="200"} 1274 1754003722
request_duration_seconds_bucket{service="checkout", le="0.25"} 981 1754003722http
GET /query?q=sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m])) by (method)
&start=2026-07-01T00:00:00Z&end=2026-08-01T00:00:00Z&step=1h
→ 200 { "resolution": "1h",
"seriesScanned": 412,
"pointsScanned": 3610080,
"result": [ { "labels": {"method":"POST"}, "values": [[…]] } ] }http
POST /rules
{ "name": "checkout_error_budget_burn",
"expr": "burn_rate(checkout_availability, 1h) > 14",
"for": "5m",
"severity": "page",
"runbook": "https://…",
"owner": "team-checkout" }http
GET /series/cardinality?by=metric&order=growth # the query that ends the 3 a.m. incident
→ 200 [ { "metric": "http_requests_total", "series": 8412006, "growth1h": 8400000 } ]resolution in the response says which tier answered. A query spanning a year is answered from hourly aggregates, and saying so stops someone from puzzling over why a spike they remember is not visible — it was averaged away, deliberately, and the response admits it.
seriesScanned and pointsScanned are how a query becomes chargeable and limitable. A query that would touch a million series must be rejected with a clear message rather than allowed to consume the cluster, and these are the fields the limit is expressed in.
Every rule carries an owner and a runbook link, because section 7.5 argues that an alert without a documented expected action is not an alert. Making them required fields in the API is how that becomes true rather than aspirational.
The cardinality endpoint exists because it is the first query anyone runs during the incident at the top of this page, and it needs to work when the cluster is struggling.
4. Data model
series
series_id UINT64 -- hash of metric name + full label set
metric TEXT
labels MAP<TEXT, TEXT>
first_seen, last_seen TIMESTAMPTZ
points -- append-only, compressed blocks
series_id, timestamp, value
label_index -- inverted index
(label_key, label_value) → sorted set of series_id
blocks -- immutable, in object storage
block_id, time_range, min_ts, max_ts, series_count, pathAccess patterns:
| Query | Frequency | Returns |
|---|---|---|
| Append a point to a series | 10,000,000/s | — |
| Resolve a label matcher to series ids | on every query | thousands of ids |
| Read a compressed block range | on every query | a stream of points |
| Evaluate an alert rule | thousands, every 15 s | one number per series |
| Count series by metric | on demand and by alarm | a few hundred rows |
A series is identified by its metric name plus its complete label set. http_requests{service="api", status="200"} and the same metric with status="500" are two entirely separate series, each with its own index entry, its own memory, and its own compression stream.
Therefore cardinality is the product of label value counts. Five services × four methods × eight statuses is 160 series, which is fine. Add a label with a million distinct values and it becomes 160 million. Section 7.2 is about defending this, and it is the operational heart of the system.
Querying by label is an inverted index problem — exactly the machinery from 11.12. status="500" and service="checkout" are two posting lists to intersect, producing the matching series identifiers whose blocks are then read and aggregated. Recognising it as the same problem is worth doing out loud, because everything learned there about intersection cost applies here.
Blocks are immutable once sealed, which is the same log-structured discipline as the search index. They compress well, cache well, and need no locking on read.
5. Architecture
6. Deep dives
6.1 Compression, which decides whether the system exists
Two encodings, both exploiting the specific shape of metric data.
Delta-of-delta on timestamps. Measurements arrive on a schedule — every ten seconds, near enough. Storing raw 8-byte timestamps wastes almost all of that space. Storing the difference gives a repeated 10000. Storing the difference between the differences gives 0 almost every time, which encodes in a single bit, with a few extra bits reserved for the occasional jitter. Timestamp cost collapses from eight bytes to a fraction of a bit in the common case.
Bitwise-difference encoding on values. Consecutive values of a real metric are similar: processor usage 42.1% then 42.3%, memory 8.1 GB then 8.1 GB. Take the bitwise difference of two similar floating-point numbers and the result has many leading and trailing zeros, so you store how many leading zeros there are, how many meaningful bits follow, and only those bits. An unchanged value differs by nothing and costs one bit; a small change costs a handful.
Together — the two were published as one scheme for a system called Gorilla, which is why you will hear the pairing named that way — a data point averages about 1.3 bytes instead of 16.
Why this is foundational rather than an optimisation: at 10 million points a second, raw is 160 MB a second and 13.8 TB a day, and compressed is 13 MB a second and 1.1 TB a day. One of those is a system and the other is a proposal that gets rejected.
And it makes queries fast for a reason people miss. A month of one series reads roughly 340 KB compressed instead of 4 MB raw, so far more of the working set fits in memory and disk stops being the bottleneck. The compression buys latency as well as cost.
The constraints it imposes are the honest trade. Each value's encoding depends on the previous one, so a block is a stream with no random access and no in-place update. Blocks must therefore be written sequentially and sealed immutably. And out-of-order data does not fit — which is why late arrivals are accepted only into the mutable head block and rejected beyond a bounded window.
6.2 Cardinality, the operational heart
The catastrophe comes from a single unbounded label. Adding user_id with a million users multiplies series by a million. Adding request_id is worse and unbounded — every request creates a series that receives exactly one point and is never written to again. path="/orders/8821" with identifiers embedded in URLs does the same thing. So does error="connection refused to 10.2.3.4:5432", because an error string containing an address is effectively unbounded.
It is the number-one operational risk for four reasons together. It is trivially easy to do. It is invisible in review, because one extra label looks harmless and even helpful. Its effect is nearly instantaneous, because the series are created as soon as the deploy rolls. And it destroys the system you would use to diagnose it.
Defences, in layers.
Hard limits that reject, loudly. Per-metric and per-tenant series caps enforced at ingest, returning a clear error naming the offending metric. Rejecting is the correct behaviour, and it is worth defending explicitly: absorbing means the cluster dies and everyone loses monitoring, while rejecting means one metric is broken and its owner is told which one.
Attribution and detection. A view of series count by metric and by team, with an alarm on rate of change rather than on an absolute number. A metric whose series count triples in an hour is a deploy that needs reverting, and catching it at 14:05 rather than 14:11 is the difference between a message and an incident.
Prevention at authoring time. Allow-listed label keys for high-volume metrics, and an automated check in the build pipeline that flags labels whose values look unbounded.
And the cultural half, which is genuinely half the fix. The rule is that a label's value set must be small, bounded, and known in advance. Anything per-user, per-request, per-URL-with-an-identifier, or derived from free text belongs in logs or traces, never in a metric label. That boundary is exactly the division between the three observability signals in 10.10, and stating it as a rule people can apply is more effective than any limit.
6.3 Reduced resolution, which buys both cost and speed
Raw data for 15 days. Five-minute aggregates for 90 days. One-hour aggregates for 13 months.
A year-long dashboard then reads about 8,760 points per series instead of 3.15 million, which is the difference between a second and a timeout — so this is a query-latency mechanism at least as much as a storage-cost one.
Store minimum, maximum, sum and count — never just the average. This is the detail that gets skipped and it matters twice over. Averages cannot be re-aggregated correctly: the average of a hundred hourly averages is not the daily average unless every hour had identical counts. And an average destroys exactly what you are looking for — the five-second spike to 100% that caused the incident disappears entirely into a smooth hourly line, while a stored maximum preserves it.
Percentiles need the same care. You cannot average percentiles either. Storing histogram buckets rather than computed percentiles is what lets a p99 be recomputed correctly over any time range and any set of series.
6.4 Alerting mechanics, which matter more than the query language
A for duration — fire only if the condition has held for five minutes — eliminates the majority of false alarms from transient spikes, and it is the single cheapest improvement available to a noisy alerting setup.
Grouping and deduplication turn four hundred hosts breaching one threshold into one notification with a count, rather than four hundred pages.
Inhibition suppresses downstream alerts when an upstream cause is already firing. When the database is unreachable, page for the database and not for the fifty services that depend on it. This alone often halves paging volume during an incident, which is precisely when a human's attention is most valuable.
Alert on symptoms, not causes. High error rate and high latency wake a human. High processor usage is a dashboard. A service at 90% processor serving every request perfectly is not a problem, and the metric is neither reliably harmful nor reliably present when things actually break.
And evaluate against fresh data from the writer, not against the query tier reading reduced-resolution history. Alerting duplicates a little query logic in exchange for keeping the breach-to-notification time under thirty seconds, and that is the right trade.
6.5 Burn-rate alerting, the highest-leverage refinement
Threshold alerts have a structural problem: any single threshold is either too sensitive during a brief blip or too slow during a gradual degradation.
Alert on how fast the error budget is being consumed instead. If the service promises 99.9% availability, the month's budget is 0.1% of requests. A burn rate of 1 means the budget will be exactly exhausted at the end of the month; a burn rate of 14 means it will be gone in about two days.
Use two windows together. A short window catches sudden severe breaches quickly, and a long window catches slow erosion that a short window would dismiss as noise. Requiring both to be burning prevents a one-minute blip from paging.
The effect is that a brief incident costing 0.1% of the month's budget does not wake anyone, while a sustained degradation does — which is the behaviour every team wants and very few configure.
6.6 Independence, stated bluntly
The monitoring system must not depend on the infrastructure it monitors. Separate deployment, separate storage, separate network path.
And the part most often skipped: the notification path must survive the outage. If paging goes through your own email service, an outage of that service is invisible — the alert fires correctly, the notification is generated correctly, and nobody is told. The paging path should be an external provider, with a fallback route through a different provider.
Meta-monitoring is non-negotiable. An independent system verifies that the alerting pipeline is evaluating rules and delivering notifications, because the worst failure in this domain is silence that looks like health.
And verify it with drills, because untested independence is assumed independence. The question to answer is not "is it separate on the architecture diagram" but "when the main region went down last quarter, did the page arrive".
6.7 Metrics are loss-tolerant, and saying so keeps the system affordable
This is not a ledger. If an ingest shard dies, the series it owned lose data for the outage window, and that is acceptable — a gap in a graph, not a missing payment.
Saying this out loud matters, because the alternative instinct is to make ingestion durable and acknowledged end to end, which multiplies cost and adds a failure mode to the thing that must not fail. A write-ahead log in the writer covers process restarts, and beyond that the correct answer is that a gap is a gap.
The exception worth naming: anything used for billing or for a contractual commitment is not a metric in this sense, and it belongs in the durable-accounting path from 11.14 — which is exactly the boundary the rate limiter's drill in 11.3 drew.
7. Decision Ledger
| Decision | Alternatives | Why this | What it costs |
|---|---|---|---|
| Delta-of-delta and bitwise-difference encoding | raw storage; general-purpose compression | ~12× reduction, which is the difference between existing and not | a stream with no random access; out-of-order data does not fit |
| Immutable sealed blocks with a memory head | mutable time-series rows | fast appends, no locking, cheap object storage | late data is awkward and must be bounded |
| Reduced-resolution tiers | keep raw forever | a year-long query reads thousands of points, not millions | aggregates must keep min/max/sum/count, never just the average |
| Hard cardinality limits that reject | trust the people writing metrics | one bad label breaks one metric instead of the platform | legitimate cases get rejected, so errors must be clear and appeals easy |
| Scraping as the default | push only | discovery, liveness from the scrape itself, no client-side buffering | reachability constraints; short-lived jobs need a push path |
| Alerts evaluate on the writer's fresh data | evaluate through the query tier | keeps breach-to-page under 30 seconds | alerting duplicates a little query logic |
| Burn-rate alerts over thresholds | fixed thresholds | fires in proportion to real harm, not to instantaneous badness | needs an agreed objective and budget first |
| Metrics are loss-tolerant | durable acknowledged ingestion | keeps cost sane and removes a failure mode from the safety system | gaps during restarts, which must be understood and accepted |
8. Scale and failure
At 10×, shard writers by series hash, which scales linearly. Separate the query path from ingestion so a heavy dashboard cannot slow data collection. And tier storage — recent on fast disk, older in object storage with a cache — which is the shape every mature time-series system converges on independently.
| What breaks | Blast radius | How you find out | What keeps it running | Recovery |
|---|---|---|---|---|
| Cardinality explosion | the whole cluster, in minutes | series-count growth rate per metric | hard limits that reject at ingest | roll back the deploy; drop the metric at ingest; purge the series |
| Ingest shard down | that shard's series, for the window | ingest rate by shard | metrics are loss-tolerant; a gap is acceptable | it resumes; the log covers process restarts |
| A query too expensive | the cluster, if unlimited | seriesScanned and pointsScanned | per-query limits with a clear error | reject it; suggest a coarser resolution |
| Dashboards starve ingestion | data collection itself | ingest lag rising while queries are heavy | separate query and ingest paths | shed queries first, always |
| Alert evaluation falls behind | silence that looks like health | evaluation-lag metric, and meta-monitoring | protect alerting over dashboards under pressure | shed dashboards; never shed alerting |
| Notification path shares fate | pages are generated and never arrive | meta-monitoring from outside, plus drills | an external provider with a second route | this is a design failure, not an incident |
| Late-arriving data | those points | rejected-point counter | accept within a bounded window, reject beyond | none — out-of-order into sealed blocks is not worth the complexity |
| Monitoring system itself down | visibility into everything | meta-monitoring from an independent system | independence, verified by drills | restore; and treat any missing page as the primary finding |
Two rows deserve emphasis and both are about silence. Alert evaluation falling behind produces no errors — rules are evaluating, just late — and the system looks healthy while nobody is being told about anything. And a notification path that shares fate with production produces perfectly correct alerts that nobody receives. Both need a detector that lives outside this system entirely, which is what meta-monitoring means (10.10).
And the ordering rule for load shedding, which should be written down before it is needed: under pressure, sacrifice the ability to look at graphs before the ability to be told about incidents. Dashboards go first, queries second, alert evaluation last.
What the interviewer will push on
"How do you store ten million points a second?" Start with the raw arithmetic — 16 bytes a point is 160 MB a second and 13.8 TB a day, which is not a system — then the two encodings and the 1.3-byte result, and the observation that a twelve-fold reduction is the difference between existing and not. Then name the constraint it buys: each value depends on the previous one, so blocks are streams that must be sealed immutably and out-of-order data does not fit.
"What is cardinality and why is it dangerous?" The count of distinct series, which is the product of label value counts, because a series is a metric name plus its full label set. Then the arithmetic that makes it fatal rather than merely large: the data rate does not change at all when a bad label is added — the same requests produce the same measurements — but memory scales with series count, so a fixed per-series cost is multiplied by a million. And the four properties that make it the top risk: easy, invisible in review, instantaneous, and it destroys the tool you would debug with.
"Why can't you just store averages when you reduce resolution?" Two reasons. Averages cannot be re-aggregated — the average of hourly averages is not the daily average unless every hour had the same count. And an average destroys the thing you are looking for: a five-second spike to 100% vanishes into a smooth hourly line, while a stored maximum preserves it. Same for percentiles, which is why histogram buckets are stored rather than computed percentiles.
"Your on-call gets forty pages a night. Fix it." The framing first: alert fatigue is a safety problem, not a comfort one, because an engineer receiving forty pages stops reading them and misses the one that mattered. Then the mechanisms — for duration, grouping, deduplication, inhibition so an upstream cause suppresses fifty downstream effects — and then the structural change, which is burn-rate alerting so pages are proportional to real harm rather than to instantaneous badness. Close with governance: every alert has an owner, a runbook and an expected action, and any alert that repeatedly fired without anyone acting gets deleted.
"What do you refuse to page on?" Causes. Processor usage, memory, disk activity, queue depth, one unhealthy host. They are diagnostics — useful on a dashboard, useless as a page, because they are neither reliably harmful nor reliably present when things break. The exception is a leading indicator with guaranteed lead time and a required action, such as a disk filling with four hours of headroom.
"The metrics cluster is down. How do you find out?" From a different system. Meta-monitoring is the answer, and the follow-up that separates a considered answer is the notification path: if paging depends on infrastructure this system also monitors, the alert fires correctly, the notification is generated correctly, and nobody is told. Verify with drills, because untested independence is assumed independence.
Volunteer this, because nobody asks: write down the load-shedding order before you need it. Under pressure, this system should sacrifice the ability to look at graphs before the ability to be told about incidents — dashboards first, expensive queries second, alert evaluation never. That ordering is obvious in a calm room and completely non-obvious at three in the morning, when the instinct is to keep the dashboards working so you can see what is happening. Deciding it in advance, and building the shedding to enforce it, is what stops the monitoring system's own degradation from blinding you at exactly the wrong moment.
Next: 11.20 — from measuring what happened to predicting what somebody wants. The feed from 11.8 gets a ranking layer, and with it a latency budget for a model, a feature store, and a feedback loop that quietly trains on its own output.
Recall
- Compression decides whether the system exists: delta-of-delta timestamps (fixed intervals mean the second difference is usually zero, so one bit) plus bitwise-difference float encoding (consecutive values are similar) gives ~1.3 bytes a point against 16 raw — 13 MB a second instead of 160. The trade: blocks are streams, sealed immutably, and out-of-order data does not fit.
- A series is a metric name plus its full label set, so cardinality is the product of label value counts. One unbounded label (
user_id,request_id, a path with identifiers, an error string) multiplies series by millions — while the data rate does not change at all, because memory scales with series count. - Defend with hard limits that reject loudly, attribution by metric and team, an alarm on growth rate rather than an absolute number, allow-listed label keys, and the rule that unbounded values belong in logs or traces.
- Storage: an in-memory head block with a write-ahead log, sealed into immutable compressed blocks. Label queries are an inverted index — the same machinery as 11.12.
- Reduced resolution (raw 15 days → 5 minutes for 90 days → 1 hour for 13 months) buys query latency as much as storage: a year reads ~8,760 points per series rather than 3.15 million. Store min, max, sum and count — never just the average, because averages cannot be re-aggregated and they erase the spike you are looking for.
- Alerting mechanics beat query languages: a
forduration, grouping, deduplication, inhibition so an upstream cause suppresses downstream effects, symptoms rather than causes, and evaluation against the writer's fresh data. - Burn-rate alerts fire in proportion to error-budget consumption, with a short and a long window, so a brief blip does not page and a slow erosion does.
- Independence includes the notification path. Separate deployment, storage, network — and a paging route that survives the outage, verified by drills. Meta-monitoring from outside, because the worst failure here is silence that looks like health.
- Metrics are loss-tolerant by design. A gap during a restart is acceptable; anything billable belongs in the durable accounting path instead.
Self-test: Why does compression decide feasibility, and what does it cost you? Define cardinality and explain why the data rate is irrelevant to it. Why can reduced-resolution tiers not store averages? Give four noise-control mechanisms and the structural one that beats them. What must be true of the notification path, and what is the load-shedding order?
Quiz Bank
FoundationalExplain time-series compression and why it is the design's foundation.
The data has an exploitable shape, and two encodings exploit it.
Delta-of-delta on timestamps. Measurements arrive on a schedule — every ten seconds, near enough. Storing raw 8-byte timestamps wastes almost all of that space. Storing deltas gives a repeated 10000. Storing the delta of the deltas gives 0 almost every time, which encodes in a single bit, with a few bits reserved for the occasional jitter when a collection runs slightly late. Timestamp cost collapses from eight bytes to a fraction of a bit in the common case.
Bitwise-difference encoding on values. Consecutive values of a real metric are similar — processor usage 42.1% then 42.3%, memory 8.1 GB then 8.1 GB. Take the bitwise difference between two similar floating-point numbers and the result has many leading and trailing zeros. Store the count of leading zeros, the number of meaningful bits, and only those bits. An unchanged value differs by nothing and costs one bit; a small change costs a handful. The two techniques were published together as one scheme, which is why they are usually named as a pair.
Together, a data point averages about 1.3 bytes against 16 raw.
Why it is foundational rather than an optimisation. At 10 million points a second, raw storage is 160 MB a second and 13.8 TB a day — not a budget line but a different company. Compressed, it is 13 MB a second and about 1.1 TB a day, which is an ordinary infrastructure cost. There is no version of this system that stores raw points and survives, so the encoding is not a performance detail, it is the enabling condition.
And it buys latency, not only cost, which people miss. A query over a month of one series reads roughly 340 KB instead of 4 MB, so far more of the working set fits in memory and disk stops being the bottleneck. Compression makes queries fast for the same reason it makes storage cheap.
The constraints it imposes are the honest trade. Each value's encoding depends on the previous value, so a block is a stream: no random access into the middle of it, and no in-place update. Blocks must therefore be written sequentially and sealed immutably once complete. And out-of-order data does not fit the stream at all, which is exactly why late arrivals are accepted only into the mutable in-memory head block and rejected beyond a bounded window — supporting arbitrary out-of-order writes into sealed blocks would mean giving up the encoding, and with it the system.
InterviewWhat is cardinality, why is it the number one operational risk, and how do you control it?
Cardinality is the number of distinct time series, and a series is uniquely identified by its metric name plus its complete label set. So http_requests{service="api", method="GET", status="200"} and the same metric with status="500" are two entirely separate series, each carrying its own index entry, its own labels in memory, and its own open compression stream.
Cardinality is therefore the product of label value counts. Five services × four methods × eight statuses is 160 series, which is completely reasonable.
The catastrophe is a single unbounded label. Add user_id with a million users and 160 becomes 160 million. Add request_id and it is unbounded — every request creates a series that receives exactly one data point and is never written to again. A path label with identifiers in it (/orders/8821) does the same. So does an error label carrying free text with an address in it.
And here is the part that makes it fatal rather than merely large: the data rate does not change at all. The same requests produce the same number of measurements. What changes is the number of distinct series, and every fixed per-series cost — index entry, label set, compression stream — is multiplied by a million. A cluster sized for 10 million series runs out of memory within minutes of the deploy.
It is the number-one operational risk because four things are true at once. It is trivially easy to do. It is invisible in code review, because one extra label looks harmless and often looks helpful. Its effect is nearly instantaneous. And it takes down the system you would otherwise use to diagnose it.
Control comes in layers.
Hard limits that reject. Per-metric and per-tenant series caps enforced at ingest, returning a clear error naming the metric. Rejecting is the correct behaviour and worth defending: absorbing means the cluster dies and everyone loses monitoring, whereas rejecting means one metric is broken and its owner is told which one.
Attribution and detection. Series count by metric and by team, with an alarm on rate of change rather than an absolute number — a metric whose series count triples in an hour is a deploy that should be reverted, and catching it six minutes after the deploy rather than nine turns an incident into a message.
Prevention at authoring time. Allow-listed label keys for high-volume metrics, and an automated check in the build pipeline flagging label values that look unbounded.
Education, which is genuinely half the fix. The rule is that a label's value set must be small, bounded, and known in advance. Anything per-user, per-request, per-URL-containing-an-identifier, or derived from free text belongs in logs or traces, never in a metric label — which is precisely the boundary between the three observability signals (10.10).
StaffDesign the alerting layer to minimise both missed incidents and alert fatigue. What do you page on, and what do you refuse to page on?
Start from the premise that alert fatigue is a safety problem rather than a comfort problem. An on-call engineer receiving forty pages a night stops reading them carefully, and the one that mattered is the one that gets missed. So the design goal is not "alert on everything important" but "every page is actionable and worth waking someone for", with everything else visible and silent.
Page on symptoms, from the user's point of view (10.10): elevated error rate on user-facing endpoints, latency past the objective, and request volume collapsing toward zero — which catches a whole class of failure that error rates miss entirely, because nothing is failing when nothing is arriving.
And page on error-budget burn rate rather than raw thresholds. This is the highest-leverage change most teams can make. If the service promises 99.9%, the month's budget is 0.1% of requests, and a burn rate of 14 means that budget is gone in about two days. Combine a short window that catches sudden severe breaches with a long window that catches slow erosion, and require both — so a one-minute blip costing 0.1% of the budget does not page, while a sustained degradation does.
Refuse to page on causes. Processor usage, memory, disk activity, queue depth, a single unhealthy host. These are diagnostics: valuable on dashboards and in the incident channel, useless as pages, because they are neither reliably harmful — a service at 90% processor serving every request perfectly is fine — nor reliably present when something actually breaks. The one exception is a leading indicator with a guaranteed lead time and a required action: a disk filling with four hours of headroom pages, because by the time it is a symptom it is an outage and the fix takes longer than that.
The noise-suppression mechanics that must exist. A for duration, so a condition has to persist — this alone removes most transient false alarms. Grouping, so four hundred hosts breaching one threshold is one notification with a count. Deduplication across rules describing the same failure. Inhibition, so an upstream cause suppresses downstream effects — when the database is unreachable, page for the database and not for the fifty services that depend on it, which often halves paging volume during an incident. And routing by severity, so only genuine wake-a-human alerts page and everything else becomes a channel message or a ticket.
Governance, which is what keeps it good over time. Every alert has an owner, a runbook link, and a documented expected action. An alert whose runbook says "investigate" is not an alert, it is a notification pretending to be one, and it should be downgraded or deleted. Review paging volume monthly with the on-call rotation, and delete alerts that repeatedly fired without anyone acting — a human looking at an alert and doing nothing, twice, is the strongest available evidence that the alert is worthless.
And meta-monitoring is not optional: an independent system verifies that rules are being evaluated and notifications delivered, because the worst failure in this domain is silence that looks like health.
The framing to state: the alerting layer's real output is human attention, which is the scarcest resource in the whole system. It should be spent like a budget, and every alert must justify its cost in interruptions against the incidents it actually catches.
Flashcards
FlashCompression
Delta-of-delta timestamps (fixed intervals mean the second difference is usually zero, so one bit) plus bitwise-difference float encoding. ~1.3 bytes a point against 16 raw — 13 MB/s instead of 160. Cost: streams, sealed immutably, no out-of-order data.
FlashCardinality
Distinct series = product of label value counts. One unbounded label means millions of series — with no change in data rate at all, because memory scales with series count. Reject at a hard limit and alarm on growth rate.
FlashReduced resolution
Raw 15 days → 5 minutes for 90 days → 1 hour for 13 months. A year reads ~8,760 points instead of 3.15 million. Store min, max, sum and count — never just the average, which cannot be re-aggregated and erases spikes.
FlashAlert noise control
for duration · grouping · deduplication · inhibition (upstream suppresses downstream) · severity routing. And the structural one: burn-rate alerts with a short and a long window, instead of fixed thresholds.
FlashIndependence
Separate deployment, storage, network — and notification path, or pages are generated correctly and never arrive. Meta-monitoring from outside. Verified by drills, because untested independence is assumed independence.
FlashLoad-shedding order
Under pressure: dashboards first, expensive queries second, alert evaluation never. Decide it in advance, because at 3 a.m. the instinct is to keep the graphs working.
Scenario Drill
DrillThe metrics cluster is at 95% memory and rising, ingestion is lagging, and dashboards are timing out. It is 3 a.m. What do you do, in order?
First minute — stabilise, do not diagnose. The failure to avoid is the cluster running out of memory entirely, because a metrics cluster that dies takes your visibility into every other system with it, and recovery means replaying logs while production traffic keeps arriving.
Shed the cheapest thing first. Enforce or lower the query cost limits: heavy dashboard and ad-hoc queries compete for the same memory as ingestion, and killing expensive queries buys headroom in seconds while losing no data at all. Then lower the ingest cardinality limit so that new series are rejected while existing ones continue — this preserves the metrics you already have, which are the ones you need to diagnose, and stops the growth. And explicitly protect alert evaluation over dashboards: if something must be sacrificed, sacrifice the ability to look at graphs before the ability to be told about incidents.
Second — find the cause, which is almost certainly cardinality. Memory in a time-series database scales with series count, not with data volume, so memory rising under steady traffic means new series are being created. Query series count by metric, ordered by recent growth. The offender is nearly always one metric, and its labels will show the unbounded one — a user identifier, a request identifier, a URL path with an identifier in it, or an error string. Correlate the growth's start time with deploys, and the answer is usually a deploy from the last few hours.
Third — stop it at the source. If it is a deploy, roll it back; that is faster and safer than any server-side mitigation. If a rollback is not immediate, drop that metric at ingest with a rule or a blocklist so the cluster stops accepting it — losing one metric is vastly better than losing all of them. Then purge or expire the bad series to reclaim memory, which may mean compacting or restarting shards in a rolling fashion once ingestion is stable.
Fourth — verify recovery in the right order. Memory stabilising, then ingest lag draining, then alert evaluation running on time — verify this explicitly, because a metrics cluster can look healthy while alerting is silently behind, and that is the dangerous state — and only then dashboards.
Fifth — the post-mortem findings, which matter more than the fix.
A rejecting cardinality limit should have prevented this entirely. If it was not set, or was set high enough to permit an out-of-memory kill, that is the primary finding. The limit exists precisely so that one bad metric breaks one metric rather than the platform.
Detection was far too late. An alarm on series-count growth rate per metric would have fired minutes after the deploy, when rolling back was trivial. Its absence is why this became a three-in-the-morning incident instead of a message in a channel at two in the afternoon.
No guard at authoring time. An automated check flagging label values that look unbounded would have caught it before the change merged.
Query and ingest share fate. Separating the query path from ingestion, so that a heavy dashboard cannot threaten data collection, is the structural fix and it should be scheduled rather than discussed.
Meta-monitoring must confirm that alerting kept working throughout — and if it did not, that is a more serious finding than the outage itself, because it means that for some period nobody would have been told about anything.
The general principle: the monitoring system's own failure is uniquely costly, because it removes the ability to see every other failure. So it needs stricter limits, earlier alarms and more aggressive load shedding than the systems it watches — and every one of those should degrade toward preserving alerting above everything else.