12.4.4 — Pipelines, Validation and Feature Stores

Every Monday at 2 a.m., Cartwise's fraud model retrains automatically on the newest matured labels, as the case study in 12.3.7 required. One Monday in March, the job finished without a single error, the new model passed its automatic checks, and it went live. By Wednesday, the fraud reviewers were being sent almost no orders, and fraud losses were climbing.

The cause was a change three teams away. The payments team had switched the amount field from rupees to paise, a hundredth of a rupee, to avoid rounding errors. A ₹2,000 order was now recorded as 200,000. Nothing crashed. Ordinary software would have thrown an error on a wrong type, but this was still a perfectly valid number, just a hundred times bigger. The training job happily learned that "normal" orders are enormous, and the live model, comparing today's amounts with that new normal, decided almost nothing looked unusual.

This is the defining risk of ML pipelines: bad data does not break them, it silently changes what they learn. This last data page covers the machinery that stops that from happening: the pipeline that runs every step the same way every time, the validation that catches bad data before training, the feature store that guarantees training and serving compute features identically, and the versioning that lets you reproduce any past model exactly.

1. The pipeline: one path, run the same way every time

Everything in Chapters 12.4.1 to 12.4.3, labelling, feature computation, point-in-time joins, resampling, has to happen again every time the model retrains. If a person does those steps by hand in a notebook, they will be done slightly differently each time, and nobody will be able to say exactly what produced a given model. So the steps are written as code and chained into a data pipeline: a sequence of automated stages, each taking the previous stage's output as its input.

The pipeline behind the fraud model. Data is validated before anything learns from it, features are defined once and written to two stores, training reads history from the offline store, and the live service reads the latest values from the online store.

Walk through it. Raw data comes from the orders database and from the stream of order events.

• ① Validation checks it. If anything is wrong, the pipeline stops and alerts someone, instead of training on bad data.
• ② Feature computation turns the checked records into features, using definitions written once, as code.
• ③ The features are written to a feature store. It has two halves, for two very different jobs.
• ④ The training job reads history from the offline half, exactly as it was at each past moment. It produces a new model, stamped with the version of everything that went into it.
• ⑤ The live fraud service reads the latest feature values from the online half, within a few milliseconds.

The stages are run in order, on a schedule, by a tool called a workflow orchestrator. It knows which stage depends on which, runs each when its inputs are ready, retries failures and records what happened. Apache Airflow, started at Airbnb in 2014, is the most widely used; Dagster, Prefect and the cloud providers' own services do the same job. Chapter 12.12.4 returns to the full lifecycle around this pipeline.

2. Validation: check the data before it teaches anything

The paise bug would have been caught in seconds by a check as simple as "the median order amount should be between ₹500 and ₹5,000". Data validation means writing such expectations down and checking every batch of data against them before it is used. There are three levels of check, each catching a different class of problem.

Schema checks: is the shape right? The expected columns exist, with the expected types. A renamed column, a number arriving as text, a new column that appeared from nowhere. These are the failures ordinary software would also catch, and they are the easy ones.

Value checks: is each value sensible? Amounts are positive. Account age is not negative. Payment method is one of the known values. The country code has two letters. These catch corrupted and malformed records.

Distribution checks: does the batch look like normal data? This is the level that catches the paise bug, because every individual value was valid. The batch was wrong: its median was a hundred times the usual. Distribution checks compare summary statistics of today's batch with a reference period: the median and spread of each number, the share of missing values, the mix of categories. A feature that is suddenly missing in 12 percent of rows instead of 0.1 percent, as with the new payment method in 12.1.4, or a category share that doubles overnight, deserves a human look before anything trains on it.

Here is what the core of such a check looks like in plain pandas:

import pandas as pd

EXPECT = {                                                  # (1) written down once
    "amount":      {"min": 1, "median_between": (500, 5_000), "max_null": 0.0},
    "account_age": {"min": 0, "max_null": 0.01},
    "payment":     {"allowed": {"card", "upi", "wallet", "cod", "bnpl"}},
}

def validate(batch: pd.DataFrame) -> list[str]:
    problems = []
    for col, rule in EXPECT.items():
        if col not in batch:                                 # (2) schema
            problems.append(f"{col}: missing column"); continue
        s = batch[col]
        if s.isna().mean() > rule.get("max_null", 1.0):      # (3) missing share
            problems.append(f"{col}: {s.isna().mean():.1%} missing")
        if "min" in rule and (s < rule["min"]).any():        # (4) value range
            problems.append(f"{col}: values below {rule['min']}")
        if "median_between" in rule:                         # (5) distribution
            lo, hi = rule["median_between"]
            if not lo <= s.median() <= hi:
                problems.append(f"{col}: median {s.median():,.0f} outside {lo}–{hi}")
        if "allowed" in rule and not set(s.dropna()) <= rule["allowed"]:
            problems.append(f"{col}: unknown values {set(s.dropna()) - rule['allowed']}")
    return problems

monday = pd.DataFrame({"amount": [149_900, 200_000, 52_000_00],
                       "account_age": [10, 400, 3], "payment": ["upi", "card", "card"]})
print(validate(monday))     # → ['amount: median 200,000 outside 500–5000']

Line (1) writes the expectations down as data, which is the important habit: once they are written down they can be reviewed, versioned and shared. Line (2) is the schema check. Line (3) checks the share of missing values. Line (4) checks the range of individual values. Line (5) is the distribution check, and it is the one that fires on Monday's batch: every amount is a valid positive number, but the median is 200,000, which is impossible in rupees and exactly right in paise. The pipeline halts, an engineer looks, and the model never learns from the bad data.

In practice teams use a library rather than hand-written checks. Great Expectations and pandera are the common Python ones: you declare expectations like those above, and the library runs them, reports failures and keeps a history. Whatever the tool, the hard part is choosing the expectations, and the best source of them is every incident you have had so far.

Data contracts

The paise change was not a mistake by the payments team. It was a sensible change, made without knowing that a model three teams away depended on the old meaning. The organisational fix is a data contract: a written, versioned agreement between the team that produces a dataset and the teams that consume it, stating the columns, types, units, allowed values and expected freshness. The producer agrees not to change any of it without notice, and the validation checks above become the automated enforcement of the contract. In practice that means a field like amount_paise gets added alongside amount for a transition period, consumers switch over, and only then is the old field removed. It is the same discipline as versioning an API, from Chapter 9.6.3, applied to data.

3. The feature store: one definition, two speeds

The fraud model needs "orders on this card in the last hour" in two completely different situations.

• For training, it needs the value as it was at the moment of each of two million past orders. That is a huge batch computation over history, run once a week, where taking ten minutes is fine.
• For serving, it needs the value right now for one card, inside the 50 millisecond budget. There is no time to scan history. The value must already be sitting somewhere, ready to read.

If two different teams write those two computations, in two different languages, they will differ in some small way, such as whether the current order is counted, and the model will see different numbers live than it learned from. That is the training-serving skew from 12.3.7. A feature store exists to prevent it. You define each feature once, and the store produces it in both forms:

• The offline store keeps the full history of every feature value with timestamps, usually in a data warehouse or in files in cloud storage. It supports the point-in-time join from 12.4.2: "give me each of these features as it was at each of these two million moments".
• The online store keeps only the latest value of each feature for each key (each card, each account), in a fast key-value database such as Redis from Chapter 7.6, so that a lookup takes a millisecond or two.

The most widely used open-source feature store is Feast, short for feature store. Here is roughly what a feature definition looks like in it:

from datetime import timedelta
from feast import Entity, FeatureView, Field
from feast.types import Int64, Float32

card = Entity(name="card", join_keys=["card_id"])                       # (1)

card_activity = FeatureView(
    name="card_activity",
    entities=[card],                                                     # (2)
    ttl=timedelta(days=2),                                               # (3)
    schema=[Field(name="orders_last_1h", dtype=Int64),
            Field(name="amount_last_24h", dtype=Float32)],
    source=card_activity_source,                                         # (4)
)

# training:  store.get_historical_features(entity_df=orders_with_timestamps, features=[...])
# serving:   store.get_online_features(entity_rows=[{"card_id": "c1"}], features=[...])

Line (1) declares the thing features are about, here a card, and the column that identifies it. Line (2) attaches a group of features to cards. Line (3) says how long a value stays valid: if a card's features have not been updated in two days, the store treats them as unknown rather than serving stale numbers. Line (4) points at where the computed values come from, a table or stream produced by the feature computation stage. The two commented calls at the bottom are the payoff: training asks for historical features at each order's timestamp, and the store does the point-in-time join; serving asks for online features for one card, and gets the latest values in milliseconds. Both come from the same definition.

A feature store is real infrastructure with real upkeep. A team with one model and features computed in a single SQL query does not need one; a saved, tested preprocessing pipeline like the one in 12.4.2 is enough. It earns its place when several models share features, when features must be fresh to the second, or when point-in-time joins over large histories have become the main source of bugs.

4. Versioning: be able to rebuild any model exactly

In June, a customer whose order was blocked in January files a complaint, and the regulator asks Cartwise to explain the decision. Which model made it? What data was it trained on? Could you train it again and get the same model?

If the training job simply ran SELECT * FROM orders WHERE date < '2026-01-01' against the live database, the answer is no. Since January, rows have been corrected, late disputes have arrived, and some records have been deleted. The same query today returns different data. To answer the regulator, and to debug models in general, every model must be traceable to the exact data, code and settings that produced it. Three practices make that possible.

Snapshot the training data. Write each training dataset to storage as files that are never changed afterwards, and give the snapshot an identifier. A tool called DVC (data version control) does this in a Git-like way: the large data files live in cloud storage, while small pointer files that record their content fingerprint are committed to Git alongside the code. Checking out an old commit and running dvc pull brings back the exact data of that time. Another approach is to store tables in a format that keeps its own history: Delta Lake and Apache Iceberg record every change to a table, so you can query it "as of" any past moment, a feature usually called time travel.

Record everything that went into each model. For every trained model, store the data snapshot identifier, the Git commit of the training code, every hyperparameter, the library versions, and the evaluation results, next to the model file itself. This bundle is what a model registry holds, and experiment-tracking tools record it automatically during training; Chapter 12.5.7 and Chapter 12.12.4 cover both.

Version the labels and guidelines too. As 12.4.1 showed, labels change when guidelines change. A snapshot of the data must include the labels as they were, and which guideline version produced them.

With all three, the June question has an answer: "Model fraud-2026-01-05 made the decision. It was trained on snapshot orders-2025-12-31, code commit 4f9c2e1, with these settings. Here are the top reasons for this order." And if a bug is found in a feature, you can find every model trained with it.

5. What the data chapter adds up to

Step back and look at Chapters 12.3 and 12.4 together, because they make one argument. The algorithms of 12.3 are mature, free and available in a few lines of scikit-learn. What decides whether a model works in production is almost entirely in 12.4: whether the labels mean one consistent thing, whether every feature was knowable at prediction time, whether rare cases were counted properly, and whether the whole process runs identically every week with bad data stopped at the door. That is why experienced ML engineers say, only half joking, that the job is mostly data engineering. The deep learning chapters that come next change what models can learn from, images and text instead of rows, but every rule on these four data pages still applies to them.

Recall

• Bad data rarely crashes an ML pipeline; it silently changes what the model learns, as the rupees-to-paise change did.
• A data pipeline runs validation, feature computation, storage, training and serving as automated stages, scheduled by a workflow orchestrator such as Airflow.
• Data validation checks schema, individual values, and batch distributions; distribution checks catch errors where every value is valid but the batch is wrong. A data contract is the producers' written promise about columns, units and changes.
• A feature store defines each feature once and serves it two ways: an offline store of timestamped history for point-in-time training data, and an online store of latest values for millisecond lookups. It prevents training-serving skew. Feast is the common open-source one.
• Version data snapshots (DVC, or time-travel table formats like Delta Lake and Iceberg), labels and guidelines, and record data version, code commit and settings for every model, so any model can be explained and rebuilt.

Self-test: Which of the three levels of validation catches the paise change, and why do the other two miss it? Why does the online store keep only the latest value while the offline store keeps the full history? What three things must be recorded to rebuild January's model in June?

Next: 12.5.1 The Neuron and Activation Functions — rows and columns are behind us; to learn from photos, speech and text, models must learn their own features, and that starts with a single artificial neuron.