Appearance
10.26 — Big Data & Large-Scale Processing
Processing terabytes or petabytes of data requires fundamentally different paradigms than querying a single relational database. When data exceeds the capacity of one machine's disk, memory, or compute, we must distribute the processing.
This chapter breaks down how modern systems ingest, process, and store massive datasets.
Batch vs. Stream Processing
Big data processing splits into two primary paradigms based on the nature of the data and latency requirements.
Batch Processing operates on bounded datasets. You run a job on a fixed set of data that already exists (e.g., all logs from yesterday). It optimizes for high throughput and handles massive volumes, but execution takes minutes or hours.
Stream Processing operates on unbounded datasets. Data arrives continuously (e.g., live clickstream events). It processes records as they arrive, optimizing for low latency (milliseconds to seconds).
| Feature | Batch Processing | Stream Processing |
|---|---|---|
| Data Scope | Bounded (historical) | Unbounded (continuous) |
| Latency | High (minutes to hours) | Low (milliseconds to seconds) |
| Execution | Discrete jobs | Continuous running topology |
| Use Case | End-of-day billing, ML model training | Fraud detection, live dashboards |
MapReduce: The Foundation of Distributed Batch
MapReduce was the breakthrough that made big data processing accessible. Instead of moving massive data to the compute node, MapReduce moves the compute to the data nodes.
It consists of two user-defined functions:
- Map: Takes a raw input record and outputs a key-value pair.
- Reduce: Takes a key and a list of all values associated with that key, and aggregates them into a final result.
Between Map and Reduce is the Shuffle phase (handled by the framework), which groups all identical keys from the mappers and routes them to the same reducer.
Example: Calculating Daily Sales Volume
Imagine we have millions of transaction records distributed across a cluster. We want to calculate the total revenue per product ID.
typescript
// 1. Define the input data structure
interface Transaction {
transactionId: string;
productId: string;
amount: number;
timestamp: number;
}
// 2. The Map function
// Emits a key-value pair: (productId, amount)
function mapTransaction(record: Transaction): Array<[string, number]> {
// If the record is invalid, we can filter it out by returning an empty array
if (record.amount < 0) return [];
return [[record.productId, record.amount]];
}
// 3. The framework handles the Shuffle phase implicitly.
// It gathers all emitted values for 'prod_A' and passes them to the reducer.
// e.g., 'prod_A' -> [100.50, 45.00, 20.00]
// 4. The Reduce function
// Aggregates the list of values for a single key
function reduceSales(productId: string, amounts: number[]): [string, number] {
const totalRevenue = amounts.reduce((sum, current) => sum + current, 0);
return [productId, totalRevenue];
}Note: Modern frameworks like Apache Spark abstract away raw MapReduce, using directed acyclic graphs (DAGs) and in-memory processing to speed up execution, but the underlying partitioning and shuffling principles remain the same.
Data Pipelines: ETL vs. ELT
To analyze data, you must move it from operational databases (like PostgreSQL) into analytical systems. This movement is called a data pipeline.
ETL (Extract, Transform, Load)
- Extract: Pull data from sources (APIs, DBs).
- Transform: Clean, join, and aggregate the data on an intermediate processing server.
- Load: Write the clean, structured data into the target storage. Best for: Legacy systems with strict storage limits, where computing power outside the warehouse is cheaper.
ELT (Extract, Load, Transform)
- Extract: Pull raw data.
- Load: Dump the raw data directly into the target storage (Data Lake/Warehouse).
- Transform: Use the immense compute power of the storage system (e.g., Snowflake, BigQuery) to transform the data in place via SQL. Best for: Modern cloud architectures. Storage is cheap, so keeping raw data is valuable. Transformation happens on-demand.
Data Warehouse vs. Data Lake
Where do these pipelines deposit the data?
Data Warehouse
A highly structured, relational database optimized for analytics (OLAP).
- Structure: Schema-on-write. Data must conform to strict tables before insertion.
- Data Type: Relational, structured data.
- Tools: Amazon Redshift, Snowflake, Google BigQuery.
- Use Case: Business intelligence (BI) dashboards, financial reporting.
Data Lake
A massive, flat storage repository that holds raw data in its native format.
- Structure: Schema-on-read. You just dump files (JSON, CSV, Parquet) and parse the schema when you query it.
- Data Type: Structured, semi-structured, and unstructured (images, text logs).
- Tools: Amazon S3, Hadoop HDFS, Azure Data Lake.
- Use Case: Machine learning, log archiving, exploratory data science.
(Modern systems are blending these into a "Data Lakehouse"—providing the management features of a warehouse directly on top of the cheap storage of a lake.)
Lambda vs. Kappa Architectures
When building systems that need both historical accuracy and real-time insights, two architectural patterns dominate.
The Lambda Architecture
Lambda maintains two parallel pipelines.
- Batch Layer: Processes all historical data periodically. Slow, but highly accurate and fault-tolerant.
- Speed (Stream) Layer: Processes only recent data in real-time. Fast, but might occasionally drop or duplicate a message.
- Serving Layer: Merges the views. When querying, it combines the accurate batch view with the real-time speed view.
Trade-off: You have to maintain two completely separate codebases and infrastructures (one for batch, one for stream).
The Kappa Architecture
Kappa simplifies this by treating everything as a stream. Data is stored in a highly durable log (like Apache Kafka) with an infinite or very long retention period.
- Real-time processing: Read the stream from the tail.
- Batch processing (Backfilling): Rewind the stream to the beginning and process it as fast as possible.
Trade-off: Simplifies code (only one framework, e.g., Flink), but storing years of historical data in an event streaming platform can be expensive and operationally complex.
What the interviewer will push on
"How do you handle late-arriving data in a stream?" Expect to discuss Watermarks. In stream processing, event time (when the event happened) differs from processing time (when the server sees it). A watermark is a threshold indicating "I do not expect any more events with a timestamp older than X." When the watermark passes a window's end time, the window is closed and processed.
"Can we guarantee exactly-once processing in streaming?" Be precise. You can guarantee exactly-once state updates within the stream processing framework (using distributed snapshots like Chandy-Lamport in Flink), but end-to-end exactly-once requires the destination database to support idempotent writes or two-phase commits.
"Why use Parquet instead of JSON in a Data Lake?" JSON is row-oriented. Parquet is column-oriented. If a query only needs 2 out of 50 columns, Parquet allows the engine to read only those two columns from disk, dramatically reducing I/O and increasing speed. Furthermore, homogeneous column data compresses much better.
Recall
- Batch is for bounded, historical data; Stream is for continuous, low-latency data.
- MapReduce scales computation by moving the processing to the data. Map transforms to key-value; Shuffle groups by key; Reduce aggregates.
- ETL transforms before storage; ELT loads raw data and uses the storage engine's compute to transform.
- Data Warehouses enforce schema-on-write for structured BI. Data Lakes use schema-on-read for cheap, raw storage.
- Lambda Architecture uses parallel batch and stream pipelines. Kappa Architecture treats everything as an immutable stream.
Quiz
- In a MapReduce job counting website page views, what is the structure of the data emitted by the Map phase, and what happens to it before Reduce?
- Why is ELT becoming more popular than traditional ETL in modern cloud architectures?
- How does the Kappa architecture eliminate the need for a separate batch processing layer?
Next: 10.27 turns all of this into numbers: how much hardware, how much money, and where the bottleneck is.