3.8.3 — Buffer & Binary Data

Browser JavaScript historically rarely touched raw bytes; a server does constantly — file contents, TCP packets, image uploads, hashes, protocol frames. JavaScript's native types are wrong for this: strings are UTF-16 character sequences (3.6.7), not byte containers, and mangling bytes through string conversion is how uploads corrupt. Node's answer, present since day one, is the Buffer: a fixed-length chunk of raw memory holding bytes 0–255. This page covers what a Buffer really is (including where its memory lives), the allocation APIs and the security story behind them, encodings done correctly, the TypedArray family it belongs to, and the sharp edges — slice aliasing, multi-byte characters split across chunks — that cause real bugs.

1. What a Buffer is, physically

const b = Buffer.from("hi👍", "utf8");
console.log(b);            // → <Buffer 68 69 f0 9f 91 8d>  — six raw bytes
b.length;                  // → 6   bytes, not characters ("hi👍".length is 4!)
b[0];                      // → 0x68 (104) — indexable like an array of 0–255

A Buffer is fixed-length raw memory. Two physical facts explain its behavior:

• The bytes live outside V8's managed heap (in "external" memory the GC tracks by size but doesn't scan) — so multi-hundred-MB buffers don't bloat GC work (3.6.9), and process.memoryUsage().external is where they show up. Watch that number when debugging "RSS grows but heap looks fine."
• A Buffer is a Uint8Array — since Node 4, Buffer subclasses the standard TypedArray (1.4's typed views over an ArrayBuffer). Everything TypedArrays can do, Buffers can; Buffer adds the encoding/reading/writing conveniences that servers need.

Length is fixed at allocation — no push, no resize; "growing" means allocating a new Buffer and copying (Buffer.concat).

2. Allocation — and why allocUnsafe exists

const a = Buffer.alloc(1024);         // 1 KB, ZERO-FILLED — safe default
const c = Buffer.from([0xde, 0xad]);  // from bytes
const d = Buffer.from("café", "utf8");// from a string + encoding
const e = Buffer.allocUnsafe(1024);   // 1 KB, NOT initialized — old memory!

allocUnsafe skips zero-filling, handing you whatever bytes previously occupied that memory — potentially fragments of earlier secrets (keys, request bodies). It exists because zeroing costs time in hot paths, and Node itself uses it internally where every byte will be overwritten immediately. The rule: alloc unless you will provably overwrite every byte before any read — and the reason this API is named scary is a security lesson: uninitialized memory disclosure is a real vulnerability class (Heartbleed's family). (allocUnsafe also draws small allocations from a shared internal pool — faster, but one more reason its contents are foreign.)

3. Encodings: bytes ↔ text, explicitly

Converting between bytes and text requires naming an encoding — this explicitness is Buffer's quiet virtue, versus platforms that guess:

const buf = Buffer.from("café", "utf8");     // text → bytes: 63 61 66 c3 a9
buf.toString("utf8");     // → "café"        ✅ same encoding back
buf.toString("latin1");   // → "café"       // wrong encoding = mojibake (1.4)
buf.toString("hex");      // → "63616663a9"… bytes as hex text
buf.toString("base64");   // → "Y2Fmw6k="    bytes as base64 text

The encodings that matter: utf8 (the default and the world's text interchange format — 1.4); hex and base64 (not human languages but byte-to-text armor — how binary travels inside JSON, URLs, and JWTs — Chapter 8.4's tokens are base64url all the way down); latin1 (one byte per char, legacy protocols); utf16le (what JS strings are internally; Windows interop). ascii is a trap (it mangles high bytes) — use latin1 or utf8.

The chunk-boundary bug — the one that ships to production: UTF-8 characters span up to 4 bytes, and a network/file chunk boundary can land mid-character. Calling chunk.toString("utf8") per chunk then yields � replacement characters at the seams — intermittently, only for non-ASCII users:

// ❌ per-chunk decoding corrupts multi-byte characters at boundaries
stream.on("data", (chunk) => process(chunk.toString("utf8")));   

// ✅ string_decoder holds incomplete trailing bytes until the next chunk
const { StringDecoder } = require("string_decoder");
const dec = new StringDecoder("utf8");
stream.on("data", (chunk) => process(dec.write(chunk)));         

(Or set stream.setEncoding("utf8"), which wires a StringDecoder internally; or, best, don't decode mid-stream at all — pipe bytes through and decode once at the end. 3.8.4 continues this.)

4. Reading, writing, slicing — and the aliasing trap

Buffers read/write structured numbers at offsets — the toolkit for binary protocols (every network protocol header, Part 5, is "integers at offsets with an agreed endianness" — 1.3):

const frame = Buffer.alloc(8);
frame.writeUInt32BE(0xdeadbeef, 0);   // 4-byte int, big-endian ("network order"), offset 0
frame.writeUInt16LE(80, 4);            // 2-byte int, little-endian, offset 4
frame.readUInt32BE(0).toString(16);    // → "deadbeef"

And the API everyone mis-learns: buf.subarray(start, end) (and the deprecated-in-spirit buf.slice) does not copy — it returns a view onto the same memory. Mutating the view mutates the original; retaining a tiny view retains the whole underlying allocation:

const big = Buffer.allocUnsafe(10 * 1024 * 1024);   // 10 MB
const header = big.subarray(0, 16);                  // 16-byte VIEW, not copy
header[0] = 0xff;                                    // big[0] is now 0xff too!
cache.set(id, header);          // [!code warning] // retains all 10 MB via the view
cache.set(id, Buffer.from(header));                  // ✅ real copy — 16 bytes retained

This is the closure-retention lesson (3.6.2) in byte form: a small reference can pin a large allocation. Buffer.from(view) / Buffer.copyBytesFrom make genuine copies. The rest of the daily toolkit: Buffer.concat(list) (join chunks — the standard "collect a stream" move), buf.copy(target), buf.equals(other) (byte equality — and for secrets, crypto.timingSafeEqual, Chapter 8.2's timing-attack defense), buf.indexOf(needle) (works for delimiters in protocols).

JSON note: JSON.stringify(buf) yields {"type":"Buffer","data":[…]} — array-of-numbers, ~4× the size. Ship binary as base64 in JSON, or better, don't put binary in JSON (multipart, raw bodies, or dedicated formats — 3.10).

5. The expert lens

Explicit encodings are a design position. Systems that guess text encodings (old HTTP, filesystems, Windows codepages — 1.4's mojibake museum) produced decades of corruption; Buffer's insistence — every bytes↔text crossing names its encoding — makes the conversion a visible, reviewable decision. Carry the discipline outward: any API you design that accepts "text" from bytes should demand the encoding rather than defaulting silently.

Views versus copies is a systems-wide dichotomy. subarray's zero-copy aliasing is the same trade as mmap vs read (2.5), slices in Go, &[u8] in Rust: views are fast and dangerous (aliasing, lifetime pinning), copies are safe and costly. Node picks view as the default for performance — so the burden of knowing which you hold is yours. The habit: at every function boundary, know whether you're passing ownership of bytes or a window into someone else's; copy at trust boundaries, view inside hot paths.

Buffers are where Node meets Part 1. Endianness, two's complement, UTF-8's variable width, base64 armor — the representation fundamentals of 1.3/1.4 stop being theory the day you parse a real protocol frame. Engineers fluent here debug "impossible" data corruption in minutes (wrong encoding? split character? aliased view? endianness?) because the bug space is a checklist, not a mystery.

Next: bytes in motion — 3.8.4: the four stream classes, backpressure mechanics, and why pipeline() is the production spelling.

Recall

• A Buffer is fixed-length raw memory (bytes 0–255), stored outside V8's heap (external memory), and is a Uint8Array subclass (1.4). No resizing — Buffer.concat allocates anew.
• Allocation: Buffer.alloc (zero-filled — default), Buffer.from (string+encoding / bytes / copy-of-view), Buffer.allocUnsafe (uninitialized old memory + pooled — only when every byte is provably overwritten; uninitialized-memory disclosure is a real vulnerability class).
• Encodings are explicit at every bytes↔text crossing: utf8 (default), hex/base64 (byte-to-text armor — JWTs, JSON), latin1, utf16le; wrong encoding = mojibake. Chunk-boundary bug: UTF-8 characters split across chunks corrupt under per-chunk toString — use StringDecoder/setEncoding, or decode once at the end.
• Structured access: read/writeUInt32BE/LE etc. — binary protocols are "integers at offsets + endianness." subarray/slice are views, not copies — mutations alias and small views pin whole allocations; Buffer.from(view) copies. equals for bytes, crypto.timingSafeEqual for secrets.

Self-test: Where do a Buffer's bytes live and where do they appear in memoryUsage()? When is allocUnsafe acceptable, and what's the risk class? Why does per-chunk toString("utf8") corrupt some users' text only sometimes — and what are the two fixes? What does retaining a 16-byte subarray of a 10 MB buffer cost? Which encodings are "armor" and where do you meet them?

Quiz Bank

FoundationalWhat is a Buffer, and how does it relate to TypedArrays and V8's heap?

A Buffer is Node's fixed-length container of raw bytes (values 0–255), created because JavaScript strings — UTF-16 character sequences — cannot hold arbitrary binary safely. Physically its bytes live outside V8's managed heap in externally-tracked memory (visible as process.memoryUsage().external), so huge buffers don't inflate GC scanning work (3.6.9). Since Node 4, Buffer subclasses Uint8Array — it is a standard TypedArray view over an ArrayBuffer (1.4) — adding server conveniences: explicit-encoding string conversion, offset readers/writers (readUInt32BE…), concat, compare. Length is fixed at allocation; growth = allocate + copy.

FoundationalBuffer.alloc vs Buffer.allocUnsafe — what's the difference and the rule?

Buffer.alloc(n) returns zero-filled memory — the safe default. Buffer.allocUnsafe(n) skips initialization: it hands you whatever bytes last occupied that memory (and small sizes come from a shared internal pool), which is faster but means the buffer may contain fragments of previous data — keys, tokens, request bodies. Reading it before overwriting is an information-disclosure bug (the uninitialized-memory vulnerability class). Rule: alloc unless you will provably overwrite every byte before any possible read — the pattern in tight loops that immediately fill/copy/write the full length. The deliberately alarming name is API design as security control: the dangerous path announces itself in code review.

AppliedWhy can decoding a stream chunk-by-chunk corrupt text, and what are the correct approaches?

UTF-8 encodes characters in 1–4 bytes (1.4), and stream chunk boundaries fall at arbitrary byte positions — so a chunk can end mid-character. chunk.toString("utf8") must then decode a dangling partial sequence, emitting the � replacement character; the next chunk's leading bytes decode wrong too. The bug is intermittent (boundaries move with network timing) and invisible in ASCII-only testing — classic "works for me, corrupts for non-English users." Fixes: (1) StringDecoder (string_decoder module) — its write(chunk) buffers incomplete trailing bytes until the next chunk completes them; (2) stream.setEncoding("utf8"), which installs a StringDecoder internally so data events yield safe strings; (3) best when possible: stay in bytes through the pipeline and decode once at the end (Buffer.concat(chunks).toString("utf8")), or let a proper parser stage handle it (3.8.4).

InterviewWhat does buf.subarray return, and what two bug classes follow?

A view — a new Buffer object sharing the same underlying memory; no bytes are copied (slice on Buffers behaves the same way, unlike Array.prototype.slice — a naming trap). Bug class 1: aliasing — writes through the view mutate the original (and vice versa), so "I modified my copy" corrupts data someone else is reading; any function that mutates a received buffer-view is mutating its caller's memory. Bug class 2: retention pinning — the view keeps the entire underlying allocation alive, so caching a 16-byte header view of a 10 MB upload retains 10 MB (3.6.2's reachability lesson in byte form); heap profiles show it as unexpectedly-retained ArrayBuffers. The fix for both is an explicit copy at the boundary — Buffer.from(view) — and the habit: at every API boundary, know whether you're handing over bytes or a window onto them; copy at trust/lifetime boundaries, view inside hot paths.

StaffAn upload service intermittently produces corrupted filenames with é→é style damage, and a memory profile shows gigabytes of retained ArrayBuffers though the app 'only keeps small metadata'. Connect both to this page and fix.

Two textbook Buffer pathologies. Corruption: é→é is the mojibake signature of UTF-8 bytes decoded as latin1 (1.4) — somewhere a bytes→text crossing omits or mis-names the encoding (toString("latin1"), a legacy default, or a multipart parser fed pre-decoded strings); if the damage is instead sporadic � characters, it's the chunk-boundary split — per-chunk toString("utf8") on the metadata stream. Audit every toString/setEncoding site: name utf8 explicitly, use StringDecoder for incremental decodes, and decode filenames exactly once, after fully assembling their bytes (multipart headers can themselves split across chunks — the parser must buffer).

Retention: "small metadata" was carved from big buffers with subarray/slice — views pinning each full upload's allocation; the tell in heap snapshots is small JS objects retaining huge ArrayBuffer backing stores. Fix: at the point metadata is extracted, copy (Buffer.from(view) or toString — a string is inherently a copy) and let the upload buffer die; add a lint/utility convention (copyForRetention()) so future extractions choose consciously.

Systemic guards: tests with multi-byte fixtures (é, 中, 👍 — never ASCII-only), a canary asserting external memory returns to baseline after N uploads, and — since uploads should stream to disk/S3 rather than assemble in RAM anyway — the 3.8.4 refactor that makes the retained-buffer class largely impossible. Principle:

name every encoding, copy at every lifetime boundary, and never let test data be ASCII.

Flashcards

FlashBuffer physically

Fixed-length raw bytes outside V8's heap (memoryUsage().external); a Uint8Array subclass; resize = concat/copy.

Flashalloc vs allocUnsafe

alloc: zero-filled, default. allocUnsafe: old memory + pooled — only when every byte is overwritten first; disclosure risk otherwise.

FlashEncodings that matter

utf8 (default), hex/base64 (byte→text armor: JWTs, JSON), latin1 (legacy), utf16le (JS internal). Wrong pick = mojibake.

FlashChunk-boundary rule

UTF-8 chars split across chunks ⇒ per-chunk toString corrupts. StringDecoder / setEncoding / decode-once-at-end.

Flashsubarray/slice

Views, not copies: mutations alias; small view pins whole allocation. Buffer.from(view) to truly copy.

FlashBinary protocol toolkit

read/writeUInt32BE/LE at offsets (BE = network order), indexOf for delimiters, equals for bytes, timingSafeEqual for secrets.

Scenario Drill

DrillImplement the framing layer for a TCP protocol where each message is: 4-byte big-endian length, 1-byte type, then a UTF-8 JSON payload of that length. Messages arrive fragmented and concatenated arbitrarily. List every Buffer concept the correct implementation must exercise, and sketch it.

TCP is a byte stream — Part 5 — so framing is your job: chunks arrive split mid-header, mid-payload, or with several messages glued together; correctness means buffering until a complete frame exists, repeatedly. Concepts exercised: accumulation with Buffer.concat; structured reads (readUInt32BE(0) for length — big-endian because the protocol says network order, 1.3); view-vs-copy discipline (subarray to examine the frame cheaply, explicit copy only if a frame outlives the parse); one-shot UTF-8 decode of the complete payload (never per-chunk — section 3's boundary rule handled by construction, since we decode only whole frames); and defensive limits. Sketch:

const HEADER = 5;                       // 4B length + 1B type
const MAX_FRAME = 16 * 1024 * 1024;     // refuse absurd lengths — see below
let acc = Buffer.alloc(0);

socket.on("data", (chunk) => {
  acc = acc.length ? Buffer.concat([acc, chunk]) : chunk;
  while (acc.length >= HEADER) {
    const len = acc.readUInt32BE(0);
    if (len > MAX_FRAME) return socket.destroy();     // poisoned length ≠ OOM
    if (acc.length < HEADER + len) break;             // incomplete — wait for more
    const type = acc[4];
    const payload = acc.subarray(HEADER, HEADER + len);  // view: parse in place
    handle(type, JSON.parse(payload.toString("utf8")));  // whole-frame decode ✅
    acc = acc.subarray(HEADER + len);                 // advance past the frame
  }
  if (acc.length === 0) acc = Buffer.alloc(0);        // drop the view → free memory
});

The senior notes: the MAX_FRAME check is non-negotiable — a hostile/corrupt 4-byte length of 4 GB otherwise instructs your server to buffer 4 GB (self-inflicted DoS; validate before allocating — trust-boundary rule, Chapter 8.5). The trailing acc = acc.subarray(...) is a view — fine while parsing continues, but note the final reassignment: when fully drained, replace it with a fresh empty buffer so the view doesn't pin the last big allocation (section 4's retention trap); high-throughput versions replace concat-per-chunk with a chunk list + running length, concat-ing only when a frame completes (O(n) vs O(n²) accumulation). handle receiving parsed JSON should validate it at this trust boundary (3.7.7). And the test fixtures must include: a frame split inside the 4-byte header, two frames in one chunk, a payload containing 👍 split across chunks, and a length field of 0xFFFFFFFF — the four ways real networks break naive framers.