11.26 Document Management & Processing

When a user uploads a multi-page PDF invoice, your system cannot simply "save it to disk" and call it a day. A production-grade document management system must securely store the bytes, extract the text for search, scan for malicious payloads, generate thumbnails, and strictly enforce who is allowed to read it. Furthermore, it must perform all of this heavy lifting asynchronously without blocking the user's web request.

Requirements

The product here stores contracts and invoices for 3,000 business customers with about 50 users each.

Functional. Upload files of any size, including scanned PDFs. Scan every file for malware before anyone can open it. Extract text, including by OCR from scanned pages, and make it searchable. Keep every version of a document. Share a document with people or groups. Place documents under legal hold so they cannot be deleted. Download with a permission check every time.

Non-functional, with numbers.

• No file can be downloaded before its virus scan has passed. Not "usually": the download path checks.
• A document of up to 50 pages is searchable within 2 minutes of upload, at p95.
• The permission check on a download takes under 100 ms, because it runs on every open.
• Files are never lost. Object storage provides the durability (S3 is designed for 99.999999999% per object per year); our job is not to lose the metadata that points at them.

Out of scope today: in-browser editing and e-signature.

The clarifying questions, and what each answer changes

"What is the largest file?" Above 5 GB a single upload request is impossible on S3, and above roughly 100 MB a single request is fragile on an ordinary connection. The answer decides when the client switches to multipart upload.

"Are the documents typed PDFs or phone photos of paper?" Typed PDFs already contain text, which is cheap to extract. Scans need OCR, which is the most expensive thing this system does. The mix decides the size of the worker fleet.

"Who can see a document: everyone at the customer, or named people?" Per-document permissions mean a permission row per share and a check on every download and every search result.

"Are there legal retention rules?" A legal hold that blocks deletion changes the delete path from "delete" to "mark, check holds, then delete", and may need storage-level locks.

"Must search respect permissions?" Always yes, and it is worth asking anyway, because a search index that returns a snippet from a contract the user cannot open has leaked the contract.

Estimation

Upload rate. 3,000 customers × 50 users × 20 documents a month = 3 million documents a month, about 100,000 a day. Divided by 100,000 seconds that is about 1 a second, and perhaps 10 a second at month-end, when invoices arrive together. What that forces: the API servers are almost idle; what they do is sign upload URLs. All the heavy work happens after the bytes land.

Storage. An average of 2 MB per file is 200 GB a day, about 73 TB a year, and with about 30% extra for new versions, about 95 TB a year. What that forces: the bytes live in object storage with lifecycle rules, and the database holds only metadata: 100,000 rows a day at about 1 KB is a mere 100 MB a day.

OCR, the expensive part. Say an average document has 5 pages and OCR costs about 2 CPU-seconds a page (a rough figure for a Tesseract-class engine on a scanned page; measure your own). That is 10 CPU-seconds per document: about 10 cores busy on average and about 100 at the month-end peak. What that forces: OCR runs on its own autoscaled worker pool, separate from virus scanning. If both shared one queue, a month-end OCR backlog would also delay scanning, and no file can be opened until its scan is done.

Search index. About 3 KB of text per page × 5 pages = 15 KB per document, so 1.5 GB of text a day. With index overhead of about 2×, about 3 GB a day, roughly 1.1 TB a year. What that forces: a search cluster sized for steady growth, with the index split by customer so a query only touches its own customer's shard.

Downloads. About 10 opens per upload: 1 million a day, about 10 a second. What that forces: the API does the permission check and returns a short-lived pre-signed URL; the bytes go straight from object storage to the browser.

API

POST /documents
{ "name": "Lease-2026-Unit-4B.pdf", "sizeBytes": 3481920,
  "contentType": "application/pdf", "sha256": "9f2c…e1" }

→ 201 Created
{ "documentId": "doc_8c1", "revisionId": "rev_1",
  "upload": { "method": "PUT", "url": "https://bucket.s3…?X-Amz-Signature=…",
              "expiresAt": "2026-10-18T10:15:00Z" } }

For files over 100 MB the response instead contains { "uploadId": "…", "partSize": 16777216 } and the client asks for one signed URL per part, as in the multipart client of 10.23.

POST /documents/doc_8c1/revisions/rev_1/complete
→ 202 Accepted   { "status": "SCANNING" }

GET /documents/doc_8c1
→ 200 OK  { "documentId": "doc_8c1", "status": "READY", "currentRevision": "rev_1",
            "pages": 7, "searchable": true }

GET /documents/doc_8c1/content
→ 302 Found     Location: https://bucket.s3…?X-Amz-Expires=300&…
→ 409 Conflict  { "error": { "code": "NOT_READY", "message": "Virus scan in progress.",
                             "requestId": "req_4d1" } }
→ 403 Forbidden                              (no read permission)

GET /search?q=termination+clause&limit=20&cursor=…   → only documents the caller may read

POST /documents/doc_8c1/shares   { "principal": "group:legal", "permission": "read" }

The client sends the file's SHA-256 up front. After upload the worker recomputes it, so a truncated or corrupted upload is caught before the document is marked ready, and identical re-uploads can be recognised.

complete returns 202. Scanning and OCR take seconds to minutes, so the response says "accepted, currently scanning", and the client polls GET /documents/{id} or receives a webhook.

A download before the scan finishes is a 409, not a 404. The document exists; it is just not in a state where it can be served, and the response says why.

Data model

documents
  customer_id    UUID NOT NULL
  id             UUID                     -- PRIMARY KEY (customer_id, id)
  name           TEXT NOT NULL
  owner_id       UUID NOT NULL
  current_revision UUID
  legal_hold     BOOLEAN NOT NULL DEFAULT false

document_revisions
  customer_id, document_id, id UUID
  version_number INT NOT NULL             -- UNIQUE (document_id, version_number)
  s3_key         TEXT NOT NULL            -- never reused; objects are immutable
  size_bytes     BIGINT, sha256 BYTEA, page_count INT
  status         TEXT NOT NULL            -- UPLOADING | SCANNING | READY | QUARANTINED | ERROR
  created_at     TIMESTAMPTZ NOT NULL

permissions
  customer_id, document_id
  principal_type TEXT, principal_id UUID  -- user | group
  permission     TEXT                     -- read | write | share
  PRIMARY KEY (customer_id, document_id, principal_type, principal_id)

Access patterns:

Query	Frequency	Returns
Can user U read document D	10/s, every download	yes or no
A user's recent documents	moderate	one page
Revisions of one document	low	a few rows
Revisions stuck in SCANNING	every minute (reconciler)	usually none
Documents under legal hold	rare (deletion, audits)	a list

Partition by customer_id. Every query is scoped to one customer, including search, and the search index is split the same way so the permission filter never has to scan other customers' documents.

The permission check reads permissions plus the user's groups, both small and cacheable for a few seconds, which keeps it well under 100 ms. The status check (READY) is on the revision, because a new version that is still scanning must not replace the previous, already-scanned one in the download path.

Separation of Storage and Metadata

The foundational rule of document management is to strictly separate the unstructured file bytes from the structured data that describes them.

Object Storage (The Bytes)

The physical file is stored in an Object Storage system like AWS S3, Google Cloud Storage, or MinIO. Object storage provides virtually infinite scalability, built-in redundancy, and automatic lifecycle management (e.g., moving cold files to cheaper, slower storage tiers like Glacier). You never store large files directly in your database.

Metadata Database (The Brain)

Information about the file—its human-readable name, the owner's user ID, its size, MIME type, and processing status—lives in a relational database (like PostgreSQL). This allows you to quickly query, sort, and paginate a user's documents without ever touching the actual file bytes.

Here is how the initial upload sequence works securely using Pre-signed URLs, which prevents your API servers from becoming a bottleneck during large file transfers:

Upload Sequence Diagram (Separating Storage and Metadata)

The Processing Pipeline

Once the bytes land in S3, they must be processed. Because document processing (especially OCR and virus scanning) is highly CPU-intensive and unpredictable in duration, it must be handled by an asynchronous worker pool, often coordinated as a Directed Acyclic Graph (DAG) of tasks.

Asynchronous Document Processing Pipeline

TypeScript Implementation: Pipeline Worker

Here is how an event-driven worker processes an uploaded document. We use a step-function approach, ensuring that if a specific step (like OCR) fails, it can be retried independently.

import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { scanBuffer } from "clamav.js"; // Fictional ClamAV wrapper
import { extractText } from "./ocr"; // our wrapper: PDF pages → images → OCR (e.g. Tesseract)
import { db } from "./db";
import { elastic } from "./elasticsearch";

// Represents a message received from our Queue (e.g., SQS or RabbitMQ)
interface DocumentUploadedEvent {
  documentId: string;
  s3Bucket: string;
  s3Key: string;
}

export async function processDocument(event: DocumentUploadedEvent): Promise<void> {
  const { documentId, s3Bucket, s3Key } = event;
  
  try {
    // 1. Fetch the file bytes into memory (assume small files for this example; 
    //    in reality, use streams for large files)
    const fileBuffer = await downloadFromS3(s3Bucket, s3Key);

    // 2. Virus Scanning
    const isClean = await scanBuffer(fileBuffer);
    if (!isClean) {
      await db.documents.update(documentId, { 
        status: "FAILED_QUARANTINE" 
      });
      // Move to quarantine bucket, alert security, etc.
      return;
    }

    // 3. OCR / Text Extraction
    // This is computationally expensive.
    const extractedText = await extractText(fileBuffer, { language: 'eng' });

    // 4. Index for Full-Text Search
    await elastic.index({
      index: "documents",
      id: documentId,
      document: {
        content: extractedText,
        // Embeddings could also be generated here for semantic search
      }
    });

    // 5. Mark as complete
    await db.documents.update(documentId, { 
      status: "READY" 
    });

  } catch (error) {
    console.error(`Failed to process ${documentId}:`, error);
    await db.documents.update(documentId, { 
      status: "ERROR", 
      errorMessage: error.message 
    });
    throw error; // Let the queue retry mechanism handle it
  }
}

async function downloadFromS3(bucket: string, key: string): Promise<Buffer> {
  // Implementation omitted for brevity
  return Buffer.from("");
}

Permissions and Access Control

Because document management systems often store sensitive data (legal contracts, medical records), access control must be rigorous.

Do not rely on obscuring S3 URLs (security by obscurity). Instead, block all public access to the S3 bucket. When a user requests a file:

1. The API verifies the user's session.
2. The API queries the metadata DB to check if the user has READ permission for the specific documentId.
3. If authorized, and the document's status is READY, the API generates a Pre-signed GET URL with a short expiration (e.g., 5 minutes) and returns it to the client. The status check matters: a file that has not finished its virus scan must not be downloadable, or the scanner protects nobody.
4. The client browser uses that temporary URL to fetch the file directly from S3.

Document Versioning

In systems like Google Drive or SharePoint, users repeatedly edit and re-upload documents.

Infrastructure Level (S3 Versioning): Cloud providers offer bucket-level versioning, meaning every overwrite is kept. This is great for disaster recovery (preventing accidental deletion), but it is a poor fit for application-level features because the IDs are opaque to your database.

Application Level (The Right Way): Handle versioning in your metadata DB. Treat documents as a parent entity, and file uploads as child "revisions."

CREATE TABLE documents (
    id UUID PRIMARY KEY,
    name VARCHAR(255),
    owner_id UUID
);

CREATE TABLE document_revisions (
    id UUID PRIMARY KEY,
    document_id UUID REFERENCES documents(id),
    s3_key VARCHAR(255),
    version_number INT,
    created_at TIMESTAMP
);

When a user uploads a "new version", you insert a new row in document_revisions and write a brand new object to S3 using a unique s3_key. The S3 objects themselves are entirely immutable.

---

Scale and failure

What breaks	Blast radius	How you find out	What keeps it running	Recovery
Month-end OCR backlog	search freshness	age of the oldest OCR job	separate OCR pool; scanning unaffected	autoscale on queue age; it drains
Storage event lost	a file stuck in SCANNING	reconciler finds old SCANNING rows	the client's complete call as a second trigger	reconciler re-queues the job
Malformed PDF crashes the worker	that worker, then the next	repeated failures of one job	attempt counter and dead-letter queue	engineer inspects; file marked ERROR
Scanner signatures out of date	malware marked clean	signature-age metric	rescan recent files after an update	re-scan the window, quarantine hits
Pre-signed URL shared outside the company	one file, until expiry	none at the time	5-minute expiry	shorten expiry for sensitive classes
Search returns a document the user cannot open	a contract leaks through a snippet	permission audit of results	filter results by permission	re-index permissions with documents

What the interviewer will push on

How do you handle massive files (e.g., 50GB video files)? A single pre-signed PUT cannot do it: S3 accepts at most 5 GB in one upload request, and one dropped connection near the end restarts the whole transfer. Streaming it through your API servers is worse, because every gigabyte crosses your network twice and ties up a server for the whole upload. Use S3 Multipart Upload. The client splits the file into parts, asks your API for a pre-signed URL per part, uploads the parts in parallel directly to S3, retries only the parts that fail, and finally sends a "complete" request so S3 joins them. Size the parts from the file: S3 allows at most 10,000 parts and every part except the last must be at least 5 MiB, so a 50 GB file needs parts of a little over 5 MiB, and a fixed 5 MiB part size would be rejected. The worked client code is in 10.23.

How do you handle "poison pills"? A user uploads a malformed PDF that causes the C-bindings in your OCR library to segfault, crashing the worker process. If your queue automatically retries failed messages, the message goes back to the queue, gets picked up by another worker, and crashes that one too, eventually taking down your entire cluster. Solution: Implement a Dead Letter Queue (DLQ). If a message crashes or fails processing 3 times, move it out of the main queue into a DLQ for manual engineering inspection.

How do you optimize costs? S3 standard storage gets expensive fast. Implement an S3 Lifecycle Rule to automatically transition objects older than 90 days (which are rarely read) to S3 Standard-Infrequent Access (IA), and objects older than 1 year to Glacier. Your database metadata remains intact, but retrieving an old file might require warning the user that "This file will be available in 3-5 hours."

---

Recall

• Storage vs Metadata: Never store files in a database. Put files in object storage (S3) and metadata in PostgreSQL.
• Pre-signed URLs: Use them to offload upload and download bandwidth entirely to the object store, keeping your API servers free.
• Asynchronous Pipeline: Heavy processing (OCR, virus scanning, indexing) must happen asynchronously via a message queue to prevent blocking the user request.
• Immutability: Treat S3 objects as immutable. For versioning, write a new file to S3 and add a new "revision" row in your database.

Quiz:

1. Why is relying on S3 bucket-level versioning usually insufficient for a document management application?
2. What is a "poison pill" in the context of a document processing pipeline, and how do you protect your system from it?
3. What is the fundamental mechanism used to allow clients to download private files securely without funneling the bytes through your API server?