11.29 Configuration & Feature-Flag Service

A configuration and feature-flag service is the remote control for your distributed system. It allows engineers and product managers to alter system behavior, roll out new features gradually, or immediately disable failing components without deploying code.

Underneath, it is a store of small, versioned rule sets that is read far more than it is written. The rules are not evaluated by the flag service at all: each application downloads them and evaluates them itself, in memory, against the user making the request. That one decision explains most of this page, and it comes with a hard requirement: if the flag service goes down, every application must keep working with the rules it already has, or with safe defaults.

Requirements

The company here runs about 300 services on 8,000 server instances, plus web and mobile apps with 5 million monthly users.

Functional. Create flags with targeting rules and percentage rollouts. Keep separate settings per environment (development, staging, production). Evaluate flags inside each application. Push changes out quickly. Offer a kill switch that turns a feature off everywhere at once. Require approval for production changes, and record every change. Help remove flags once they are fully rolled out.

Non-functional, with numbers.

• Evaluating a flag takes microseconds and makes no network call.
• A change reaches every server within 10 seconds at p99.
• An outage of the flag service changes no flag's result. Applications keep their last known rules.
• Every production change is attributable: who, what, when, and which approval.

Out of scope today: experiment analysis (which variant won), which belongs to an analytics system.

The clarifying questions, and what each answer changes

"Are flags evaluated on servers, in browsers and phones, or both?" Servers can download every rule. Browsers and phones must not: rules often contain lists of customer ids and internal names, and anything shipped to a phone is public. Client-side flags therefore need a different path, where the server evaluates the flags for one user and sends only the answers.

"What should happen when an application starts and cannot reach the flag service?" The answer decides whether every flag needs a coded default, and whether a stale copy is kept on local disk.

"Who can change production flags?" If the answer involves compliance, changes need approvals and a reviewable history, which turns a toggle into a small workflow (11.23).

"How quickly must a kill switch work?" "Within seconds" rules out polling every few minutes and forces a push channel.

"How many flags will there be in two years?" Usually far more than anyone expects, because nobody deletes them. The answer decides whether ruleset size per application is a problem.

Estimation

Evaluation rate. The backend serves about 200,000 requests a second in total, and a typical request checks about 10 flags: 2 million evaluations a second, about 250 a second on each of the 8,000 instances. What that forces: local evaluation is not an optimisation, it is the only option. A central flag service would need to answer 2 million requests a second, and each request would add network time ten times over. Evaluated locally at about 2 microseconds each, 250 evaluations cost an instance half a millisecond of CPU per second.

Writes. About 200 flag changes a day, so 200 ÷ 86,400 ≈ 0.002 a second. Against 2 million evaluations a second that is a read-to-write ratio of nearly a billion to one. What that forces: the write side is a single PostgreSQL database with careful auditing; no part of it needs to scale.

Ruleset size. 3,000 flags × about 2 KB of rules = about 6 MB per environment. What that forces: a server keeps the whole ruleset in memory easily. A phone app downloading 6 MB at every launch would not, which is the second reason client flags are evaluated on the server.

Client connections. 5 million monthly users means perhaps 200,000 app sessions open at the busiest moment. A streaming connection for each would be 200,000 long-lived connections. Fetching evaluated flags (about 2 KB) at launch and then every 5 minutes is 200,000 ÷ 300 ≈ 670 requests a second. What that forces: servers stream (8,000 connections is nothing); clients poll an endpoint that is cheap and cacheable for a few seconds. The kill switch still reaches clients within one poll interval, which the product team must agree is acceptable.

API

PUT /flags/new-checkout/environments/production
If-Match: "v12"
{ "rules": [ { "attribute": "isBetaTester", "operator": "EQUALS", "value": true },
             { "attribute": "userId", "operator": "PERCENTAGE", "value": 10 } ],
  "defaultValue": false,
  "reason": "Widen beta to 10% of users" }

→ 202 Accepted   { "approvalRequestId": "apr_311", "status": "pending_approval" }    (production)
→ 200 OK         { "version": 13 }                                                 (other environments)
→ 412 Precondition Failed                                                          (someone changed v12 first)

GET /sdk/rulesets/production
Authorization: Bearer sdk_server_…
If-None-Match: "r-48812"
→ 304 Not Modified                         (nothing changed)
→ 200 OK  ETag: "r-48813"  { "version": 48813, "flags": [ … ] }

GET /sdk/stream/production                 (Server-Sent Events)
event: ruleset-changed
data: {"version": 48814}

POST /client/evaluate                      (browsers and phones)
Authorization: Bearer sdk_client_…
{ "userId": "u_52", "country": "IN", "appVersion": "8.3.0" }
→ 200 OK  Cache-Control: private, max-age=30
{ "flags": { "new-checkout": true, "dark-mode": false } }

POST /flags/payments-provider-b/kill
→ 200 OK  { "version": 48815, "value": false }      (no approval needed; audited)

If-Match prevents lost updates. Two people editing the same flag at once would otherwise overwrite each other's rules silently (9.6.3).

The stream sends only a version number. On receiving it, the SDK fetches the ruleset with If-None-Match. This keeps the stream tiny, and an SDK that missed messages while disconnected simply fetches the latest version when it reconnects; nothing has to be replayed.

Client SDKs never see rules. /client/evaluate returns answers for one user, so customer lists and internal flag logic stay on the server.

The kill switch skips approval on purpose. Turning a broken feature off must take seconds; it is still recorded in the audit log, and turning it back on goes through the normal approval.

Architecture

Reads outnumber writes by nearly a billion to one (derived in the estimation above). Updates happen rarely, when someone changes a flag, but reads happen constantly, because every user request evaluates several flags.

To handle this, the architecture uses a pull-with-push-invalidation model combined with local evaluation.

Configuration and Feature-Flag Architecture

Instead of the application making a network call to the flag service for every request (which would add latency and destroy availability), the SDK fetches the entire ruleset on startup and caches it in memory. The SDK then evaluates flags locally. When a flag changes, the Control Plane pushes the update via Server-Sent Events (SSE) or gRPC streaming to the SDKs, which update their in-memory cache.

Data Model & Versioned Configuration

Every change to a configuration must be versioned. If a developer breaks the system with a bad config, you need a one-click rollback to the previous version.

CREATE TABLE flags (
    flag_key VARCHAR(255) PRIMARY KEY,
    description TEXT,
    created_at TIMESTAMP
);

CREATE TABLE flag_rules (
    flag_key VARCHAR(255) REFERENCES flags(flag_key),
    version INT,
    rules_json JSONB, -- The rollout logic
    updated_by VARCHAR(255),
    updated_at TIMESTAMP,
    PRIMARY KEY (flag_key, version)
);

The two tables above leave out two things every real flag service needs: environments and approvals.

CREATE TABLE flag_environment_rules (
    flag_key      VARCHAR(255) REFERENCES flags(flag_key),
    environment   VARCHAR(32),             -- development | staging | production
    version       INT,
    rules_json    JSONB NOT NULL,
    default_value BOOLEAN NOT NULL,
    updated_by    VARCHAR(255),
    approval_id   VARCHAR(64),             -- required when environment = 'production'
    updated_at    TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (flag_key, environment, version)
);

CREATE TABLE ruleset_versions (            -- one row per published change, per environment
    environment   VARCHAR(32),
    version       BIGINT,                  -- the number SDKs see in the stream
    published_at  TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (environment, version)
);

Access patterns:

Query	Frequency	Returns
Current ruleset for an environment	on every SDK fetch (cached)	~6 MB
History of one flag in one environment	rare (incident review)	tens of rows
Roll a flag back to an earlier version	rare	one new row
Flags at 100% for over 14 days	daily (cleanup bot)	a list

Never update a rules row; insert a new version. Rollback is then "publish version N−1's rules again as version N+1", and the full history stays intact for audit.

ruleset_versions gives every environment one ever-increasing number. That is what the stream announces and what the ETag carries, so an SDK can tell in one comparison whether it is up to date.

The Evaluation Engine & Rollout

A feature flag isn't just true or false. It is a set of rules evaluated top-to-bottom. Common rules include:

1. Targeting: Is the user in the "beta-testers" group?
2. Percentage Rollout: Is the user in the 10% of users receiving the feature?

To ensure a user consistently gets the same experience (you don't want a UI button blinking in and out of existence on every refresh), we use deterministic hashing.

import { createHash } from 'node:crypto';

interface FlagContext {
  userId: string;
  country: string;
  isBetaTester: boolean;
}

interface Rule {
  attribute: keyof FlagContext;
  operator: 'EQUALS' | 'IN' | 'PERCENTAGE';
  value: any;
}

interface Flag {
  key: string;
  version: number;
  rules: Rule[];
  defaultValue: boolean;
}

class FeatureFlagSDK {
  private flags: Map<string, Flag> = new Map();

  // Called locally, zero network latency
  public evaluate(flagKey: string, context: FlagContext): boolean {
    const flag = this.flags.get(flagKey);
    
    // Kill Switch / Safety Fallback: If flag doesn't exist, return safe default
    if (!flag) return false; 

    for (const rule of flag.rules) {
      if (rule.operator === 'EQUALS') {
        if (context[rule.attribute] === rule.value) return true;
      }
      
      if (rule.operator === 'PERCENTAGE') {
        // Deterministic routing based on User ID and Flag Key
        // This ensures the same user always falls into the same bucket for this specific flag
        const hashKey = `${flagKey}-${context.userId}`;
        const hashResult = createHash('sha1').update(hashKey).digest().readUInt32BE(0); // first 4 bytes as a number
        const bucket = hashResult % 100; // 0 to 99
        
        if (bucket < rule.value) return true;
      }
    }

    return flag.defaultValue;
  }
}

Why hash, and why the flag key goes into the hash

Hashing gives every user a fixed bucket from 0 to 99 without storing anything: the same user and flag always produce the same number, so the button does not flicker between refreshes. Any well-mixed hash works; this code uses SHA-1 from Node's standard library because it is already there, and nothing here depends on SHA-1's security, only on its output being evenly spread. (MurmurHash3, which many flag SDKs use, is faster, but at about 2 microseconds per evaluation the difference does not matter.) We put the flagKey in front of the userId before hashing so that a user is not stuck in the "first 10%" bucket for every feature flag. Measured over 100,000 users: a 10% rollout selects 10,018 of them; raising it to 20% keeps every one of those 10,018 and adds about 10,000 more, so a rollout only ever grows; and the first 10% of two different flags overlap by 1,021 users, almost exactly the 1,000 you would expect if the two choices were independent.

Caching, Consistency, and Kill Switches

Multi-Tier Caching

1. L1 (In-Memory): The SDK stores the latest rules payload in memory. Evaluation takes microseconds.
2. L2 (Local Agent / Sidecar): In containerized environments (Kubernetes), a sidecar container fetches the rules and provides them to the application via localhost. This prevents massive memory bloat if you have thousands of Node.js processes, and allows the application to restart without re-fetching from the network.
3. L3 (Redis / CDN): When SDKs boot up, they fetch the initial ruleset from a CDN (for frontend clients) or Redis (for backend services), avoiding load on the primary DB.

Consistency Guarantee

Feature flags require eventual consistency. If you toggle a flag, it is acceptable if it takes 5-10 seconds to propagate to all nodes globally. We use Server-Sent Events (SSE) to push invalidations.

Kill Switches

A kill switch is a high-priority feature flag designed to turn off a misbehaving component (e.g., disabling a 3rd-party payment gateway that is timing out). When the SDK cannot reach the flag service (e.g., network partition), it must not crash. It must rely on its last known cached state. If it boots with no cache, it must use hardcoded defaultValues (fail-safe).

Audit & Compliance

Because configuration changes instantly alter production behavior, they are inherently risky. Every write operation must be appended to an immutable audit log detailing who changed what and when.

// Audit log structure appended to a Kafka topic and stored in cold storage (S3)
interface AuditEvent {
    eventId: string;
    actor: string; // "indranil@company.com"
    action: "FLAG_CREATED" | "FLAG_UPDATED" | "FLAG_DELETED";
    flagKey: string;
    previousState: object;
    newState: object;
    timestamp: Date;
    approvalRequestId?: string; // Link to the Jira/ServiceNow ticket for SOX compliance
}

Scale and failure

What breaks	Blast radius	How you find out	What keeps it running	Recovery
Flag service down	no new changes reach anyone	SDK fetch errors; stream disconnects	last known rules in memory, and on disk	SDKs refetch when it returns
A bad rule published	every user it targets	error rate jumps right after a version change	one-click rollback; kill switch	publish the previous version again
An application starts with no copy of the rules	that instance	startup log; "defaults in use" metric	coded defaultValue for every flag	it picks up rules on first successful fetch
Stream silently stalls	changes stop arriving	ruleset version age per instance	periodic fetch as a backstop	reconnect and fetch latest
Rules leaked to clients	customer lists exposed	review of what client SDKs receive	server-side evaluation for clients	rotate keys; remove sensitive targeting
Thousands of dead flags	slow reviews, risky code paths	count of flags at 100% for 14+ days	cleanup bot opens removal PRs	delete the flag after the code is gone

Alarm on ruleset version age, per instance. "This server is 40 versions behind" is the signal that a kill switch will not reach it, and it is only visible if every SDK reports the version it is running.

What the interviewer will push on

1. How do you handle massive rulesets? If you have 100,000 flags, the memory payload becomes too large for a client-side SDK. Answer: Segment flags by application. A mobile app only needs mobile flags. Use environment-specific API keys to fetch only the relevant subset.

2. What happens during a split-brain network partition?Answer: The SDKs stop receiving SSE updates. They continue evaluating based on their stale L1 cache. Availability > Consistency. We prioritize keeping the app running over having the exact latest configuration.

3. How do you prevent flag staleness?Answer: Over time, codebases become littered with if (flag) { new } else { old }. This is technical debt. Introduce automated workflows: once a flag reaches 100% rollout for 14 days, a bot opens a Pull Request to remove the else block and delete the flag from the service.

Recall

• Core Metric: reads outnumber writes by nearly a billion to one, so evaluation is local and writes are rare and audited.
• Evaluation: Happens locally in the SDK for zero network latency.
• Rollout: a stable hash of flagKey + userId (SHA-1 here, MurmurHash3 in many SDKs) maps each user to a fixed bucket from 0 to 99.
• Updates: SDKs pull initial state, then listen for Server-Sent Events (SSE) to update in-memory caches.
• Failure Mode: Default to last-known good state or hardcoded safe defaults. Never block the critical path.

Quiz

1. Why is the flagKey combined with the userId before hashing in the percentage rollout logic?
2. Why is Server-Sent Events (SSE) preferred over standard polling for updating feature flags in the SDK?
3. If the centralized PostgreSQL database goes down entirely, what is the immediate impact on the application's ability to evaluate feature flags?