8.6.1 — Secrets, Keys and Rotation

A developer commits a cloud access key to a public repository at 14:02. At 14:06 it is used to launch compute instances for cryptocurrency mining in three regions. The bill reaches five figures before anyone notices.

The team's first response is to delete the commit and force-push. That is the wrong first action, and it is the one almost everyone takes. The key is already in someone's clone, in a fork, in the platform's event feed, and in the caches of the automated scanners that watch public commits continuously. The only action that helps is revoking the key, and every minute spent rewriting history is a minute it stays valid.

1. The lifecycle

A secret is any value that grants access: passwords, API keys, database credentials, private keys, signing keys, tokens, connection strings, webhook signing secrets, encryption keys.

Six stages, and most organisations only think about two.

Generate — from a CSPRNG (Chapter 8.2.1), long enough that guessing is hopeless. Distribute — get it to the workload without it landing somewhere permanent. Store — encrypted, access-controlled, audited. Use — in memory, ideally never written to disk. Rotate — replace it on a schedule and on demand. Revoke — make the old value useless immediately.

Rotation and revocation are the neglected two, and they are the ones that matter during an incident. A secret you cannot rotate quickly is a secret you cannot respond to.

2. Where secrets must not be

Source control. Including private repositories — they get cloned, forked, made public by accident, and backed up. And history is forever: a secret committed once and removed in the next commit is still in the history, in every clone, and in every fork.

Container images. A build argument or a copied .env file is in a layer, and layers are extractable from any registry the image reaches.

CI logs. An echoed variable, a debug dump, a failing test printing its environment. Masking helps and is not complete — a base64-encoded or partially printed secret defeats a naive mask.

Chat, tickets, wikis, email. They are searchable, long-lived, and shared with people who join later.

Client-side code. Anything shipped to a browser or a mobile app is public. A "secret" in a mobile binary is not a secret — it is obfuscated at best.

Kubernetes Secrets, without extra configuration. They are base64-encoded, not encrypted, and readable by anyone with API access to that namespace. Enable encryption at rest for etcd, apply proper access control, and prefer an external store.

3. Environment variables, honestly

Environment variables are the common answer and are better than source control and worse than a secrets manager. The specific weaknesses:

• They are readable at /proc/<pid>/environ by anyone who can read that process.
• They are inherited by child processes, so a shelled-out tool receives every secret.
• They appear in crash dumps and in some error reporters' automatic context capture.
• Many platforms display them in a console and in deployment configuration.
• They are set once at start, so rotation requires a restart.

Use them for non-secret configuration and for the one credential needed to reach the secret store, and prefer mounted files or a runtime API for the rest. A file can have restrictive permissions, can be updated without a restart, and is not inherited by children.

4. Secret managers

HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, AWS Parameter Store. They give centralised storage, encryption at rest, fine-grained access control, an audit trail of every read, versioning, and an API rather than a file.

Vault's genuinely different idea is dynamic secrets, and it is worth understanding even if you use something else. Rather than storing a database password, Vault holds credentials that let it create database users. An application asks for database access and receives a freshly generated username and password with a one-hour lease.

What that changes:

• There is no long-lived shared credential to leak.
• Every application instance has distinct credentials, so an audit log shows exactly which one did what.
• Revocation is immediate — Vault deletes the user.
• Rotation stops being an event, because nothing lives long enough to need it.

Vault's transit engine is the second useful piece: encryption as a service, where the application sends plaintext and receives ciphertext without ever holding the key. That makes key rotation a Vault-side operation.

A managed cloud service is the pragmatic default for most teams — Vault is powerful and is a system you then operate, with its own availability and unsealing procedures.

5. Key management and envelope encryption

For encryption keys specifically, a key management service (AWS KMS, Azure Key Vault, Google Cloud KMS) holds the master key and never releases it. You send data to be encrypted or a key to be wrapped; the key itself never leaves, and for hardware-backed keys it never exists outside a hardware security module.

Envelope encryption is the pattern (introduced in Chapter 8.2.1):

1. Ask the service for a new data key. It returns the key in plaintext and wrapped under the master key.
2. Encrypt your data locally with the plaintext data key — fast, and no size limit.
3. Store the wrapped key alongside the ciphertext, and discard the plaintext key from memory.
4. To decrypt, send the wrapped key back to be unwrapped.

Why this shape: the expensive, audited, rate-limited service handles only small keys; bulk encryption happens locally at full speed. And rotating the master key means re-wrapping small keys, not re-encrypting terabytes.

Two details that matter in practice. Every KMS call is logged, which gives you an audit trail of decryption attempts — a spike in unwrap calls is a strong exfiltration signal, and it is one of the few detections that works after an attacker already has valid credentials. And key policies are the actual access control: the ability to decrypt is the ability to read the data, so the policy on the key deserves more scrutiny than the policy on the storage bucket.

6. Rotation without downtime

Rotation limits the window in which a leaked credential is useful. Its value is largely that the ability to rotate fast is what you need during an incident — a system that cannot rotate without a coordinated deployment cannot respond.

The pattern that works is two active credentials:

1. Create a second credential; both are valid.
2. Deploy configuration pointing at the new one.
3. Confirm nothing is still using the old one — usage metrics, not assumption.
4. Disable the old one.
5. Delete it after a waiting period.

This is expand-and-contract (Chapter 7.2.4) applied to credentials, and it is why services offer two keys — Azure's "Key 1 and Key 2", storage account keys, API key pairs. The second key exists precisely so rotation is not a cutover, which is the answer to "why are there two".

For encryption keys, rotation means re-wrapping, and it only works if ciphertext carries a key version — the version byte from Chapter 8.2.1. Without it you cannot tell which key decrypts an old value, and rotation becomes impossible after the fact.

Rotate immediately, not on schedule, when: someone with access leaves, a secret appears anywhere it should not, a dependency is compromised, or you cannot rule out exposure.

7. Secret zero

Every design above has the same question underneath: how does the application authenticate to the secret store? If the answer is another secret, you have moved the problem rather than solved it. This is the secret zero problem.

The real answer is workload identity: the platform vouches for the workload, so no credential is distributed at all.

• Cloud instances have an identity issued by the platform (Chapter 8.4.9's managed identities; the metadata service that Chapter 8.5.2 warned about is the same mechanism, which is why IMDSv2 matters so much).
• Kubernetes projects a signed service account token that a cloud provider or Vault can verify, binding a pod to a role.
• SPIFFE/SPIRE issues short-lived identity documents to workloads across platforms, which then feed mTLS (Chapter 8.6.2).

The chain terminates in something unforgeable and platform-provided rather than in a stored value. That is the whole point, and it is why "use a managed identity" is the single most valuable secrets-management advice for anything running in a cloud.

8. Detection and response

Prevent at commit time with a pre-commit hook — gitleaks, detect-secrets, talisman. It is optional per developer, so it is a nudge, not a control.

Enforce server-side, where it cannot be bypassed: scanning on push, and platform push protection that blocks a commit containing a recognised credential pattern. Cloud providers also scan public repositories themselves and will quarantine a leaked key, which is how many teams first learn.

Detection is pattern-based plus entropy-based. Patterns catch known formats (AKIA…, sk_live_…, ghp_…) with few false positives. Entropy catches unknown formats and produces noise — hashes, minified code and test fixtures all look random. Tune the entropy rules or people will ignore the tool, which is the usual failure.

The incident order matters, and it is the opposite of instinct:

1. Revoke. Immediately. Before anything else, before investigation, before cleaning history.
2. Rotate. Issue the replacement and deploy it.
3. Investigate. Check logs for use of the old credential — this is where the KMS and cloud audit trails earn their place.
4. Clean. Rewrite history if you must, understanding it does not un-leak anything.
5. Fix the cause. Why was it there, and what makes the next one impossible?

Do not skip step three because the key is revoked. The question "was it used before we noticed" is the one an incident report has to answer.

What the interviewer will push on

"A key is committed to a public repository. What do you do?" Revoke first — it is already cloned, forked and scanned within minutes, so rewriting history helps nothing while the key stays valid. Then rotate, then check logs for use, then clean, then fix the cause. Ordering revocation first is the whole answer, and most people say "remove the commit".

"Are environment variables acceptable for secrets?" Better than source control, worse than a secrets manager: readable at /proc, inherited by child processes, captured in crash dumps, and unchangeable without a restart. Use them for the one credential that reaches the store, and prefer mounted files or a runtime API otherwise.

"What are dynamic secrets and why do they matter?" The store generates a fresh database user per application with a short lease. No long-lived shared credential exists, every instance is individually attributable, revocation is immediate, and rotation stops being an event because nothing lives long enough to need it.

"Explain envelope encryption." Data key encrypts the data locally; the master key in the KMS wraps only the data key and never leaves the service. Bulk speed with a small audited surface, and master key rotation means re-wrapping small keys rather than re-encrypting everything. Then add that ciphertext must carry a key version or rotation is impossible later.

"How do you rotate a credential without downtime?" Two active credentials, deploy the new one, confirm the old is unused by metrics rather than assumption, then disable and delete. That is why cloud services give you two keys — the second exists so rotation is not a cutover.

"How does the application authenticate to the secret store?" Not with another secret — that is secret zero. Workload identity: a managed identity, a Kubernetes projected token, or a SPIFFE document, verified by the platform. The chain must terminate in something unforgeable rather than in a stored value.

One thing to volunteer: point out that KMS decrypt calls are logged, so an unusual spike in unwrap operations is one of the few exfiltration signals that still works after an attacker holds valid credentials. Most detection assumes the attacker is unauthenticated; this one does not.

Recall

• Six stages: generate, distribute, store, use, rotate, revoke. The last two are neglected and are the ones that matter in an incident — a secret you cannot rotate fast is one you cannot respond to.
• Never in: source control (history is forever, in every clone and fork), container layers, CI logs, chat or tickets, client-side code, or plain Kubernetes Secrets (base64 is not encryption).
• Environment variables: readable at /proc, inherited by child processes, captured in crash dumps, and need a restart to change. Use them for the bootstrap credential only.
• Dynamic secrets — a per-application database user with a short lease — remove the long-lived shared credential entirely, make every instance attributable, and turn rotation into a non-event.
• Envelope encryption: a data key encrypts the data locally, the master key wraps only the data key and never leaves the KMS. Ciphertext must carry a key version or rotation is impossible afterwards. Key policies are the access control.
• Rotate with two active credentials — expand and contract — confirming the old one is unused by metrics, not assumption. That is why services give you two keys.
• Secret zero is answered by workload identity, not another secret: managed identities, Kubernetes projected tokens, SPIFFE. The chain terminates in something the platform vouches for.
• Incident order: revoke → rotate → investigate → clean → fix the cause. Cleaning history first is the common and useless first move. A spike in KMS unwrap calls is an exfiltration signal that works even against an authenticated attacker.

Self-test: Why is deleting the commit the wrong first action? · Name three specific weaknesses of environment variables · What does a dynamic secret remove that rotation only reduces? · Why does envelope encryption make master key rotation cheap? · What is secret zero and what actually solves it? · Which log tells you an authenticated attacker is reading data?

Next: 8.6.2 covers the boundaries around the workloads themselves — network segmentation, how services prove their identity to each other, and the supply chain that puts other people's code inside yours.