Cruxly

Cruxly › Cloud and DevOps › Cloud & DevOps: Security

Secrets Management: Why Environment Variables Aren't Enough

auditability, rotation friction, and dynamic secrets

Open the interactive lesson

Prompt: a service currently reads its database password from an environment variable, injected at container start. Design a secrets management approach that supports rotating that credential without a deploy or restart, and that provides an audit trail of which process accessed which secret and when — neither of which the current approach can do.

30 minutes first. The naive assumption is that an environment variable is "basically a secret" as long as it's not committed to source control. Not committing it to git solves one problem and leaves several others completely unaddressed.

What an environment variable structurally cannot do

A secret injected via environment variable at container start has no mechanism to be updated while the process is running (a dedicated secrets manager fixes this only if the app re-reads the value — covered two sections down) — the process read the variable once, at startup, and has no way to be told "this value changed, please re-read it." Rotating the secret therefore requires restarting every process that consumes it, converting what should be a routine background operation into a coordinated deploy event across every consumer, which is precisely the kind of friction that makes rotation happen on a much longer interval than it should — teams routinely rotate less often specifically because the mechanism makes it costly, not because the risk of infrequent rotation is actually lower.

There is no audit trail of which process read which secret, or when. An environment variable, once set in a container's environment, is readable by that process with no logging of the read itself — during an incident investigation, "did the compromised process actually access this secret, and when" is frequently unanswerable from environment- variable-based storage alone, because the mechanism was never built to record that.

Access can't be scoped per-consumer at the granularity a real secrets manager provides. Every process with access to a given environment gets the same secret value, with no way to grant one consumer read access to secret A and a different consumer read access to secret B without provisioning entirely separate environments — a real secrets manager (Vault, a cloud KMS-integrated secrets service) scopes access at the identity level instead, per secret, with policy the platform actually enforces.

Dynamic secrets invert the rotation problem instead of just automating it

A static secret has one value, valid until someone manually replaces it everywhere it's used — even automated rotation of a static secret still has to coordinate updating every consumer at roughly the same moment, or accept a window where some consumers hold the old value and some hold the new one.

A dynamic secret is generated on demand, per lease, by a system like Vault's database secrets engine — when a service needs database credentials, it requests them from Vault, which creates a genuinely new, unique credential on the database itself at that moment. The service keeps using that credential, renewing the lease as needed up to its max TTL, then requests a new one — it's per lease, not per database query. The credential expires automatically without anyone needing to coordinate a manual replacement anywhere. This inverts the question from "how do we safely replace a long-lived secret everywhere it's used" to "how short can we make the lease" — a fundamentally more tractable problem, because shortening a lease is a configuration change, not a coordinated multi-consumer rotation event.

This also directly bounds the value of a leaked credential, because a leaked dynamic secret is only useful for the remainder of its lease — minutes to hours, depending on configuration — rather than being valid indefinitely until someone notices the leak and manually revokes it, which for a static secret can be an arbitrarily long window if nothing prompts anyone to look.

The trade-offs are real, though:

A lease of hours (not minutes), renewed by an agent, is a common compromise.

Rotation that actually works, and what Kubernetes Secrets really protect

A secrets manager doesn't make rotation restart-free by itself — the app has to re-read the secret. If the app reads the secret once at startup and keeps it in memory, rotating it in Vault changes nothing until a restart. Rotation without a restart needs one of these:

The app must also replace pooled database connections that are still using the old credential.

Rotate with two valid credentials, not one. If a password is changed in place, some consumers have the new value and some still have the old one, and the old ones fail until they re-read. The standard fix keeps two users or keys and alternates between them:

  1. Create or update credential B while A is still valid.
  2. Move every consumer to B.
  3. Only then revoke A.

AWS Secrets Manager's "alternating users" rotation does this, and it's the same pattern used for API keys and signing keys.

Environment variables leak in more places than people expect:

Kubernetes Secrets are base64-encoded, not encrypted. Anyone who can get a Secret can decode it, and list permission on Secrets in a namespace means reading all of them. To protect them:

The tooling landscape has changed too. HashiCorp moved Vault to the Business Source License in 2023, IBM completed its acquisition of HashiCorp in 2025, and OpenBao is the Linux Foundation's open-source fork. Cloud-native stores (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) cover many teams' needs without running Vault at all.

How to say it in the interview

"Not committing a secret to git is necessary, but it's not the same problem as being able to rotate it, audit who read it, or scope access to it per-consumer. An environment variable injected at container start has no mechanism for the running process to be told the secret changed - rotating it means restarting every consumer, which is exactly the friction that causes rotation to happen far less often than it should."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons