Cruxly › Cloud and DevOps › Cloud & DevOps: Infrastructure as Code
Terraform State: The File That Can Take Down Your Infrastructure
locking, corruption, and why state is not just a cache
Open the interactive lessonPrompt: a team runs Terraform against shared infrastructure from several CI pipelines and, occasionally, an engineer's laptop for a quick fix. Design the state management setup such that two concurrent applies can never corrupt the state file, and such that losing the state file is a recoverable event, not a catastrophe.
30 minutes first. The naive setup keeps state as a local file, or in a shared bucket with no locking at all — both look fine until two applies actually happen close together, which is exactly when it matters.
What state actually is, and why local state is a landmine
Terraform computes every plan from three inputs: your config (desired), the state file (last-known), and a live refresh against the provider (actual). State is not optional bookkeeping — it's the only record Terraform has of which real-world resource corresponds to which block in your configuration. Delete it, and Terraform doesn't know your existing VPC is the VPC your config describes; the next apply may try to create a duplicate, or fail confusingly when it collides with a resource it no longer recognizes as its own.
Local state (a .tfstate file only on one engineer's laptop) is a single point of failure and a guaranteed collision the moment a second person runs Terraform against the same infrastructure — there is no shared source of truth, so two people's local state files can each believe they're authoritative while describing infrastructure that's since diverged under the other person's changes. Remote state (S3, GCS, HCP Terraform — formerly Terraform Cloud — or an equivalent) is not a convenience feature, it's the minimum requirement for more than one person or pipeline to safely touch the same infrastructure at all.
Locking prevents corruption, not disagreement
A state lock's job is narrow and specific: prevent a second apply from starting against the same state while a first one is still in flight — it does not resolve disagreement about what infrastructure should look like, it only prevents two concurrent writers from corrupting the state file's own internal consistency by both reading the same starting point and writing back two different, conflicting results. Without a lock, this race is entirely possible in any team where CI and a human both have apply access to the same state — a pipeline-triggered apply and someone troubleshooting locally, close enough in time, are enough.
The locking mechanism itself is backend-specific and has gotten simpler over time — worth knowing the current state of the tooling rather than repeating an older pattern by habit. The long-standing AWS pattern paired an S3 bucket (for the state file itself) with a separate DynamoDB table purely for locking, because S3 alone had no native locking primitive. Terraform 1.10 added native S3 locking via S3 conditional writes, opt-in with use_lockfile = true in the backend block; Terraform 1.11 made it generally available and deprecated the dynamodb_table setting. It isn't automatic — a team that upgrades Terraform but doesn't set use_lockfile still has whatever locking it had before. OpenTofu 1.10 supports the same thing. GCS and HCP Terraform have handled locking natively for longer.
A stuck lock (a crashed CI job that acquired the lock and never released it) is an operational scenario that needs a deliberate, audited unlock procedure — not "just force-unlock it" as a reflex. terraform force-unlock exists for exactly this recovery case, but running it without first confirming no other apply is genuinely still in flight reintroduces the exact race the lock existed to prevent, just manually.
Backing up and recovering from a lost or corrupted state file
State should be versioned, not just stored — S3 with versioning enabled, Terraform Cloud's own history, or an equivalent, so a corrupted or accidentally-overwritten state file has a prior, known-good version to roll back to rather than starting from a completely blank memory of every managed resource.
Import rebuilds a lost mapping between config and real resources, and it's much less painful than it used to be. Since Terraform 1.5, import {} blocks in config let you declare imports, review them in a normal plan, and have Terraform generate matching configuration (terraform plan -generate-config-out=generated.tf); since 1.7 they support for_each for importing many resources at once. It's still the fallback for when versioned backups have already failed, not a substitute for having them. A team's actual disaster recovery plan for state loss should assume import is the last resort, and size the effort of "how many resources would we need to re-import" as a real number, in a tabletop exercise, before an actual incident forces that math to happen for real under time pressure.
State almost always contains sensitive values (database passwords, generated secrets, connection strings) in plaintext, because Terraform needs to know a resource's real attributes to diff against them on the next plan. State storage therefore needs the same access controls as the secrets it contains — a permissive bucket policy on a "just infrastructure metadata" state bucket is, in practice, a permissive policy on every secret Terraform has ever provisioned.
Keeping secrets out of state, keeping state small, and the tooling landscape
Better than locking down plaintext secrets in state: don't put them there. Recent Terraform releases added two features for exactly this:
- Ephemeral values and resources (Terraform 1.10) — values such as a freshly fetched secret that are used during a run but never written to state or plan files.
- Write-only arguments (Terraform 1.11) — a resource argument (a database password, say) that Terraform sends to the provider but never stores. You bump a version number to tell it to send a new value.
- OpenTofu 1.7+ adds native state encryption, so even what does land in state is encrypted with a key you control.
Split state to limit the blast radius. One giant state file for a whole company means every plan is slow, every apply locks everything, and one mistake can touch anything. Split by environment and by component — networking, data stores, each service — so a change to one service's stack can't plan a destroy of the VPC. Workspaces are for near-identical copies of one configuration (dev/staging/prod with the same shape), not for unrelated components. Terragrunt or HCP Terraform Stacks help wire many small states together.
Pin what you depend on. Commit .terraform.lock.hcl (provider versions and checksums) and set version constraints on modules, so a provider release can't change behaviour under an unchanged config.
The licensing landscape, which interviewers do ask about. In August 2023 HashiCorp moved Terraform to the Business Source License. The community fork, OpenTofu, is now under the Linux Foundation, and IBM completed its acquisition of HashiCorp in February 2025. The tools remain largely compatible, but OpenTofu has features of its own (state encryption, for example) — say which one a team uses and why.
How to say it in the interview
"State is not a cache you can regenerate by re-running terraform - it's Terraform's own memory of what it created and where. Losing it, or having two applies corrupt it by writing conflicting changes at the same time, doesn't just lose a file, it makes every future apply blind to what already exists in your real infrastructure."
Checklist
- Used remote state (S3/GCS/HCP Terraform), not a local file, for any infrastructure touched by more than one person or pipeline
- Enabled state locking so two applies can never write conflicting changes to the same state file concurrently
- Used the current locking mechanism for the backend in use rather than an outdated pattern copied from old documentation
- Treated force-unlock as an audited recovery procedure, not a reflex, when encountering a stuck lock
- Versioned the state backend so a corrupted or overwritten state file has a known-good prior version to recover
- Kept secrets out of state with ephemeral values and write-only arguments, and split state by environment and component
- Applied the same access controls to the state backend as to the secrets it contains