Cruxly › Cloud and DevOps › Cloud & DevOps: Infrastructure as Code

Idempotency, Plan Review Discipline, and the Drift a Console Change Leaves Behind

why terraform apply twice must be safe, and the incident-week landmine

Prompt: an engineer widened a security group rule through the cloud console during an active incident to restore traffic quickly, and the incident resolved before anyone updated the Terraform config to match. Design the review and drift-detection discipline that catches this before a completely unrelated future apply silently reverts the emergency fix.

30 minutes first. The naive assumption is that Terraform "just handles" whatever's actually running. It doesn't — it only knows what its own state file remembers, and a console change updates reality without updating that memory at all.

Idempotency is a guarantee about repeated applies, not a description of the tool

"Idempotent" means running terraform apply twice in a row, with no config changes between them, produces no additional changes the second time — the first apply converges reality to match desired state; the second finds nothing left to do. This has to be true for Terraform to be trustworthy at all: a tool where re-running the same command sometimes produces different results is one you can't safely retry after a transient failure (a timeout talking to the cloud API, a CI job that got killed mid-apply) without first manually verifying nothing double-applied.

In a CI pipeline, the plan that was reviewed has to be the plan that runs. Interactively, bare terraform apply already shows its plan and waits for yes — only -auto-approve skips that. The problem is a pipeline where the plan is reviewed in a pull request and the apply runs later, in a different job:

• terraform plan -out=tfplan then terraform apply tfplan applies exactly the reviewed changes. If state has changed since the plan was made, Terraform refuses the saved plan as stale instead of quietly re-planning.
• A pipeline that re-plans at apply time (apply -auto-approve) executes whatever the world looks like then, which may not be what anyone reviewed.
• Protect the plan file like state. It contains the same sensitive values in plaintext.

Detecting the drift a console change leaves behind

terraform plan -refresh-only is the specific command for answering "has anything changed outside Terraform" without risking an accidental apply — it refreshes state against the live provider and shows the diff between last-known and actual, without proposing any changes to reality. This is the tool for exactly the incident-week scenario: after things have calmed down, running a refresh-only plan against production surfaces the widened security group rule as drift, as a deliberate check, rather than discovering it by accident when an unrelated apply reverts it. Note what fixes drift: apply -refresh-only only updates state to match reality. The config still says the old rule, so the next normal apply will still revert it. To keep the emergency change, codify it in the config.

Whether the console change gets reverted depends on how the rule is modelled. With inline ingress {} blocks on the security group, Terraform owns the whole rule set and will revert an extra rule. With separate rule resources (aws_vpc_security_group_ingress_rule), Terraform only manages the rules it created — a console-added rule isn't tracked at all and persists forever, invisibly. Both are bad; they're bad differently.

Scheduled, unattended drift checks (a nightly or weekly CI job running plan against production and posting the result, even when no config change was intended) are the practical way this actually gets caught in time, rather than relying on someone remembering to check. A team that only runs plan when they intend to change something has no mechanism at all for catching a change that happened outside Terraform in the meantime — the entire value of a scheduled check is that it runs whether or not anyone expects drift to be there.

The durable prevention, not just detection, is closing off the path that created the drift in the first place: restrict write access to Terraform-managed production resources to the Terraform/CI role and an audited break-glass role. IAM can't tell a console click from an API call, so the rule has to be about who, not how: an SCP that denies write actions on managed (for example, tagged) resources unless aws:PrincipalArn is the CI role or the break-glass role. Resource Control Policies and permission boundaries can tighten it further. Then the emergency fix during an incident goes through the same pipeline that manages everything else.

Refactoring without destroying, and policy as code on the plan

Renaming a resource in config looks like "destroy the old one, create a new one" to Terraform — for a database, that's data loss. Tell Terraform what you meant instead:

• moved blocks record that aws_db_instance.main is now module.db.aws_db_instance.this, so the next plan shows a move, not a replace.
• removed blocks stop managing a resource without destroying it (for example, handing it to another stack).
• terraform apply -replace=ADDRESS forces a specific resource to be recreated — the reviewed, plan-visible replacement for the old taint command.
• Lifecycle guards: prevent_destroy makes any plan that would destroy a critical resource fail, and create_before_destroy builds the replacement first to avoid downtime.

Policy as code checks the plan before a human ever looks at it. Export the plan as JSON (terraform show -json tfplan) and evaluate it with OPA /Conftest or Sentinel: no public S3 buckets, no security group open to 0.0.0.0/0 on port 22, every resource tagged with an owner, no destroy of anything with prevent_destroy intent. Static scanners such as Checkov or Trivy catch misconfigurations in the .tf files earlier still. In a large plan, an automated policy failure is much harder to miss than one line in a 400-line diff.

How to say it in the interview

"Terraform doesn't watch your infrastructure continuously - it only compares its state file to reality at the moment you run plan or apply. A manual console change made in the meantime is invisible to Terraform right up until some unrelated future apply recomputes desired state from the unchanged config and quietly reverts a fix nobody remembers making, weeks after the incident that required it."

Checklist

• Verified applies are idempotent — a second apply with no config change produces no further changes
• In CI, applied the saved plan that was reviewed rather than re-planning with -auto-approve
• Ran plan -refresh-only to check for drift deliberately, rather than discovering it via an unrelated apply's side effect
• Ran drift checks on a schedule (CI job, not 'whenever someone remembers')
• Restricted direct console/CLI write access to Terraform-managed production resources via IAM, with a narrow audited break-glass exception
• Codified the emergency change in config instead of only running apply -refresh-only
• Used moved/removed blocks and prevent_destroy to refactor without destroying, and ran policy as code on the plan JSON

Related lessons

• Terraform State: The File That Can Take Down Your Infrastructure