Terraform Infrastructure Drift: Catching Manual Changes Before the Next Apply

Terraform runs on an assumption nobody says out loud often enough: that state is the single source of truth for what exists in the cloud, and that nothing changes those resources except through an apply. That assumption holds as long as the whole team sticks to the flow declared in the repository. It breaks the first time someone opens the AWS console, resizes an instance, adds a rule to a security group, or bumps an autoscaling group's limit to kill an alert at two in the morning. The real resource no longer matches what state believes exists, and Terraform has no way of knowing that until someone runs a plan.

We call that mismatch drift: the gap between what the code declares, what state recorded the last time an apply ran, and what actually exists in the cloud right now. The problem isn't that the manual change was necessarily wrong -sometimes it's exactly the right call to resolve an incident fast-. The problem is that the change stays invisible to the rest of the team until someone touches that same resource from Terraform again, and at that point Terraform silently tries to undo something a teammate changed on purpose, with nobody noticing that's what's happening.

Why drift doesn't hurt until it does

The scenario we run into most often goes like this: an on-call engineer responds to a capacity incident by manually bumping an autoscaling group's desired_capacity from the AWS console, because that's the fastest way to mitigate in the moment. The incident gets resolved, the alert clears, and nobody circles back to it. Weeks later, someone else on the team makes an unrelated change to the same Terraform module -adjusting a scaling policy, say- and runs terraform apply. Terraform compares the code against state, not against reality, so it has no reason to know the real desired_capacity differs from what state last recorded. The apply silently reverts capacity to whatever value was in the code before the incident, and the system ends up back under the same pressure that caused the original problem, except this time nobody decided that on purpose.

This kind of incident is hard to diagnose because it doesn't show up as a Terraform error. The apply runs clean, no warnings, because from Terraform's point of view it isn't reverting anything: it's just applying the code on top of a state that, until that moment, nobody had updated to reflect the manual change.

terraform plan in CI as the first line of defense

The first practice we put in place was running terraform plan in a CI pipeline on every pull request that touches the infrastructure directory, before merging. This doesn't detect drift by itself -it compares code against state, not against the cloud-, but it establishes the discipline that no infrastructure change reaches production without someone having seen the full plan, including resources the PR's author didn't intentionally touch.

# .github/workflows/terraform-plan.yml
name: terraform-plan
on:
  pull_request:
    paths:
      - 'infra/**'

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
        working-directory: infra
      - run: terraform plan -detailed-exitcode -out=tfplan
        working-directory: infra

The -detailed-exitcode flag is what makes this step useful: instead of the usual binary exit code (0 success, 1 error), Terraform returns 0 when there are no changes, 1 when there was a real error, and 2 when there are pending changes to apply. That lets the pipeline distinguish "this PR doesn't change infrastructure" from "this PR does change something," and post the full plan as a PR comment so the reviewer reads it before approving, instead of trusting that the author described the change accurately in the PR description.

Scheduled drift detection, without waiting for the next PR

terraform plan in CI catches drift introduced after a manual change, but only once someone touches that module from Terraform again. If nobody touches that resource for weeks, the drift stays invisible the whole time. To cover that gap we added a second mechanism: a scheduled job that runs terraform plan against every infrastructure module on a regular cadence, with no code change involved, purely to compare state against reality.

# .github/workflows/terraform-drift-check.yml
name: terraform-drift-check
on:
  schedule:
    - cron: '0 12 * * 1-5'

jobs:
  drift:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        module: [networking, compute, data]
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
        working-directory: infra/${{ matrix.module }}
      - name: Check drift
        id: plan
        working-directory: infra/${{ matrix.module }}
        run: |
          terraform plan -detailed-exitcode -no-color -input=false
        continue-on-error: true
      - name: Notify on drift
        if: steps.plan.outputs.exitcode == '2'
        run: |
          echo "Drift detected in ${{ matrix.module }}"
          # notify the team channel with the plan output

This job runs Monday through Friday at noon, against each module separately, and only notifies when the exit code is 2 -there are real differences between state and the cloud-. The notification includes the full plan output, so whoever reviews it sees exactly which resource changed and in which attribute, instead of having to run the plan by hand to find out.

What didn't work: auto-fixing drift

The first version of this job didn't just detect drift, it tried to fix it: whenever it found differences, it ran terraform apply -auto-approve to push the resource back to whatever the code declared, with no human in the loop. The idea was to close the full cycle -detect and fix- without depending on someone seeing the notification and acting on it. We reverted that part within a few days, because the same autoscaling group scenario played out again, this time automated: a legitimate manual change, made to mitigate an incident in progress, got reverted by the drift job while the incident was still active, and the person who made the manual change had no idea anything had undone it. Auto-fixing assumes every instance of drift is a mistake that needs reverting, and that assumption is false at the worst possible moment: during an incident, which is exactly when someone is most likely to have changed something by hand on purpose. The job now only detects and notifies; the decision of whether drift should be fixed with an apply or folded into the code as a deliberate change is made by a person, not the pipeline.

The real problem is console access

Both mechanisms -plan in CI and scheduled detection- treat the symptom. The root cause is that too many people hold write permissions in the AWS console over resources Terraform manages. As long as that access exists, drift will keep happening, because nobody in the middle of an incident is going to wait for a Terraform PR to clear review before mitigating. The fix we're migrating toward, still in progress, is cutting console permissions in production accounts down to read-only for most of the team, and reserving write access for a breakglass role with a temporary, audited session -something anyone can activate during an incident, but that leaves an explicit record that it was used. That doesn't eliminate drift -it will keep happening during those incidents-, but it makes it visible the moment it happens, instead of leaving it as a surprise someone else finds weeks later in a plan that shouldn't have shown any changes.