Backends: Where State Lives & How It Stays Locked

The Default Will Hurt You in Production

By default, Terraform stores state locally in ./terraform.tfstate. For learning and solo projects? Fine. In any environment where more than one person touches the infrastructure? It’s a disaster waiting to happen.

Three ways the default will ruin your day:

  • Your laptop dies → state gone → Terraform has no memory of what it built
  • Two engineers apply at the same time → state corruption → resources in an inconsistent state nobody understands
  • State contains secrets → it’s sitting in a plaintext file with zero access controls → anyone with the file has your database passwords

Backends solve all three. They define where state is stored and how concurrent access is controlled.

The backend Block

Configured inside the terraform {} block:

terraform {
  backend "s3" {
    bucket         = "company-tf-state"
    key            = "environments/prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

After adding or changing the backend, run terraform init to initialize (and optionally migrate existing state). You’ll forget this at least once. Everyone does.

The Local Backend (Default)
terraform {
  backend "local" {
    path = "relative/path/to/terraform.tfstate"
  }
}

Or just don’t declare a backend — local is the implicit default.

Fine for: Learning, personal projects, single-person throwaway configs. Not for: Teams, production, anything you’d lose sleep over if it broke.

Remote Backends — The Production Standard

S3 Backend (AWS)

This is what most AWS shops use. S3 for the state file, DynamoDB for the lock.

terraform {
  backend "s3" {
    bucket         = "company-tf-state"
    key            = "prod/vpc/terraform.tfstate"
    region         = "us-east-1"

    # State locking via DynamoDB
    dynamodb_table = "terraform-state-lock"

    # Encryption at rest
    encrypt = true

    # Optional: specific IAM role to assume
    role_arn = "arn:aws:iam::123456789:role/TerraformStateRole"
  }
}

You’ll need to create the bucket and DynamoDB table before you can use them. Here’s the bootstrap config most teams run once manually:

# Create the S3 bucket (do this manually or in a bootstrap config)
resource "aws_s3_bucket" "tf_state" {
  bucket = "company-tf-state"

  lifecycle { prevent_destroy = true }
}

resource "aws_s3_bucket_versioning" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  rule {
    apply_server_side_encryption_by_default { sse_algorithm = "AES256" }
  }
}

# DynamoDB table for state locking
resource "aws_dynamodb_table" "tf_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

GCS Backend (GCP)

terraform {
  backend "gcs" {
    bucket = "company-tf-state"
    prefix = "prod/vpc"
  }
}

GCS has built-in object versioning and native locking. No separate DynamoDB-equivalent needed. One of the things GCP got right over AWS for this use case.

Azure Backend

terraform {
  backend "azurerm" {
    resource_group_name  = "tf-state-rg"
    storage_account_name = "companytfstate"
    container_name       = "tfstate"
    key                  = "prod.terraform.tfstate"
  }
}
State Locking — Preventing Concurrent Chaos

When a backend supports locking, Terraform grabs a lock at the start of every state-writing operation (plan with -out, apply, destroy). Nobody else can run a concurrent apply while that lock is held.

You’ll see this when someone else has the lock:

│ Error: Error acquiring the state lock
│
│ Error message: ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│   ID:        f9a2a0e0-...
│   Path:      prod/vpc/terraform.tfstate
│   Operation: OperationTypeApply
│   Who:       [email protected]
│   Version:   1.5.7
│   Created:   2024-03-04 00:00:00 +0000 UTC
│   Info:

Alice is running an apply. Bob tries to run one at the same time. Bob gets this error. He waits, or messages Alice.

If Terraform crashes mid-apply, the lock gets orphaned. Force-unlock it:

terraform force-unlock <LOCK_ID>

Be absolutely certain no other apply is running before you do this. Force-unlocking while someone else is actively applying will corrupt your state. I’ve seen this happen exactly once. The cleanup took two days.

Not all backends support locking out of the box. The S3 backend only locks when you configure a DynamoDB table. GCS and Azure Storage support locking natively. The local backend has basic file-system locking which provides minimal protection. HCP Terraform always supports locking.

Backend Configuration Security

Never put credentials in the backend block. These run before providers, so you can’t use provider-level auth mechanisms:

# WRONG — credentials in config, will end up in Git
terraform {
  backend "s3" {
    access_key = "AKIAEXAMPLE"
    secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    ...
  }
}

# RIGHT — use environment variables or IAM roles
# AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in environment
# or IAM instance profile / OIDC if running in CI
terraform {
  backend "s3" {
    bucket = "company-tf-state"
    key    = "prod/terraform.tfstate"
    region = "us-east-1"
  }
}
Partial Configuration

For backends with sensitive config (account IDs, bucket names that reveal org structure), leave the config out of .tf files and pass it at init time:

# terraform config — minimal
terraform {
  backend "s3" {}
}

# Fill in at init time
terraform init \
  -backend-config="bucket=company-tf-state" \
  -backend-config="key=prod/terraform.tfstate" \
  -backend-config="region=us-east-1" \
  -backend-config="dynamodb_table=terraform-locks"

Or use a config file:

# backend.hcl (note: .hcl, not .tf — won't be auto-loaded)
bucket         = "company-tf-state"
key            = "prod/terraform.tfstate"
region         = "us-east-1"
dynamodb_table = "terraform-locks"

terraform init -backend-config="backend.hcl"

This is how most CI/CD pipelines handle it. The backend config stays out of the repo, injected at runtime by the pipeline.

State File Versioning

Turn on versioning for your S3 state bucket. This is not optional. When (not if) a state file gets corrupted, versioning is your recovery mechanism:

# List versions of the state file
aws s3api list-object-versions \
  --bucket company-tf-state \
  --prefix prod/vpc/terraform.tfstate

# Restore a previous version
aws s3api copy-object \
  --bucket company-tf-state \
  --copy-source "company-tf-state/prod/vpc/terraform.tfstate?versionId=<version-id>" \
  --key prod/vpc/terraform.tfstate

I’ve used this to recover from state corruption three times in my career. Every time, the recovery took minutes because versioning was enabled. Without it? You’re manually importing every resource back into state. For a 200-resource workspace, that’s a full day’s work.

Key Takeaways

The S3 backend does NOT lock state unless you configure a DynamoDB table. Without dynamodb_table, two concurrent applies can both write to S3 at the same time, producing corrupted state. I’ve seen teams run for months without the lock table, wondering why their state “randomly” breaks. It wasn’t random.

Changing the backend requires terraform init again. Change the backend config without re-initializing and Terraform keeps using the old backend from last init. It won’t even warn you.

Here’s the chicken-and-egg problem everyone hits: you can’t use Terraform to create the S3 bucket if that bucket IS the state backend. Bootstrap the state bucket and DynamoDB lock table manually (or with a separate Terraform config that uses local state) before setting up your main configs. It’s ugly but it’s a one-time thing.

Related Posts