CI/CD Pipelines: From Pull Request to Production

Nobody Runs terraform apply From Their Laptop in Production

In a learning environment, you run terraform apply from your terminal. That’s fine. In production, you don’t. Period. Infrastructure changes go through a pipeline — same as application code. A developer writes a change, opens a PR, the pipeline validates it, a reviewer approves it, automation applies it.

If someone on your team is running terraform apply against production from their laptop, your process has a hole in it. Fix it before you fix anything else.

The PR-Driven Infrastructure Workflow

Every infrastructure change follows this lifecycle:

Developer creates branch
    ↓
Makes Terraform changes (add subnet, modify security group, etc.)
    ↓
Opens Pull Request
    ↓
CI Pipeline triggers automatically:
    ├── terraform fmt -check
    ├── terraform validate
    ├── terraform plan (output posted to PR)
    ├── Policy checks (Sentinel / OPA / tfsec / Checkov)
    └── Cost estimation (optional: Infracost)
    ↓
Peer review + CODEOWNERS approval
    ↓
Merge to main branch
    ↓
CD Pipeline triggers:
    ├── terraform plan (fresh plan on main)
    ├── Approval gate (manual or automatic depending on environment)
    └── terraform apply
    ↓
State updated. Infrastructure deployed.

That’s the whole flow. Every CI/CD setup in the industry is some variation of this. Simpler ones skip policy checks. More complex ones add cost estimation and Slack notifications. But the skeleton is always the same.

Pipeline Stages in Detail

Stage 1: Format and Validate

# Does the code follow HCL formatting standards?
terraform fmt -check -recursive
# Exit code 0 = formatted correctly
# Non-zero = someone didn't run fmt

# Is the configuration syntactically valid?
terraform init -backend=false    # Init without touching state
terraform validate

These are fast — they don’t touch cloud APIs, don’t need credentials, and run in seconds. They catch the easy stuff: syntax errors, missing required fields, formatting inconsistencies.

That -backend=false is important. Without it, your format-check stage tries to authenticate to your S3 backend. A formatting check shouldn’t need cloud credentials.

One thing to understand about terraform validate: it checks syntax and internal consistency. It does NOT check whether your AMI exists, your region is valid, or your instance type is available in that region. Those are runtime errors you’ll only catch during plan or apply. Validate is necessary but not sufficient.

Stage 2: Plan

terraform init
terraform plan -out=tfplan

This is where Terraform connects to your cloud, reads current state, compares it against your config, and produces a diff showing exactly what will change.

The -out=tfplan flag saves the plan to a binary file. This is non-negotiable in CI/CD. Here’s why: without a saved plan, the apply stage would re-run plan internally. If something changes in the cloud between plan and apply (someone modifies a resource, an auto-scaling event occurs), you’d apply something nobody reviewed. The saved plan is the contract — what was reviewed is what gets applied.

Posting the plan to the PR:

Most CI systems support posting plan output as a PR comment. Reviewers see creates, updates, and destroys before they approve:

# GitHub Actions example (simplified)
- name: Terraform Plan
  id: plan
  run: terraform plan -out=tfplan -no-color 2>&1 | tee plan_output.txt

- name: Post Plan to PR
  uses: actions/github-script@v6
  with:
    script: |
      const plan = require('fs').readFileSync('plan_output.txt', 'utf8');
      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body: `## Terraform Plan\n\`\`\`\n${plan}\n\`\`\``
      });

Use -no-color when capturing plan output. Terraform’s ANSI color codes turn into garbage characters in markdown. Some teams use terraform show -json tfplan for machine-readable output that integrates with dashboarding tools.

Stage 3: Policy Checks

After the plan is generated, policy-as-code tools inspect it:

Sentinel (HCP Terraform native):

# No public S3 buckets
import "tfplan/v2" as tfplan

main = rule {
    all tfplan.resource_changes as _, rc {
        rc.type is not "aws_s3_bucket" or
        rc.change.after.acl is not "public-read"
    }
}

OPA/Rego (open source):

# All instances must have tags
package terraform
deny[msg] {
    r := input.resource_changes[_]
    r.type == "aws_instance"
    not r.change.after.tags["CostCenter"]
    msg := sprintf("Instance %s missing CostCenter tag", [r.address])
}

tfsec (static security scanner):

tfsec .    # Scans .tf files for security misconfigurations

Checkov (compliance scanner):

checkov -d . --framework terraform

Infracost (cost estimation):

infracost breakdown --path .    # Shows estimated monthly cost changes

Policy check fails → PR is blocked → developer fixes → pushes again → pipeline re-runs. The whole point is catching problems before they reach production.

Here’s a nuance worth understanding: Sentinel and OPA run against the plan output (what Terraform will do). tfsec and Checkov scan the static .tf files (what the code says). They catch different things. I run both — static scanning first because it’s fast, plan-based policies after because they’re deeper.

Stage 4: Approval Gate

Two models, and most production setups use both:

Peer review approval (Git-based): The plan is posted to the PR. A team member reviews it. They approve the PR. Merge triggers the apply.

Explicit apply approval (pipeline-based): Even after PR merge, the apply doesn’t run automatically. An approver clicks “Confirm & Apply” in HCP Terraform — or approves a pipeline gate in GitHub Actions.

# GitHub Actions: Manual approval before apply
jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: terraform init && terraform plan -out=tfplan
      - uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: tfplan

  apply:
    needs: plan
    runs-on: ubuntu-latest
    environment: production   # <-- Requires manual approval in GitHub
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: tfplan
      - run: terraform init && terraform apply tfplan

The environment: production with required reviewers means someone must click “Approve” before the apply job runs.

This is an important distinction: a PR approval says “this code looks correct.” An environment approval says “I’m authorizing this to run against production right now.” They’re different decisions, made by potentially different people. Treat them that way.

Stage 5: Apply

terraform init
terraform apply tfplan    # Apply the exact saved plan — no re-planning

The saved plan file is the contract. What was reviewed is what gets applied. No drift, no surprises, no re-planning with different inputs.

After apply: state is updated in the remote backend, run history is logged, outputs are available for downstream consumers.

Branch Strategies for Infrastructure

Strategy 1: Feature Branch → Main (Most Common)

main ←── feature/add-private-subnet
     ←── feature/update-security-groups
     ←── fix/nat-gateway-routing

All changes branch from main. PR targets main. Merge triggers apply. Different environments use different workspaces or .tfvars files. Simple, effective, works for most teams.

Strategy 2: Environment Branches (Promotion Model)

dev ←── feature/add-subnet
staging ← (merge from dev after testing)
prod    ← (merge from staging after approval)

Feature branches target dev. Promoted to staging via merge/PR. Promoted to prod with additional approvals. Each branch has its own pipeline targeting the corresponding environment.

More ceremony, but compliance-friendly. Some auditors want to see explicit promotion records.

Strategy 3: Single Branch + Workspace Selection

main ←── feature/add-subnet

Pipeline uses workspace selection or .tfvars to target environments:

# In CI, determined by pipeline variable or tag
terraform workspace select ${ENVIRONMENT}
terraform plan -var-file=envs/${ENVIRONMENT}.tfvars -out=tfplan

Strategy 1 for most teams. Strategy 2 when compliance requires promotion records. Strategy 3 when using HCP Terraform workspaces. Don’t overthink this.

Run Triggers: Cascading Changes

When the network team applies a change, downstream teams may need to re-plan.

In HCP Terraform: Configure a run trigger: networking-prod → triggers compute-prod. When networking state changes, compute auto-plans. If the plan shows changes (because compute references networking outputs), it goes through normal approval.

In custom CI/CD:

# networking pipeline — after successful apply
- name: Trigger downstream pipelines
  run: |
    curl -X POST "https://api.github.com/repos/acme/terraform-infra/dispatches" \
      -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
      -d '{"event_type": "network-state-changed", "client_payload": {"env": "prod"}}'

# compute pipeline — triggered by network state change
on:
  repository_dispatch:
    types: [network-state-changed]

Run triggers create dependency chains. If the networking workspace fails to apply, downstream workspaces still have the old state. Usually fine — but be aware that cascading failures can leave infrastructure partially updated. Design for eventual consistency, not atomic cross-workspace transactions.

Credential Management in Pipelines

The pipeline needs cloud credentials to run plan and apply. Three approaches, from worst to best:

Bad: Static credentials in CI secrets

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Works. But those credentials never expire. If they leak — from a log, a fork, a misconfigured action — they’re valid until someone manually rotates them.

Better: OIDC / Workload Identity Federation

# GitHub Actions — AWS OIDC
- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789:role/GitHubActionsRole
    aws-region: us-east-1

No static secrets. GitHub proves its identity to AWS via OIDC. AWS issues a short-lived credential scoped to that run. Expires when the job finishes. This is what you should be using.

Best: HCP Terraform Dynamic Credentials

TFC_AWS_PROVIDER_AUTH = true
TFC_AWS_RUN_ROLE_ARN  = arn:aws:iam::123456789:role/HCPTerraformRole

HCP Terraform handles the OIDC exchange transparently. No credentials configured anywhere — not in CI, not in variable sets.

If you’re running in GitHub Actions, use OIDC. If you’re on HCP Terraform, use dynamic credentials. Never store long-lived cloud credentials in CI if you can avoid it. The number of production incidents caused by leaked static keys is staggering.

Design Scenario: A Network Engineer Adds a Subnet

Here’s how this plays out in practice:

Monday 10:00 AM — Branch and code:

git checkout -b feature/add-database-subnet

# infra-network/main.tf (addition)
resource "aws_subnet" "database" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.50.0/24"
  availability_zone = "us-east-1a"
  tags = {
    Name        = "database-private-subnet"
    Team        = "network"
    Environment = "prod"
    Tier        = "database"
  }
}

# infra-network/outputs.tf (addition)
output "database_subnet_id" {
  value = aws_subnet.database.id
}

10:15 AM — Push and open PR.

10:16 AM — CI pipeline runs automatically:

  • ✅ terraform fmt -check — passes
  • ✅ terraform validate — passes
  • ✅ terraform plan — 1 to add, 0 to change, 0 to destroy
  • ✅ tfsec — no security issues
  • ✅ Checkov — subnet is private (no public IP auto-assign) ✓
  • Plan output posted to PR as a comment

10:30 AM — Peer review:

  • Another network engineer reviews the PR
  • Checks CIDR doesn’t overlap with existing subnets
  • Checks tags follow naming standards
  • Approves

10:35 AM — Merge to main. CD pipeline triggers.

10:36 AM — Apply pipeline:

  • Fresh terraform plan on main — confirms 1 addition
  • Network team lead approves the apply
  • terraform apply tfplan executes
  • Subnet created. State updated.

10:37 AM — Run trigger fires:

  • compute-prod workspace auto-plans
  • Plan shows no changes (compute hasn’t referenced the new subnet yet)

Tuesday — Database team consumes the new subnet:

  • Database team adds data.terraform_remote_state.network.outputs.database_subnet_id to their RDS config
  • Their own PR, their own pipeline, their own approver
  • RDS instance deploys into Sarah’s subnet

Total production downtime: zero. No console logins. No laptop applies. Every change in Git, every approval named.

Key Takeaways

Never run terraform apply -auto-approve in a production pipeline without a saved plan. -auto-approve skips the confirmation prompt, but if you don’t use a saved plan file, Terraform re-plans at apply time. If the cloud state changed between plan and apply, you’re applying something nobody reviewed. I’ve seen this cause a production outage when an auto-scaling event happened between plan and apply stages.

If your pipeline runs plan and apply in separate jobs, remember that .terraform/ must be consistent between them. Either run terraform init in both jobs, or persist the .terraform/ directory as a build artifact.

Add terraform plan output as a required status check on your PRs. Don’t let anyone merge infrastructure changes without seeing the plan. The number of production incidents caused by “I thought it would only add a tag” is genuinely impressive.

Related Posts