The End-to-End Production Scenario: How It All Fits Together

This Is How Real Companies Ship Infrastructure

You’ve learned the pieces: modules, state files, backends, remote state, workspaces, policies, CI/CD. This article puts them all together. We’re walking through a complete production scenario — a new microservice launch requiring changes from four different teams, each working independently, each going through their own approval chain, all converging into a running system.

No hand-waving. No “and then magic happens.” This is how it works at a company that takes infrastructure seriously.

The Setup: Acme Corp’s Infrastructure Platform

Teams:

Repository: Monorepo — acme/terraform-infra

terraform-infra/
├── infra-security/
├── infra-network/
├── infra-compute/
├── infra-storage/
├── modules/
├── .github/
│   ├── workflows/
│   │   ├── security.yml
│   │   ├── networking.yml
│   │   ├── compute.yml
│   │   └── storage.yml
│   └── CODEOWNERS
└── README.md

CI/CD: GitHub Actions. Each team has a pipeline triggered by changes in their directory. HCP Terraform manages state, run triggers, and approvals for production.

Authentication: OIDC dynamic credentials. No static AWS keys anywhere.

Policy enforcement: Sentinel in HCP Terraform. Hard-mandatory for security policies. Soft-mandatory for cost.

The Scenario: Launch the “Payments API” Microservice

Product has greenlit a new Payments API. It needs:

• IAM roles with least-privilege access to DynamoDB and S3
• A new private subnet for the service
• An S3 bucket for payment receipts (encrypted, no public access)
• An ECS Fargate service with an ALB

Four teams. Four workstreams. Let’s walk through it.

Phase 1: Security Team — IAM Roles & Policies

Who: Priya (Security Engineer) When: Tuesday 9:00 AM

Priya creates a branch and adds the IAM resources.

git checkout -b feature/payments-api-iam

# infra-security/payments.tf

# IAM role for the ECS task (what the container runs as)
resource "aws_iam_role" "payments_api_task" {
  name = "payments-api-task-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
    }]
  })

  tags = {
    Team        = "security"
    Service     = "payments-api"
    Environment = "prod"
    ManagedBy   = "terraform"
  }
}

# Policy: Read/Write to payments DynamoDB table
resource "aws_iam_role_policy" "payments_dynamodb" {
  name = "payments-dynamodb-access"
  role = aws_iam_role.payments_api_task.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:Query", "dynamodb:UpdateItem"]
      Resource = "arn:aws:dynamodb:us-east-1:*:table/payments-*"
    }]
  })
}

# Policy: Write receipts to S3
resource "aws_iam_role_policy" "payments_s3" {
  name = "payments-s3-receipts"
  role = aws_iam_role.payments_api_task.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:PutObject", "s3:GetObject"]
      Resource = "arn:aws:s3:::acme-payment-receipts-prod/*"
    }]
  })
}

# infra-security/outputs.tf (addition)
output "payments_api_task_role_arn" {
  value = aws_iam_role.payments_api_task.arn
}

9:15 AM — PR opened. Pipeline runs:

• ✅ fmt, validate
• ✅ terraform plan → 3 to add, 0 to change, 0 to destroy
• ✅ Sentinel: IAM roles must have tags ✓
• ✅ Sentinel: No * in IAM actions ✓ (least privilege check)
• ✅ Sentinel: No inline policies with iam:* ✓

9:30 AM — Reviewed by Marcus (Senior Security Engineer). He checks:

• Role follows least-privilege: DynamoDB actions scoped to payments-* tables only ✓
• S3 actions limited to PutObject/GetObject on the specific bucket ✓
• No overly broad Resource: * ✓
• Approves.

9:35 AM — Merged to main. CD pipeline runs:

• HCP Terraform security-prod workspace queues a plan
• Plan confirms: 3 resources to create
• Auto-apply is disabled for production. Marcus confirms the apply in HCP UI.
• ✅ Applied. IAM role ARN is now in state and available as an output.

Time to production: 35 minutes. One branch, one review, one apply. No tickets filed. No access requests. Just code.

Phase 2: Network Team — New Private Subnet

Who: Sarah (Network Engineer) When: Tuesday 10:00 AM

git checkout -b feature/payments-api-subnet

# infra-network/payments.tf

resource "aws_subnet" "payments_private" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.60.0/24"
  availability_zone       = "us-east-1a"
  map_public_ip_on_launch = false

  tags = {
    Name        = "payments-private-1a"
    Team        = "network"
    Service     = "payments-api"
    Environment = "prod"
    Tier        = "private"
  }
}

resource "aws_subnet" "payments_private_1b" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.61.0/24"
  availability_zone       = "us-east-1b"
  map_public_ip_on_launch = false

  tags = {
    Name        = "payments-private-1b"
    Team        = "network"
    Service     = "payments-api"
    Environment = "prod"
    Tier        = "private"
  }
}

# Route table association — private subnets use NAT gateway
resource "aws_route_table_association" "payments_1a" {
  subnet_id      = aws_subnet.payments_private.id
  route_table_id = aws_route_table.private.id
}

resource "aws_route_table_association" "payments_1b" {
  subnet_id      = aws_subnet.payments_private_1b.id
  route_table_id = aws_route_table.private.id
}

# infra-network/outputs.tf (addition)
output "payments_subnet_ids" {
  value = [
    aws_subnet.payments_private.id,
    aws_subnet.payments_private_1b.id
  ]
}

10:15 AM — PR opened. Pipeline runs:

• ✅ fmt, validate
• ✅ terraform plan → 4 to add
• ✅ Sentinel: No public subnets without explicit approval ✓
• ✅ Sentinel: CIDR must be within 10.0.0.0/16 range ✓
• ✅ Checkov: map_public_ip_on_launch = false ✓

10:30 AM — Reviewed by Mike (Senior Network Engineer):

• CIDRs 10.0.60.0/24 and 10.0.61.0/24 don’t overlap with existing allocations ✓
• Two AZs for high availability ✓
• Route table association uses the private (NAT) route table, not the public one ✓
• Approves.

10:35 AM — Merged. Applied. Subnets exist. Outputs published.

10:36 AM — Run trigger fires: compute-prod auto-plans. No changes (compute hasn’t referenced the new subnets yet). No action needed.

Time to production: 35 minutes.

Phase 3: Storage Team — Payment Receipts S3 Bucket

Who: David (Storage Engineer) When: Tuesday 10:30 AM (works in parallel with networking — no dependency)

git checkout -b feature/payments-receipts-bucket

# infra-storage/payments.tf

resource "aws_s3_bucket" "payment_receipts" {
  bucket = "acme-payment-receipts-prod"

  tags = {
    Team        = "storage"
    Service     = "payments-api"
    Environment = "prod"
    DataClass   = "pci"
    ManagedBy   = "terraform"
  }

  lifecycle {
    prevent_destroy = true    # Never accidentally delete payment data
  }
}

resource "aws_s3_bucket_versioning" "payment_receipts" {
  bucket = aws_s3_bucket.payment_receipts.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "payment_receipts" {
  bucket = aws_s3_bucket.payment_receipts.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_public_access_block" "payment_receipts" {
  bucket                  = aws_s3_bucket.payment_receipts.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

10:45 AM — PR opened. Pipeline runs:

• ✅ fmt, validate
• ✅ terraform plan → 4 to add
• ❌ Sentinel FAILS: “Bucket tagged DataClass=pci must have lifecycle policy for compliance”

David forgot the lifecycle policy for PCI-classified data. Sentinel caught it before production. This is exactly why policy-as-code exists — David didn’t know about the PCI lifecycle requirement, and he didn’t need to. The policy knew.

He adds the missing piece:

resource "aws_s3_bucket_lifecycle_configuration" "payment_receipts" {
  bucket = aws_s3_bucket.payment_receipts.id

  rule {
    id     = "archive-after-90-days"
    status = "Enabled"

    transition {
      days          = 90
      storage_class = "GLACIER"
    }

    expiration {
      days = 2555    # 7 years — PCI compliance retention
    }
  }
}

10:55 AM — Pushes fix. Pipeline re-runs:

• ✅ All checks pass
• Reviewed and approved by Lisa (Storage team lead)

11:00 AM — Merged. Applied. Bucket exists — encrypted, private, versioned, with a lifecycle policy that satisfies PCI compliance.

Time to production: 30 minutes (including the Sentinel failure and fix). No compliance finding in the next audit because the policy caught it before it existed in production.

Phase 4: Compute Team — ECS Fargate Service

Who: Alex (Platform Engineer) When: Tuesday 11:00 AM (after security, networking, and storage are done)

Alex’s config is the convergence point — it consumes outputs from all three upstream teams.

git checkout -b feature/payments-api-service

# infra-compute/data.tf

# Read networking outputs
data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-terraform-state"
    key    = "network/prod/terraform.tfstate"
    region = "us-east-1"
  }
}

# Read security outputs
data "terraform_remote_state" "security" {
  backend = "s3"
  config = {
    bucket = "acme-terraform-state"
    key    = "security/prod/terraform.tfstate"
    region = "us-east-1"
  }
}

# infra-compute/payments.tf

resource "aws_security_group" "payments_api" {
  name        = "payments-api-sg"
  description = "Security group for Payments API service"
  vpc_id      = data.terraform_remote_state.network.outputs.vpc_id

  ingress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Team        = "compute"
    Service     = "payments-api"
    Environment = "prod"
  }
}

resource "aws_ecs_task_definition" "payments_api" {
  family                   = "payments-api"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = 512
  memory                   = 1024
  execution_role_arn       = aws_iam_role.ecs_execution.arn
  task_role_arn            = data.terraform_remote_state.security.outputs.payments_api_task_role_arn

  container_definitions = jsonencode([{
    name      = "payments-api"
    image     = "acme/payments-api:v1.2.0"
    essential = true
    portMappings = [{
      containerPort = 8080
      protocol      = "tcp"
    }]
    environment = [
      { name = "S3_BUCKET", value = "acme-payment-receipts-prod" },
      { name = "AWS_REGION", value = "us-east-1" }
    ]
    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = "/ecs/payments-api"
        "awslogs-region"        = "us-east-1"
        "awslogs-stream-prefix" = "payments"
      }
    }
  }])
}

resource "aws_ecs_service" "payments_api" {
  name            = "payments-api"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.payments_api.arn
  desired_count   = 3
  launch_type     = "FARGATE"

  network_configuration {
    subnets         = data.terraform_remote_state.network.outputs.payments_subnet_ids
    security_groups = [aws_security_group.payments_api.id]
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.payments_api.arn
    container_name   = "payments-api"
    container_port   = 8080
  }

  depends_on = [aws_lb_listener_rule.payments_api]
}

resource "aws_lb_target_group" "payments_api" {
  name        = "payments-api-tg"
  port        = 8080
  protocol    = "HTTP"
  vpc_id      = data.terraform_remote_state.network.outputs.vpc_id
  target_type = "ip"

  health_check {
    path                = "/health"
    healthy_threshold   = 2
    unhealthy_threshold = 3
    interval            = 30
  }
}

resource "aws_lb_listener_rule" "payments_api" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 200

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.payments_api.arn
  }

  condition {
    path_pattern { values = ["/api/payments/*"] }
  }
}

11:20 AM — PR opened. Pipeline runs:

• ✅ fmt, validate
• ✅ terraform plan → 5 to add
• ✅ Sentinel: ECS tasks must have a task role ✓
• ✅ Sentinel: Fargate services must have desired_count >= 2 ✓
• ✅ Sentinel: Security groups must not allow 0.0.0.0/0 ingress ✓
• ✅ Infracost: Estimated monthly cost increase: +$47.30 (3 Fargate tasks)

11:35 AM — Reviewed by Maria (Senior Platform Engineer):

• Task role comes from security team’s state (correct ARN) ✓
• Subnets come from networking team’s state (correct reference) ✓
• Health check configured ✓
• Desired count = 3 across 2 AZs ✓
• Approves.

11:40 AM — Merged. Applied. ECS service is running. ALB routes /api/payments/* to the new service. The Payments API is live.

Time to production: 40 minutes.

The Complete Timeline

 9:00 ──── Security team starts (IAM roles)
 9:35 ──── Security APPLIED ✅
10:00 ──── Network team starts (subnets)
10:30 ──── Storage team starts (S3 bucket) ← parallel with networking
10:35 ──── Network APPLIED ✅
10:45 ──── Storage PR fails Sentinel ❌
10:55 ──── Storage fixes and re-pushes
11:00 ──── Storage APPLIED ✅
11:00 ──── Compute team starts (ECS service)
11:40 ──── Compute APPLIED ✅
─────────────────────────────────────────
Total wall-clock time: 2 hours 40 minutes
Total downtime: 0
Console logins: 0
People who ran terraform apply from a laptop: 0

Four teams. Four independent workstreams. One running microservice. Zero console clicks. Every change in Git. Every approval recorded.

Day 2: Drift Detection Catches a Problem

Thursday 2:00 PM. HCP Terraform’s scheduled drift detection runs against networking-prod.

Result: Drift detected. Someone modified a security group rule directly in the AWS console — added an ingress rule allowing port 22 from 0.0.0.0/0.

This is the scenario that keeps security teams up at night. And it happens constantly.

What happens:

1. HCP Terraform flags the workspace as “drifted”
2. Notification fires to #infra-alerts Slack channel
3. Sarah checks AWS CloudTrail — finds that an on-call developer opened SSH access during a 3 AM incident and forgot to close it
4. Sarah opens a PR to revert: terraform apply will remove the rogue rule because it’s not in code
5. PR is opened. Plan shows: 1 security group rule to destroy. Reviewed. Merged. Applied.

Infrastructure matches code again. The console change is gone.

This is the pattern: reality drifts from code. Drift detection catches it. A human decides whether to accept or revert. Either way, code and reality are back in sync.

One thing drift detection won’t tell you: why the drift happened. It sees that reality doesn’t match state, but not who made the change or why. Always check CloudTrail (AWS), Audit Logs (GCP), or Activity Log (Azure) before blindly reverting. Sometimes the manual change was a legitimate emergency fix that should be codified, not erased.

Emergency Changes: The “Break Glass” Procedure

Not every change goes through the full PR flow. Sometimes production is on fire and you need to act now.

The break-glass protocol:

1. Acknowledge — Post in #incident-response that you’re making an emergency change
2. Apply directly — Use HCP Terraform’s CLI-driven mode or the API to run plan/apply immediately, bypassing the PR flow
3. Override policies — Soft-mandatory Sentinel policies can be overridden by admins. Hard-mandatory cannot (by design — some rules are never broken)
4. Remediate — Within 24 hours, open a PR that codifies the emergency change
5. Postmortem — Document why the normal flow was insufficient

# Emergency CLI apply (requires appropriate HCP Terraform permissions)
terraform plan -out=emergency.tfplan
# Review the plan output carefully — you're bypassing peer review
terraform apply emergency.tfplan

The critical rule: emergency changes bypass the PR flow, not Terraform itself. You still use Terraform. You still get a plan. You still apply through the normal tool. You just skip the peer review gate.

If someone bypasses Terraform entirely and changes things in the console, drift detection catches it. The team decides: codify it or revert it.

I’ve seen organizations that allow emergency console changes without enforcing follow-up Terraform PRs. Within six months, their code and infrastructure were unrecognizable. Nobody trusted the Terraform configs anymore. At that point, you’ve lost the entire value of IaC. Enforce the “remediation PR within 24 hours” rule. No exceptions.

The Approval Chain — Who Signs Off

The “four eyes” principle: No single person both writes and applies a production change. The person who wrote the code is not the person who approves the apply. In regulated industries — finance, healthcare, government — this is non-negotiable. Everywhere else, it’s just good practice.

How Teams Stay Decoupled

The key insight: teams don’t wait for each other except when they have actual data dependencies.

• Security and Storage teams worked in parallel (no dependency between them)
• Networking team worked independently
• Compute team waited for all three — but only because it consumed their outputs

If the compute team used data sources with tags instead of terraform_remote_state, they could have opened their PR even before the upstream teams finished. The plan would fail (resources not found), but the code could be reviewed and ready to merge the moment upstream resources exist.

Security ────────────────→ Applied
Networking ──────────────→ Applied
Storage ─────────────────→ Applied
                                   ↘
Compute ─────── PR Ready ─────────→ Applied

Structure your work so that code review happens in parallel with upstream deployments. The compute team’s PR can be reviewed while networking is still applying. The merge waits for dependencies, but the review doesn’t have to.

What Makes This Work

1. Separate state files — Teams can’t break each other’s infrastructure
2. CODEOWNERS — Code review is enforced by the right people
3. Policy-as-code — Mistakes caught before production (David’s missing lifecycle policy)
4. Saved plan files — What you reviewed is exactly what gets applied
5. Run triggers — Downstream impacts automatically detected
6. Drift detection — Manual console changes caught and remediated
7. Dynamic credentials — No static secrets to leak or rotate
8. Audit trail — Every change in Git, every apply logged, every approval named

This isn’t theoretical. This is how companies like Stripe, Shopify, HashiCorp, and hundreds of others run their infrastructure. The specific tools vary — some use Atlantis instead of HCP Terraform, some use GitLab CI instead of GitHub Actions — but the pattern is always the same.

Code → Review → Policy → Approve → Apply → Monitor → Detect drift → Remediate.

That’s the loop. Run it well and your infrastructure becomes boringly reliable. Which is exactly what you want.

Key Takeaways

The most common failure mode isn’t technical — it’s organizational. Teams skip the PR flow “just this once” during an incident, then never create the follow-up PR. Six months later, the Terraform code and actual infrastructure are completely out of sync. Automate the remediation PR requirement if you can.

Cross-team terraform_remote_state creates implicit ordering. If the security team needs to destroy and recreate an IAM role, the compute team’s next plan will fail until the role ARN output is updated. Communicate breaking changes across team boundaries — Terraform won’t do this for you.

Draw your team dependency graph. If it looks like a tangled mess, your workspace boundaries are wrong. The ideal graph is a clean DAG — security and networking at the bottom, storage in the middle, compute at the top. Changes flow up. Outputs flow down.