LAB09_GCP_Observability: Cloud Monitoring Alert Policies

Observability: Cloud Monitoring Alert Policies

☁ GCP 🟢 Beginner Cloud Monitoring Alert Policy MQL Observability

Scenario

You want to be notified if the average CPU utilisation of your Compute Engine instances goes above 80% for more than 5 minutes. You will use a Cloud Monitoring Alert Policy. This allows you to define monitoring criteria and thresholds programmatically using Terraform.

Your Objectives
  • Create a google_monitoring_alert_policy.
  • Define a condition of type condition_threshold looking at the metric compute.googleapis.com/instance/cpu/utilization.
  • Set the threshold value to 0.8 (80%) and the duration to 300s.
  • Set the policy so it combines multiple conditions with an AND operation (even though there's only one condition here).
  • Output the Policy ID.

Additional Context

Notification Channels: In a real-world scenario, you would also create a google_monitoring_notification_channel (e.g., Email, Slack, PagerDuty) and attach its ID to the alert policy via the notification_channels array. Here, we create the alert purely so the incident appears in the GCP Console Dashboard without spamming an inbox.

Aggregations: Monitoring systems track thousands of data points. The aggregations block tells GCP how to group those points (e.g., aggregate all instances together vs track each instance individually) and how to align data over time (e.g., take the mean average every 60 seconds).

variables.tf

variable "project_id" { type = string }

main.tf

provider "google" {
  project = var.project_id
}

# Cloud Monitoring Alert Policy
resource "google_monitoring_alert_policy" "high_cpu" {
  display_name = "High VM CPU Utilization (>80%)"
  combiner     = "OR"
  
  # Condition definition
  conditions {
    display_name = "CPU utilization above 80% for 5 mins"
    
    condition_threshold {
      # The GCP metric identifier for Compute Engine CPU utilization
      filter          = "metric.type=\"compute.googleapis.com/instance/cpu/utilization\" resource.type=\"gce_instance\""
      comparison      = "COMPARISON_GT"
      threshold_value = 0.8  # 80%
      duration        = "300s" # 5 minutes
      
      # How to bucket/combine the data streams before checking the threshold
      aggregations {
        alignment_period     = "60s"
        per_series_aligner   = "ALIGN_MEAN"
        # We don't use cross_series_reducer here because we want to alert on ANY individual VM crossing 80%,
        # not the average of ALL VMs crossing 80% together.
      }
    }
  }

  documentation {
    content   = "One or more Compute Engine instances have been sustaining >80% CPU utilization. Check the VM logs."
    mime_type = "text/markdown"
  }

  # In production, you would attach notification_channels = [google_monitoring_notification_channel.email.id]
}

outputs.tf

output "alert_policy_id" {
  value = google_monitoring_alert_policy.high_cpu.id
}

Workflow Commands

terraform init
terraform apply -var="project_id=my-gcp-project-123"

# View the created policy in the console: 
# GCP Console -> Monitoring -> Alerting -> Policies

terraform destroy -var="project_id=my-gcp-project-123"
✓ Expected Output After Apply
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

alert_policy_id = "projects/my-gcp-project-123/alertPolicies/876543210123"

Related Posts