Your application is deployed. GKE cluster is humming, databases are ready, functions are awaiting triggers. You have built the ship. Now you have to sail it — and sailing in the cloud without visibility feels like navigating through fog.
Is the application healthy? Are users getting errors? Why did that one request take five seconds? Without the right tools, your system is a complete black box. This is the problem observability solves — understanding the internal state of your system by looking at its external outputs.
In Google Cloud, observability is handled by the Cloud Operations suite (formerly Stackdriver). It is not one tool but five interconnected services. Let me walk you through a real investigation scenario to show how they work together.
Cloud Monitoring — The Red Alert System
It is 3 AM. You get woken up by a notification. This is where every investigation starts.
Cloud Monitoring watches your system’s performance metrics and tells you when something is wrong:
- Metrics — Time-series measurements of anything: VM CPU utilization, load balancer latency, Pub/Sub message count, custom application metrics
- Dashboards — Your main viewscreen. Build custom charts for an at-a-glance view of system health
- Uptime Checks — Sentinels that continuously probe your public endpoints from around the world
- Alerting Policies — Watch a metric. When a condition is met (e.g., “5xx error rate > 5% for 5 minutes”), send a notification via PagerDuty, Slack, or email. Always include documentation — a link to a runbook telling the on-call engineer what to do
Our alert says the 5xx error rate is high. We know what is wrong. Now we need to find out why.
Cloud Logging — The Ship’s Logbook
Cloud Logging is the centralized, searchable library for every log from every service in your project. Most GCP services stream logs here automatically. For VMs, you install the Ops Agent.
Using the Log Explorer, you filter for severity=ERROR in the last 15 minutes. You find hundreds of entries:
Error: Request to payment-service timed out after 3000ms
Now we have a clue. It is not our main application that is failing — it is failing because its downstream call to the payment-service is timing out. But the payment service calls three other microservices. Where is the actual delay?
Cloud Trace — Following the Request Path
Cloud Trace is a distributed tracing system. Think of it as GPS tracking for your requests.
By instrumenting your code with a client library, Trace assigns a unique ID to each request and visualizes how much time was spent in each service call (each “span”). You pull up a trace for a failed request and see a waterfall graph:
The request spent 2,950ms of its 3,000ms timeout waiting for a response from the fraud-detection-service. We found the slow service.
Cloud Profiler — The Code Magnifying Glass
We know which service is slow, but why? Is it stuck in a loop? Memory allocation issue?
Cloud Profiler continuously analyzes CPU and memory usage of your applications in production with very low overhead. You pull up the CPU profile for fraud-detection-service:
The flame graph shows a function called calculateRiskScoreV2 is consuming 95% of CPU time. We narrowed the problem from a high-level alert to a single function.
Cloud Debugger — Production X-Ray
The calculateRiskScoreV2 function is complex, and the bug only happens with specific user data that you cannot easily reproduce in test. You need to see what is happening inside that function in production, without stopping the service.
Cloud Debugger lets you inspect the state of a live, running application without stopping or slowing it down. You set a snapshot on a key line inside the function. Next time a request hits that line, Debugger captures the full call stack and all local variable values.
The snapshot reveals: a variable that should contain a user ID is occasionally null, causing the algorithm to enter a massive recalculation loop. Root cause found.
The Complete Workflow
Here is how the five tools map to the investigation flow:
| Question | Tool |
|---|---|
| Is something wrong? | Cloud Monitoring — Alerts and dashboards |
| What is the specific error? | Cloud Logging — Log Explorer |
| Where is the latency in my distributed system? | Cloud Trace — Request waterfall |
| Which function is using the most CPU/memory? | Cloud Profiler — Flame graphs |
| What is the live state of my code at this moment? | Cloud Debugger — Production snapshots |
Log Sinks — Archiving Logs
Storing all logs in Cloud Logging forever is expensive. Sinks let you automatically export logs to other destinations:
- Cloud Storage — Cheap, long-term archival
- BigQuery — Complex SQL-based analysis
- Pub/Sub — Stream logs to external tools like Splunk
# Create a sink that exports ERROR logs to Cloud Storage
gcloud logging sinks create error-archive \
storage.googleapis.com/my-log-bucket \
--log-filter="severity>=ERROR"
Common Pitfalls and Best Practices
Pitfall: Not installing the Ops Agent on GCE VMs. You are blind to what is happening inside instances.
Best Practice: Install the Ops Agent on all VMs as part of your standard build process.
Pitfall: Noisy alerts that cause alert fatigue.
Best Practice: Tune alert conditions with appropriate durations. Always include documentation and runbook links.
Pitfall: Not instrumenting code for Trace and Profiler.
Best Practice: Add client libraries during development so you have deep visibility when you need it.
Pitfall: Keeping all logs in Cloud Logging indefinitely.
Best Practice: Use Sinks. Keep logs in Logging for 30 days, archive the rest to Cloud Storage.
Quick Reference
# Read logs
gcloud logging read "severity>=ERROR" --limit=20
# Create a log sink to Cloud Storage
gcloud logging sinks create [SINK] \
storage.googleapis.com/[BUCKET] \
--log-filter="severity>=ERROR"
# List notification channels
gcloud alpha monitoring channels list
# Describe an uptime check
gcloud alpha monitoring uptime-checks describe [CHECK]
