Network Operations, Observability & Troubleshooting

Building a network is one thing. Keeping it running is another. In production, the most dangerous state is not “broken”—it is “silently degraded.” A misconfigured firewall rule that blocks 5% of traffic. A route that sends packets on a suboptimal path. A health check that reports healthy when the application is returning 500 errors. These issues do not trigger alarms until a customer complains. Google Cloud provides a suite of observability and diagnostic tools under the umbrella of Network Intelligence Center, and mastering them is essential for the PCNE exam.

The philosophy behind Google’s network operations tooling is: you should never have to SSH into a machine to diagnose a network issue. Every packet, every flow, every dropped connection should be visible from the Cloud Console or an API.

VPC Flow Logs: The Packet Diary

VPC Flow Logs capture a sample of network traffic flowing to and from your VM network interfaces. They do not capture the packet payload—they capture metadata: source IP, destination IP, source port, destination port, protocol, bytes transferred, and whether the packet was allowed or denied.

Configuration
  1. Enabled per subnet, not per VM.
  2. Aggregation Interval: How often flows are grouped (5 seconds to 15 minutes). Shorter intervals = more detail = more cost.
  3. Sample Rate: What fraction of flows to capture (0.0 to 1.0). A rate of 0.5 samples half the flows.
Use Cases
  1. Security forensics: “Which IPs are connecting to my database on port 5432?”
  2. Cost optimization: “How much egress traffic is flowing between regions?”
  3. Troubleshooting: “Are packets from subnet A reaching subnet B?”

Flow Logs are the foundation of network observability. If you cannot see the traffic, you cannot diagnose the problem.

Firewall Rules Logging

Separate from Flow Logs, Firewall Rules Logging records every connection that matches a firewall rule—whether it was allowed or denied. This is critical for:

  1. Auditing: “Prove to the compliance team that no unauthorized SSH connections were allowed.”
  2. Debugging: “My application cannot connect. Was the packet dropped by a firewall rule?”
  3. Optimization: “Which firewall rules have never been matched?” (Firewall Insights provides this.)
Network Intelligence Center

This is Google’s unified network diagnostics dashboard. It contains several tools:

1. Connectivity Tests

The killer feature. You specify a source (VM, IP, GKE Pod) and a destination (another VM, an external IP, a load balancer). Google simulates the packet path through VPCs, firewall rules, routes, NAT, and load balancers—and tells you exactly where it would succeed or fail.

  • You do not need to generate real traffic.
  • The test runs against the current configuration, not live packets.
  • It identifies the specific firewall rule, route, or NAT config that would block the packet.
2. Network Topology

A visual map of your VPC resources—VMs, subnets, load balancers, Interconnects—and the traffic flowing between them. Useful for understanding traffic patterns and identifying hotspots.

3. Firewall Insights

Analyzes your firewall rules and identifies:

  1. Shadowed rules: Rules that will never match because a higher-priority rule covers the same traffic.
  2. Overly permissive rules: Rules that allow more traffic than necessary.
  3. Unused rules: Rules that have not matched any traffic in 90 days.
4. Performance Dashboard

Shows latency and packet loss between your VMs and Google’s network edge. Useful for validating whether “the network is slow” is actually a network problem or an application problem.

Putting it Together: A Pro-Engineer View

Imagine you are on call and a developer reports that their application in us-central1 cannot connect to a Cloud SQL instance in us-east1. Instead of SSHing into the VM and running tcpdump, you open Network Intelligence Center and run a Connectivity Test from the VM’s IP to the Cloud SQL private IP on port 5432. The test instantly shows that a Hierarchical Firewall Policy at the Folder level is denying TCP 5432 because the Security team recently tightened the rules. You check the Firewall Rules Log and confirm: 47 connection attempts denied in the last hour. You file a change request with the security team, they add a goto_next exception, and the connection works within minutes—all without touching a single machine.