Cloud NAT & Network Reliability Patterns

In the cloud, “no public IP” is a security best practice. Your VMs should be private. Your GKE nodes should be private. Your Cloud Run services should be private. But private VMs still need to reach the internet—to pull package updates, download container images, call third-party APIs, or push telemetry data. How does a VM with no public IP reach the internet? The answer is Cloud NAT (Network Address Translation).

Cloud NAT is not a VM you deploy. It is not a software appliance running in your project. It is a fully managed, distributed service that runs on Google’s infrastructure outside your VPC. There is no single point of failure, no bottleneck machine, and no instance to patch. Google handles the scaling, the redundancy, and the failover. Your traffic simply appears to originate from one of the NAT gateway’s external IP addresses.

How Cloud NAT Works
The Flow
  1. Your private VM (10.0.0.5) sends a packet to pypi.org.
  2. Cloud NAT intercepts the outbound packet.
  3. It replaces the source IP (10.0.0.5) with one of the NAT gateway’s external IPs (e.g., 34.100.0.1).
  4. It assigns a source port from its port pool.
  5. The response from pypi.org comes back to 34.100.0.1:12345.
  6. Cloud NAT translates it back to 10.0.0.5 and delivers it to your VM.
Key Characteristics
  1. Outbound only. Cloud NAT does not allow unsolicited inbound connections. You cannot use it to expose a service to the internet.
  2. Regional. A Cloud NAT gateway serves VMs in a single region. Multi-region deployments need one NAT gateway per region.
  3. No VM required. Unlike traditional NAT gateways (e.g., AWS NAT instances), there is no compute resource to manage.
The Port Exhaustion Problem

This is the most critical operational issue with Cloud NAT, and the exam tests it heavily.

Each NAT translation requires a unique tuple of (External IP, Source Port). The total number of simultaneous connections is limited by the number of available ports. By default, each VM is allocated 64 ports per NAT IP. If your VM opens more than 64 simultaneous connections to the same destination, it runs out of ports—and new connections fail silently.

Symptoms of Port Exhaustion
  1. Random connection timeouts to external APIs.
  2. curl commands hanging intermittently.
  3. Cloud NAT logs showing OUT_OF_RESOURCES errors.
Solutions
  1. Increase the minimum ports per VM: Change the allocation from 64 to 1024 or higher.
  2. Add more NAT IPs: Each additional external IP gives you ~64,000 more ports.
  3. Enable Dynamic Port Allocation (DPA): Lets Cloud NAT dynamically assign ports based on demand. VMs that need more ports get more; quiet VMs get fewer.

If the exam describes “intermittent connection failures from private VMs,” and Cloud NAT is in the architecture, the answer is almost always port exhaustion.

Endpoint-Independent Mapping

Cloud NAT supports Endpoint-Independent Mapping (EIM), which means the same internal IP:port is always mapped to the same external IP:port, regardless of the destination. This is critical for protocols that require consistent source endpoints, like:

  1. WebRTC (real-time video calls).
  2. SIP (VoIP telephony).
  3. Gaming servers that validate source IPs.

Without EIM, different destinations would see different source ports from the same VM, breaking protocols that expect consistency.

Cloud NAT for GKE

GKE private nodes use Cloud NAT to reach the internet. But the port math changes because each node runs dozens or hundreds of Pods, each potentially making external connections. You must plan your NAT IP pool and port allocation based on:

  1. Number of nodes.
  2. Pods per node.
  3. External connections per Pod.

A common mistake is deploying a single NAT IP for a GKE cluster with 50 nodes and 100 Pods each—the port pool is exhausted within minutes under load.

Putting it Together: A Pro-Engineer View

Imagine you are running a data processing pipeline where 200 private GKE Pods each call an external REST API at high concurrency. You deploy Cloud NAT with 10 external IPs, set Dynamic Port Allocation to a minimum of 256 and a maximum of 4096 ports per VM, and enable logging to monitor port utilization. You set up a Cloud Monitoring alert for when port usage exceeds 80%. When the pipeline scales up during peak hours, DPA automatically allocates more ports to busy nodes. When it scales down, ports are reclaimed. No manual intervention. No dropped connections. That is Cloud NAT in production—invisible, scalable, and self-healing.

Related Posts