The State Drift Problem
When running virtual servers and networks, configurations degrade over time. Machines run out of memory, processes crash, and network connections drop. If an orchestrator only updated the system when you manually typed a command, you would spend your time manually replacing dead nodes and containers. If a server hosting three critical services loses network connectivity, those services will stay offline until someone manually relocates them.
We need a continuous system that monitors the health of our infrastructure and acts on its own to resolve mismatches between what we requested and what is actually running.
Informer Caches and the Observe-Compare-Act Loop
The Kube Controller Manager (kube-controller-manager) handles this automation. It compiles multiple independent control loops into a single binary.
Each controller loop operates on a simple process. First, it observes the current state of a resource. Second, it compares that actual state with the desired state stored in the configuration database. Finally, it acts by sending API requests to reconcile any differences.
To avoid overloading the API Server with constant polling requests, controllers use a mechanism called an Informer. An Informer opens a persistent watch connection to the API Server and populates a local, memory-cached copy of the resource data. When a controller needs to inspect resources, it queries this local Lister cache instead of hitting the database. The Informer also triggers event handlers (like Add, Update, and Delete) to queue changes into a WorkQueue for processing.
Here are some of the critical built-in controllers:
* Node Lifecycle Controller: Monitors node health status. If a host node stops sending heartbeats for more than a set time (default: 40 seconds), the node controller flags the node as Unreachable and transitions it to NotReady. If the node remains dead for too long (default: 5 minutes), the controller evicts the pods, requesting the scheduler to recreate them on healthy machines.
* Replication Controller: Monitors replica counts, ensuring that the number of active pods matches your configuration specifications.
* Endpoints Controller: Watches services and pods, maintaining the lists of healthy IP addresses (Endpoints) used by Kube-proxy for traffic routing.
* ServiceAccount Controller: Provisions default API access credentials and tokens for newly created namespaces.
Observing a Self-Healing Loop
Let’s look at how the Controller Manager operates when a pod fails. First, verify that the controller manager static pod is active:
$ kubectl get pods -n kube-system | grep controller-manager
kube-controller-manager-control-plane 1/1 Running 4 (12d ago) 12d
Now, check a ReplicaSet that requires three running instances of Nginx:
$ kubectl get replicaset my-nginx-rs
NAME DESIRED CURRENT READY AGE
my-nginx-rs 3 3 3 10m
List the active pods associated with this ReplicaSet:
$ kubectl get pods -l app=nginx
NAME READY STATUS RESTARTS AGE
my-nginx-rs-abc12 1/1 Running 0 10m
my-nginx-rs-xyz34 1/1 Running 0 10m
my-nginx-rs-lmn56 1/1 Running 0 10m
We can simulate a container crash by deleting one of these pods manually:
$ kubectl delete pod my-nginx-rs-abc12
pod "my-nginx-rs-abc12" deleted
If you query the pods immediately, you will see a new container initializing:
$ kubectl get pods -l app=nginx
NAME READY STATUS RESTARTS AGE
my-nginx-rs-xyz34 1/1 Running 0 11m
my-nginx-rs-lmn56 1/1 Running 0 11m
my-nginx-rs-qrs78 0/1 ContainerCreating 0 1s
The Replication Controller observed that the running pod count dropped to two, compared it against the target count of three, and immediately asked the API Server to spin up a replacement pod. This automated loop keeps your cluster aligned with your desired state without manual intervention.
