Article 08 – Kubelet: The On-the-Ground Worker Agent

Executing the Orchestration Plan

The control plane components are great at coordinating and planning. They decide which containers should run and where they should be placed. However, these decisions are just configurations saved in the etcd database. The API Server cannot connect to a host machine directly, configure virtual interfaces, mount storage systems, and execute runtime commands.

You need an active agent running on the actual physical or virtual machines in the cluster to execute the control plane’s instructions and monitor host resources.

The Node Agent and PLEG Sync Loop

The Kubelet is the agent that runs on every node in the cluster, including the control plane host. It acts as the coordinator on the machine. It receives a configuration descriptor called a PodSpec and coordinates with the local container runtime to ensure the containers described are running and healthy.

To keep track of containers, the Kubelet runs a continuous Sync Loop. Under this loop, the Kubelet coordinates with a component called the Pod Lifecycle Event Generator (PLEG). PLEG detects state changes in the local container processes (like starting, stopping, or crashing) and generates corresponding events. The Kubelet processes these events in its sync loop to reconcile the local state with the desired specs from the API Server.

The Kubelet receives these PodSpecs from three main sources:
* The API Server: The Kubelet opens a stream connection to the API Server. If a pod is assigned to its node, the Kubelet downloads the spec and starts the container.
* A Local Directory: This is used for static pods. The Kubelet monitors a directory on the host (e.g., /etc/kubernetes/manifests). Any YAML file placed here is run directly on the host, bypassing the API Server.
* An HTTP Endpoint: The Kubelet can accept PodSpecs sent directly to a local network port.

Once the Kubelet has a PodSpec, it requests the container runtime (via CRI gRPC) to pull images, initialize namespaces, and start the processes. It also runs periodic checks called health probes:
* Startup Probe: Determines if the application inside the container has successfully started. All other probes are disabled until the startup probe succeeds.
* Liveness Probe: Determines if the container needs to be restarted. If it fails, the Kubelet kills the container and starts a new one.
* Readiness Probe: Determines if the container is ready to accept network traffic. If it fails, the service controller removes the pod’s IP from the active service endpoint list.

To announce the node’s availability, the Kubelet creates a Lease object in the kube-node-lease namespace. It updates this lease every 10 seconds. If the control plane stops receiving lease updates, it marks the node as NotReady.

Unlike the API Server or Scheduler, the Kubelet cannot run as a container inside Kubernetes. It must run directly on the host operating system as a systemd service.

Inspecting Kubelet Status and Logs

Let’s check the status of the Kubelet service running on our host machine:

$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2026-06-05 08:30:00 IST; 20min ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 12345 (kubelet)
      Tasks: 18 (limit: 4668)
     Memory: 48.2M
        CPU: 12.35s
     CGroup: /system.slice/kubelet.service
             └─12345 /usr/bin/kubelet --config=/var/lib/kubelet/config.yaml --kubeconfig=/etc/kubernetes/kubelet.conf

You can view the configuration file loaded by the service:

$ sudo cat /var/lib/kubelet/config.yaml

Inside this YAML file, you will find settings defining the container runtime socket path, cgroup driver, and the static pod manifest path:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
cgroupDriver: systemd
staticPodPath: /etc/kubernetes/manifests  # Location of static pod YAML configurations

If a node goes offline or containers fail to initialize, you can debug the Kubelet logs directly using systemd journal:

$ journalctl -u kubelet -f --no-tail

This log stream will show you if the Kubelet is having trouble contacting the API Server, failing to resolve DNS, running out of memory, or failing to pull images from your registry.

Related Posts