Article 07 – Kube Scheduler: Smart Placement in a Dynamic World

Placing Workloads Manually

When managing multiple machines, resource allocation is a constant challenge. Some servers have specialized hardware like GPUs, some have high storage limits, and others are low-resource virtual machines.

If you deploy a heavy analytics workload that requires 4 CPU cores and 16GB of RAM, and you choose a node at random, you might place it on a weak frontend server. The heavy container will starve the frontend of resources, crashing your primary web application. Choosing a host manually for every container requires you to constantly track the available memory, port allocations, and label constraints of every machine in your cluster.

The Filtering and Scoring Process

The Kube Scheduler (kube-scheduler) automates this placement. It watches the API Server for newly created pods that haven’t been assigned to a node. For each pod, the scheduler runs a two-step cycle: filtering and scoring.

During the filtering phase, the scheduler runs checks called predicates to identify which nodes can run the pod. For example, it evaluates:
* PodFitsResources: Does the node have enough allocatable CPU and memory to satisfy the pod’s container requests?
* PodFitsHostPorts: Is the port requested by the pod already occupied on the node?
* NodeSelectorMatches: Does the node match the nodeSelector labels specified in the Pod spec?
* CheckNodeSchedulability: Is the node cordoned or set to schedulable?
* CheckNodeTaints: Does the node have a “taint” (like node-role.kubernetes.io/control-plane:NoSchedule) that the pod does not tolerate?

In the scoring phase, the scheduler ranks the remaining candidate nodes using priorities. It scores each node from zero to ten. The node with the highest score is selected. The scoring priorities include:
* ImageLocalityPriority: Awards higher scores to nodes that already have the container image downloaded on disk, reducing container startup latency.
* LeastRequestedPriority: Prefers nodes with fewer requested resources, aiming to distribute the workload evenly across the cluster.
* BalancedResourceAllocation: Prefers nodes with balanced CPU and memory usage ratios to avoid resource skew.

Once the scheduler makes its choice, it writes a binding object back to the API Server, setting the pod’s node name. The local Kubelet on that node sees this binding and starts the container.

In a manually configured cluster, the scheduler runs as a local systemd service. In a kubeadm cluster, it runs as a static pod, and its configuration is defined by the manifest file at /etc/kubernetes/manifests/kube-scheduler.yaml.

You can also run multiple schedulers in a single cluster. By deploying a custom scheduler container, you can assign specific pods to use your scheduler instead of the default one by setting spec.schedulerName in your Pod specification.

Steering Workloads with Node Labels and Tolerations

Let’s inspect the scheduler configuration and look at how we can steer pods to specific nodes. View the scheduler manifest configuration:

$ sudo cat /etc/kubernetes/manifests/kube-scheduler.yaml

If you need to bypass the scheduler entirely for a specific workload, you can define the nodeName parameter in your Pod specification. However, this is fragile. A better approach is to use a nodeSelector, which asks the scheduler to filter nodes based on custom labels:

apiVersion: v1
kind: Pod
metadata:
  name: ml-job
spec:
  containers:
  - name: tensorflow-container
    image: tensorflow/tensorflow:latest
  nodeSelector:
    hardware: gpu  # Only schedules on nodes with this label

To make this schedule work, you need to apply the label to a node. Apply the label to your target worker:

$ kubectl label nodes worker-node-1 hardware=gpu
node/worker-node-1 labeled

You can also restrict nodes using Taints and Tolerations. Taints allow a node to repel a set of pods. For example, to prevent any pods from scheduling on a node unless they tolerate a specific key:

$ kubectl taint nodes worker-node-2 hardware=gpu:NoSchedule
node/worker-node-2 tainted

Now, the scheduler will filter out worker-node-2 for all pods, unless the pod explicitly declares a matching toleration:

spec:
  tolerations:
  - key: "hardware"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

Using these mechanisms, the scheduler balances workload isolation with optimal hardware resource utilization across your cluster.

Related Posts