Storing Highly Dynamic Cluster State
In a cluster running hundreds of containers across dozens of nodes, tracking the active configuration is a massive challenge. You have to store IP addresses, security keys, pod configurations, and scaling states somewhere.
If you use a traditional relational database (like PostgreSQL or MySQL), you face serious issues. Adding a new field to a resource definition shouldn’t require running database migrations across a live production cluster. Additionally, relational databases can struggle with the massive read-and-write throughput of highly dynamic workloads without complex setups. Concurrency conflicts in a multi-master environment can easily lead to data mismatches or split-brain outages.
Relational databases rely on ACID transactions that often scale using pessimistic locks. In a Kubernetes environment where resource states change millisecond-by-millisecond, locking tables or rows would cause API Server latency spikes. We need a datastore that supports concurrent, non-blocking writes and guarantees absolute consistency.
The Key-Value Approach to Consensus
Kubernetes solves this state storage challenge using etcd. It is a strongly consistent, distributed key-value store.
Unlike relational databases that organize data into tables, columns, and rows, a key-value store acts like a massive dictionary. You store data against a unique string key (such as /registry/pods/default/nginx), and the value is the raw configuration data. This flat data model is fast for reading and writing specific keys.
Under the hood, etcd v3 implements a Multi-Version Concurrency Control (MVCC) data model. When you update a key, etcd does not overwrite the old value. Instead, it appends a new revision of the key. This allows the API Server to watch resource keys for changes by reading updates sequentially from a specific revision number, making real-time event watching highly efficient.
You can install etcd directly on a server binary. Here is the process for installing it on a Linux system:
# Download the release tarball
$ curl -LO https://github.com/etcd-io/etcd/releases/download/v3.5.14/etcd-v3.5.14-linux-amd64.tar.gz
# Extract the files
$ tar -xvf etcd-v3.5.14-linux-amd64.tar.gz
# Move the binaries to your path
$ sudo mv etcd-v3.5.14-linux-amd64/etcd* /usr/local/bin/
# Start the etcd service locally
$ etcd
etcd went through a significant architecture change between version 2 and version 3. Version 2 used a hierarchical filesystem-like structure and relied on HTTP/JSON for transport, keeping all data in memory. This led to major memory consumption. Version 3 introduced a flat key space, binary gRPC communication via Protocol Buffers, and disk-backed b-tree storage.
If you work with the CLI, make sure to set the ETCDCTL_API=3 environment variable. Otherwise, the tool defaults to the older v2 API, which uses different commands and will not see your v3 data.
Reading, Writing, and Backing Up the Datastore
Let’s interact with our running etcd instance using the command-line client, etcdctl. First, set the API version in your shell:
$ export ETCDCTL_API=3
Now, let’s write a key-value pair representing a configuration key:
$ etcdctl put /configs/users/kapil "14-year-architect"
OK
You can retrieve the value using the get command:
$ etcdctl get /configs/users/kapil
/configs/users/kapil
14-year-architect
If you only want the value returned without the key name header, append the print value flag:
$ etcdctl get /configs/users/kapil --print-value-only
14-year-architect
You can also write conditional transactions. For example, write a key only if it does not already exist:
$ etcdctl txn
# This opens an interactive transaction prompt:
# compare:
value("/configs/users/kapil") = "14-year-architect"
# success:
put /configs/users/status "active"
# failure:
put /configs/users/status "unknown"
To safeguard your cluster state, you must perform regular database snapshots. Here is how you take a hot backup of a running etcd database:
$ etcdctl snapshot save backup.db
Snapshot saved at backup.db
To restore the state from a snapshot file (useful when recovering from a corrupted cluster):
$ etcdctl --data-dir=/var/lib/etcd-from-backup snapshot restore backup.db
In a Kubernetes environment, no controller can function without etcd. If etcd is slow or offline, your cluster becomes unresponsive to changes. Understanding how to interact with it directly and perform snapshots is a critical skill for recovering a damaged control plane.
