2026-07-20 · trace
When I run kubectl apply, what really happens between my terminal and a running Pod?
kubectl apply -f deploy.yaml, and a few seconds later there's a Pod running.
For a long time I treated that gap as a single magic step. Then I tried to name
every component the request actually touches, in order — and the magic turned
into a sequence of small, boring, well-defined jobs.
The question I kept asking: at each hop, who is doing the work, and what are they reacting to?
The trace
1. kubectl just talks to the API server
kubectl isn't special. It authenticates and sends a plain HTTP request to the
API server. That's the only component it talks to — and, it turns out, the
only component anything talks to.
2. The API server validates and persists — then stops
The API server authenticates the request, runs authorization and admission checks, validates the object, and writes it to etcd. And then, crucially, it does nothing else. It does not schedule the Pod. It does not start a container. Its job is: be the front door to the cluster's state, and keep that state in etcd. etcd is the single source of truth; everything else is a cache of it.
3. Controllers turn one object into many
A Deployment doesn't run anything by itself. The Deployment controller is a loop watching the API server; it sees the new Deployment and creates a ReplicaSet. The ReplicaSet controller sees that and creates the Pod objects — which at this point are just records in etcd with no node assigned. Each controller does one small translation and writes the result back through the API server.
4. The scheduler only decides where
The scheduler watches for Pods with no node. For each one it filters the nodes that could run it and scores the rest, then does exactly one thing: writes the chosen node name onto the Pod object. It doesn't contact the node. It doesn't start anything. It just records a decision.
5. The kubelet makes it real
Every node runs a kubelet, watching the API server for Pods assigned to its node. When it sees one, it finally does the physical work: it calls the container runtime (through the CRI) to pull the images and start the containers, sets up networking, and reports status back. Only here — at the very end — does software actually run.
What I understood
- Nothing pushes; everything watches. kubectl → API server is the only direct call. After that, each component is an independent loop watching shared state and reacting. That's why Kubernetes is resilient: any component can restart and just resume reconciling.
- The API server + etcd are the whole truth. Every other component is stateless-ish and rebuildable, because the real state lives in one place.
- Each step does almost nothing. Controllers translate, the scheduler decides, the kubelet executes. The power is in composing tiny reconcilers, not in any one clever component.
- "Declarative" finally clicked. I don't tell Kubernetes to do things. I write down what should be true, and a chain of watchers works to make it true and keep it true.