Building a Kubernetes controller in Rust: from reconciliation to a reliable system

Think about a thermostat. You say you’d like the room at 22 degrees, and then you stop thinking about it. The thermostat doesn’t remember what it did a minute ago or follow a script. It looks at the room, compares it with what you asked for, and nudges things a little closer. Then it looks again.

A Kubernetes controller works the same way. It is probably the most useful idea in the whole system, and it is small enough to hold in your head. In this post we’ll build one in Rust with kube-rs, and then spend most of our time on the part that matters: what turns a loop that works on your laptop into one you can leave running.

What we’re building

We’ll make a custom resource called Echo. You tell the cluster the image you want and how many copies, and the controller makes sure a Deployment exists to match. It also reports back how many copies are actually ready.

apiVersion: example.dev/v1
kind: Echo
metadata:
  name: hello
spec:
  image: nginx:1.27
  replicas: 2

It’s a toy, but the shape is the same as many real controllers, including ones that program load balancers: read a desired state, make the outside world match it, write back what happened.

Before any code, here is the whole idea in one picture. Five small steps, repeated for as long as the controller lives:

Everything else in this post is a closer look at one of those boxes, or at what keeps the loop honest when something goes wrong.

Why Rust

If you write Go, you already have a fine road: client-go and controller-runtime are mature and well documented. Rust is a different kind of fine. The compiler makes you handle errors, and the type system helps you avoid mixing up the object you were asked about with the object you found. A controller runs for months and meets every odd case eventually, so that early strictness is comforting.

The cost is a steeper start and more ceremony. For a first controller, that’s a fair trade only if you’re curious about Rust. If you’re not, there’s no shame in Go.

The dependencies, as they were in kube-rs’s own example when I wrote this (check the current versions before you copy):

[dependencies]
kube = { version = "4", features = ["runtime", "client", "derive"] }
k8s-openapi = { version = "0.28", features = ["latest"] }
schemars = "1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
futures = "0.3"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
thiserror = "2"
tracing = "0.1"
tracing-subscriber = "0.3"

Describing the resource

Kubernetes needs to know what an Echo is. With kube-rs you write a struct and let a derive macro do the rest:

#[derive(CustomResource, Deserialize, Serialize, Clone, Debug, JsonSchema)]
#[kube(group = "example.dev", version = "v1", kind = "Echo", namespaced)]
#[kube(status = "EchoStatus")]
pub struct EchoSpec {
    pub image: String,
    pub replicas: i32,
}

#[derive(Deserialize, Serialize, Clone, Debug, Default, JsonSchema)]
pub struct EchoStatus {
    pub ready_replicas: i32,
}

The spec is what the user wants. The status is what the controller observed. Keeping them apart is a quiet but important habit: people write the spec, the controller writes the status, and neither steps on the other. The derive also gives you Echo::crd(), which produces the CustomResourceDefinition to install in your cluster, so the schema and the Rust type can’t drift apart.

The reconcile function

Here is the heart of it. The function receives the object that changed, plus some shared context, and returns what to do next:

async fn apply(echo: Arc<Echo>, ctx: Arc<Ctx>) -> Result<Action, Error> {
    let ns = echo.namespace().ok_or(Error::NoNamespace)?;
    let name = echo.name_any();
    let owner = echo.controller_owner_ref(&()).expect("Echo has a uid");

    let desired = json!({
        "apiVersion": "apps/v1",
        "kind": "Deployment",
        "metadata": { "name": name, "ownerReferences": [owner] },
        "spec": {
            "replicas": echo.spec.replicas,
            "selector": { "matchLabels": { "app": name } },
            "template": {
                "metadata": { "labels": { "app": name } },
                "spec": { "containers": [{ "name": "echo", "image": echo.spec.image }] }
            }
        }
    });

    let deployments: Api<Deployment> = Api::namespaced(ctx.client.clone(), &ns);
    let params = PatchParams::apply("echo-controller").force();
    let deployment = deployments.patch(&name, &params, &Patch::Apply(&desired)).await?;

    let ready = deployment.status.and_then(|s| s.ready_replicas).unwrap_or(0);
    let status = json!({
        "apiVersion": "example.dev/v1",
        "kind": "Echo",
        "status": EchoStatus { ready_replicas: ready },
    });
    let echoes: Api<Echo> = Api::namespaced(ctx.client.clone(), &ns);
    echoes.patch_status(&name, &params, &Patch::Apply(&status)).await?;

    if ready < echo.spec.replicas {
        Ok(Action::requeue(Duration::from_secs(15)))
    } else {
        Ok(Action::await_change())
    }
}

There are a few small decisions in there worth slowing down for.

It never asks “what changed?” It describes the whole Deployment it wants and hands that over with a server-side apply. If the Deployment doesn’t exist, Kubernetes creates it. If it exists but someone edited it, Kubernetes puts it back. If it’s already right, nothing happens. We don’t need separate create, update and no-op branches, which means there’s less code to get wrong.

It sets an owner reference. That tells Kubernetes the Deployment belongs to the Echo, so when the Echo is deleted, the Deployment goes with it, without any code from us.

It returns an Action. requeue(15s) means “look again soon, we’re not there yet”. await_change() means “I’m done until something changes”. That one return value is how the controller decides how busy to be.

Wiring it up

The rest is plumbing: connect to the cluster, say which resources to watch, and run.

#[tokio::main]
async fn main() -> Result<(), kube::Error> {
    tracing_subscriber::fmt::init();

    // `cargo run -- --crd` prints the CustomResourceDefinition and exits.
    if std::env::args().any(|a| a == "--crd") {
        println!("{}", serde_json::to_string_pretty(&Echo::crd()).unwrap());
        return Ok(());
    }

    let client = Client::try_default().await?;
    let echoes: Api<Echo> = Api::all(client.clone());
    let deployments: Api<Deployment> = Api::all(client.clone());

    Controller::new(echoes, Config::default())
        .owns(deployments, Config::default())
        .shutdown_on_signal()
        .run(reconcile, error_policy, Arc::new(Ctx { client }))
        .for_each(|result| async move {
            match result {
                Ok((obj, _)) => info!("reconciled {}", obj.name),
                Err(err) => warn!("controller error: {err}"),
            }
        })
        .await;
    Ok(())
}

The line to notice is .owns(deployments, ...). It means a change to a Deployment we created wakes up reconciliation of its parent Echo. If a pod crashes and the ready count drops, we find out without polling, and our status stays honest.

How the pieces fit together

It helps to know what kube-rs is doing for you, because each piece explains a reliability rule we’ll meet later.

Your controller never polls the cluster. It opens a watch, and the API server streams changes down to it. A watcher keeps a local cache of what it has seen, so reading an object doesn’t mean another trip to the API server. Each change drops the object’s name on a work queue, and the queue merges repeats: ten quick edits to the same Echo become a single reconcile. A worker takes a name, hands the cached object to reconcile, and follows whatever Action you return. As far as I know, kube-rs won’t run two reconciles for the same object at the same time, though different objects can go side by side, so one slow Echo doesn’t hold up the others.

Two things follow from that. The cache can lag the real cluster by a moment, so what you read is a good guess, not a promise. And the queue carries names, not events, so your function is never told what changed. It has to work that out for itself.

Levels, not edges

That second point has a name. A controller is level-triggered: it reacts to the current state of the world, not to the stream of events that led there.

An edge-triggered system reacts to moments. “The replica count went from 2 to 4.” If it misses one of those, because it restarted or a network hiccup swallowed an event, it stays wrong until somebody notices. A level-triggered system just asks, every time, “what should be true, and what is true?” Miss an event and nothing is lost, because the next look sees the same gap.

Go back to the thermostat. One that only reacted to the instant the room crossed 22 degrees would be useless if it happened to be asleep at that moment. A real one simply keeps checking. That’s why your controller is allowed to crash, restart and relist everything from scratch, and still end up in the right place.

Following one change

Here is what happens when you change replicas from 2 to 4:

1. The API server stores the new Echo and sends the change down the watch.
2. The watcher updates its cache, and the Echo’s name lands on the queue.
3. reconcile runs, builds the full Deployment with four replicas and applies it.
4. Kubernetes’s own Deployment controller starts two more pods. Only two are ready, so we patch ready_replicas: 2 into the status and ask to be looked at again in 15 seconds.
5. As pods become ready, the Deployment’s status changes. Because we used .owns(), that wakes our controller right away, and the 15 seconds turn out to be only a safety net.
6. Four are ready. We write that into the status and return await_change(). Nothing more to do.

Notice that no step needed to know about the others. Each pass just looked, compared and nudged.

Trying it out

You don’t need a real cluster for this. A local one made with kind is plenty. Print the CRD from the binary, install it, then run the controller on your own machine, where it talks to the cluster through your usual kubeconfig:

cargo run -- --crd | kubectl apply -f -
cargo run
kubectl apply -f echo.yaml
kubectl get deployment hello

Then do something rude. Delete the Deployment by hand, or scale it down, and watch the controller put it back. Change replicas in the Echo and watch the status catch up. A controller is much easier to trust once you’ve seen it recover.

At this point you have a working controller. It will also, sooner or later, misbehave. The next section is about why it doesn’t have to.

Making it reliable

A controller isn’t judged on the day everything goes well. It’s judged on the day the API server is slow, a node vanishes mid-update and someone deletes the object at the worst possible moment. A few habits cover most of it.

Assume you were interrupted

Your reconcile function can be stopped at any line, and it will be called again for the same object, maybe several times in a row. So never write it as a story (“first I create this, then I create that”) that assumes it ran from the top. Write it as a set of independent checks: does this exist, does it match, is the status right. If every step is safe to repeat, every crash is safe too.

That’s the real reason server-side apply suits controllers so well. Repeating it costs nothing.

Fail loudly, and retry politely

When something goes wrong, return an error and let the controller retry. Don’t swallow it and don’t loop inside the function.

#[derive(Debug, thiserror::Error)]
pub enum Error {
    #[error("kubernetes request failed: {0}")]
    Kube(#[from] kube::Error),
    #[error("Echo has no namespace")]
    NoNamespace,
    #[error("finalizer failed: {0}")]
    Finalizer(#[source] Box<kube::runtime::finalizer::Error<Error>>),
}

fn error_policy(_echo: Arc<Echo>, err: &Error, _ctx: Arc<Ctx>) -> Action {
    warn!(%err, "reconcile failed, will retry");
    Action::requeue(Duration::from_secs(30))
}

A typed error enum makes it obvious which failures you thought about. The error policy is your retry rule: here a flat 30 seconds, which is gentle on a struggling API server. For a controller with many objects, you’ll want to add some jitter or exponential backoff, so a thousand failures don’t all retry at the same instant.

Clean up after yourself

Owner references clean up other Kubernetes objects. But what if your controller also creates something outside the cluster, such as a DNS record, a cloud load balancer or a database user? Deleting the Echo would leave that behind, forever.

Finalizers fix this. A finalizer is a note on the object saying “don’t remove me yet, I have cleanup to do”. kube-rs has a helper that handles the bookkeeping:

async fn reconcile(echo: Arc<Echo>, ctx: Arc<Ctx>) -> Result<Action, Error> {
    let ns = echo.namespace().ok_or(Error::NoNamespace)?;
    let echoes: Api<Echo> = Api::namespaced(ctx.client.clone(), &ns);

    finalizer(&echoes, "echo.example.dev/cleanup", echo, |event| async {
        match event {
            Finalizer::Apply(echo) => apply(echo, ctx.clone()).await,
            Finalizer::Cleanup(echo) => cleanup(echo, ctx.clone()).await,
        }
    })
    .await
    .map_err(|e| Error::Finalizer(Box::new(e)))
}

Apply is the normal path. Cleanup runs once when the object is being deleted, and only after it succeeds is the finalizer removed and the object allowed to disappear. Here is that whole story in one picture:

Until that last box, the object is still there with a deletion timestamp, usually shown as Terminating. That’s deliberate: it keeps the cleanup in your hands, and it’s why a forgotten or broken finalizer can leave an object stuck, so test the delete path as carefully as the create one. The catch is that a buggy Cleanup can leave an object stuck in “Terminating”, so keep it small and make it forgiving: if the thing is already gone, that’s success.

Tell the truth in status

The status is how your controller talks to the people using it. Report what you observed, not what you hoped for. Our ready_replicas comes from the Deployment itself, not from the number we asked for. If you add conditions later (Ready, Degraded), a person or another controller should be able to read them and know what to do. A status that always says “OK” is worse than none.

The things to add before production

A short list for later, because none of it is needed to understand the idea:

• Leader election, if you’ll run more than one replica, so only one copy acts at a time. Check the kube-rs docs for the current options.
• Metrics and logs for reconcile time, error counts and queue depth. They’re how you’ll learn what normal looks like.
• Events, so kubectl describe shows people what the controller did and why.
• Tests, starting with the reconcile logic against a fake client, and finishing with a run against a real cluster such as kind.

Mistakes worth avoiding

Most controller bugs are variations on a few themes. It’s cheaper to meet them here than in the middle of the night.

• Waking yourself up. Writing to the status changes the object, and a change wakes the controller. Patching the same content again is a no-op, so that’s fine. But put a “last checked” timestamp in the status and you’ve built a loop that never rests.
• Requeueing too eagerly. requeue(1s) on an object that can’t be fixed turns a small problem into constant load. Be generous with the wait and let events do the real work.
• Panicking. Look closely at the expect on the owner reference earlier. It’s fine for a toy, but in a real controller a panic takes the whole process down, and every other object with it. Return an error instead.
• Two controllers, one field. If something else also edits the Deployment, the two of you will keep undoing each other. Server-side apply records who owns which field, which is exactly why we gave our controller a name when we applied.
• Trusting one test. A controller that works on a quiet kind cluster has told you very little. Kill it mid-reconcile, delete things by hand, and fail the API calls on purpose.

Where this is useful

Controllers are how Kubernetes does almost everything, and the pattern travels well:

• Internal platforms. Turn “a database, a bucket and a service account” into one small custom resource your teammates can write in ten lines of YAML.
• Keeping outside systems in step. DNS records, certificates, cloud load balancers and firewall rules, anywhere a cluster object should be mirrored somewhere else.
• Operations nobody enjoys. Backups, rotation, scaling rules and clean-up jobs that are easy to forget and awful to forget.

Whenever you catch yourself writing a script that you run by hand whenever things drift, there’s a good chance a controller wants to be written.

Key takeaways

1. A controller is a loop, not a script. It compares what’s wanted with what exists and closes the gap, again and again.
2. Describe the whole desired state and let server-side apply sort out the differences. It means fewer branches and fewer bugs.
3. Make every step safe to repeat. You will be interrupted, so write for it from the start.
4. Return errors and let the controller retry, with a policy you chose on purpose.
5. Use finalizers for anything outside the cluster, and keep the cleanup small and forgiving.
6. Keep spec and status apart, and make the status an honest report.

One last thing

The reconcile loop looks almost too simple to be impressive. It reads, compares, nudges and looks again. But that humility is exactly what makes it dependable. It doesn’t need to remember the past or predict the future. It only needs to look at the world as it is right now, and care a little about making it better.

If you’d like to try it, start with the toy above on a local kind cluster, break things on purpose, and watch it put them right. That’s the moment it clicks.