Karpenter + Cluster API

One of Kubernetes’ great strengths is being able to manage our resources elastically. If a stateless pod consumes too much, we can add replicas; if it has irregular consumption, we slap an HPA (Horizontal Pod Autoscaler) on it and it adapts to the load. Our cluster is full? We can add nodes to it.

But while autoscaling replicas is automatic, doing the same for nodes isn’t necessarily trivial.

In my article about the Cluster API, I ended on a little teaser:

I still have other things to test with CAPI, notably integrating a program that manages auto-scaling (or even karpenter) […]

A few months late, admittedly, we’re tackling the autoscaling of a Kubernetes cluster deployed via the Cluster API.

But to achieve this node autoscaling step, two approaches are possible:

• Cluster Autoscaler, the solution designed to work 100% with CAPI.
• Karpenter, a solution that AWS donated to the CNCF and which also has a CAPI integration.

Which solutions for autoscaling#

Both know how to add (and remove) nodes, but with two opposite philosophies. Let’s compare them before going further into the PoC.

Cluster Autoscaler reasons in terms of machine groups: you give it node groups (MachineDeployments, I invite you to read my article that details this concept), and it plays with the number of replicas. If a pod doesn’t fit anywhere, it increments the replica count of a compatible group and tada: a new node joins the cluster.

Karpenter, on the other hand, reasons in terms of pod groups. It looks at the pods in Pending, computes the most suitable (and cheapest) machine to host them, and provisions it directly. No node group to size by hand: you describe constraints (NodePool) and Karpenter figures it out by finding the most suitable machine (even Spot instances that only exist for a limited time and are very cheap).

It’s worth knowing that Karpenter is historically very tied to AWS, but since its donation to the CNCF, the project’s core (sigs.k8s.io/karpenter) has become agnostic, and third-party providers have appeared. Including one that interests me particularly: karpenter-provider-cluster-api since it acts as the glue between Karpenter and CAPI-compatible cloud providers (using CAPI covers a maximum of providers, including OVH, but other clouds are also developing their own Karpenter providers for their managed offerings).

An important nuance before going further: natively (on AWS, Azure, GCP), Karpenter runs standalone. It lives inside the cluster it scales and talks directly to the cloud API without any third-party cluster: that’s actually its whole point, no management cluster. But here, going through the cluster-api provider, Karpenter no longer drives a cloud, it drives CAPI resources, which have to live somewhere. So we reintroduce a management cluster that Karpenter’s native operation doesn’t require. We’ll talk below about what that implies.

How does Karpenter work?#

Let’s first detail how Karpenter works within a Kubernetes cluster, and the resources (CRDs) it exposes:

• NodePool: these are the rules. It describes what Karpenter is allowed to create, in the form of requirements (architecture, capacity-type, zone, allowed instance types…), global limits (e.g. “no more than 20 vCPU total”), and a down-scaling policy. It points to a NodeClass.
• NodeClass: this is the provider-specific part. On AWS, an EC2NodeClass describes the AMI, subnets, security groups, IAM profile… Here, since we delegate everything to the Cluster API, we have a ClusterAPINodeClass.
• NodeClaim: this is the concrete request for a machine. When Karpenter decides a node is needed, it doesn’t create a VM directly: it creates a NodeClaim, a declarative object that corresponds to the request it will make to the provider. It’s then up to the provider to materialize it (on AWS = an EC2 instance; for us, it’s a scale-up of a MachineDeployment belonging to CAPI).

If we had to summarize the lifecycle of a Karpenter node, it would go like this:

1. Karpenter’s scheduler observes the non-schedulable pods (in Pending due to lack of resources/nodes);
2. It groups them into batches over a short window, to avoid creating one machine per pod;
3. For each batch, it simulates scheduling the way Kubernetes would (taking into account CPU/memory requests, affinities, taints/tolerations, topologySpreadConstraints) to deduce the most suitable node flavor: how much CPU, RAM, which architecture…;
4. Among the flavors compatible with the NodePool, it chooses the cheapest machine that does the job, and creates a NodeClaim;
5. the provider transforms this NodeClaim into a real machine (e.g. an EC2 instance on AWS, or a CAPI Machine in our case);

Where Cluster Autoscaler just increments the replica count of a pre-defined group and iterates, Karpenter starts from the real needs to choose the machine. It can therefore pick from very different machine types depending on the load, for example nodes with GPUs when a pod requires one.

Scale-down#

If we only ever add nodes without removing any, it’s not much use.

Karpenter’s strength is its active management of scale-down, which it calls disruption. It’s triggered by several mechanisms:

• Consolidation: if nodes are empty or underutilized, Karpenter moves the pods and deletes the surplus machines. This is the scale-down (and scale-to-zero) mechanism. Two policies: WhenEmpty (we only delete completely empty nodes) and WhenEmptyOrUnderutilized (it deletes underutilized nodes);
• Drift: if the actual configuration of a node no longer matches what its NodePool/NodeClass describe, Karpenter considers that the node is not in the desired state and recreates it.
• Expiration (expireAfter): we force node renewal after a given duration (rotation, image patching);

Creating our management cluster#

To use ClusterAPI, we need a “parent” cluster that will deploy “child” clusters. Any cluster would do as long as it has access to the API-Server and to the nodes (for example to send a configuration via SSH).

I use OVH’s Managed Kubernetes (MKS) for this management cluster, which brings a lot of convenience to the PoC. OpenTofu will make this setup easy to destroy / rebuild (and since I never write my articles in one shot, it lets me save some credits).

resource "ovh_cloud_project_kube" "mgmt" {
  service_name  = var.service_name   # = OS_PROJECT_ID
  name          = "capi-mgmt"
  region        = "GRA9"
  update_policy = "MINIMAL_DOWNTIME"
}

resource "ovh_cloud_project_kube_nodepool" "mgmt" {
  service_name  = var.service_name
  kube_id       = ovh_cloud_project_kube.mgmt.id
  name          = "default"
  flavor_name   = "b3-8"             # 2 vCPU / 8 GB
  desired_nodes = 2
  min_nodes     = 2
  max_nodes     = 2
}

The MKS cluster just serves as a support for the CAPI controllers. Nothing forces the workload cluster (the one that CAPO, the Cluster Api Provider Openstack, will manage) to be in the same region, though. Once the MKS is ready and the kubeconfig retrieved, we install the CAPI providers on it:

tofu output -raw kubeconfig > kubeconfig
export KUBECONFIG=$PWD/kubeconfig

kubectl apply --server-side \
  -f https://github.com/k-orc/openstack-resource-controller/releases/download/v2.6.0/install.yaml
clusterctl init --infrastructure openstack

Before the clusterctl init, you need to install the ORC (OpenStack Resource Controller) dependency, which exposes OpenStack resources as reconciled Kubernetes objects. It has become a mandatory dependency of CAPO: if you don’t install it before, the controller crashes in a loop with no matches for kind "Image" in version "openstack.k-orc.cloud/v1alpha1". Hence the kubectl apply of ORC just before the clusterctl init.

Deploying the child cluster#

Having a management cluster is one thing, but it’s not the one that will be autoscaled; we need the child cluster that will be deployed entirely via CAPO (and not HCL, it’s much more comfortable 😎).

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackCluster
metadata:
  name: capi-ovh
spec:
  identityRef:
    cloudName: ovh
    name: capi-ovh-cloud-config
  apiServerLoadBalancer:
    enabled: true
  externalNetwork:
    id: 6d041167-5863-4cad-a165-d352bb6720ab   # Ext-Net EU-WEST-PAR
  managedSecurityGroups: {}
  managedSubnets:
  - cidr: 10.6.0.0/24
    dnsNameservers:
    - 1.1.1.1

---
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: capi-ovh
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.0.0/16"]
    services:
      cidrBlocks: ["10.96.0.0/12"]
  controlPlaneRef:
    apiGroup: controlplane.cluster.x-k8s.io
    kind: KubeadmControlPlane
    name: capi-ovh-cp
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: OpenStackCluster
    name: capi-ovh
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackMachineTemplate
metadata:
  name: capi-ovh-cp
spec:
  template:
    spec:
      flavor: c3-4
      image:
        filter:
          name: ubuntu-2404-noble
      identityRef:
        cloudName: ovh
        name: capi-ovh-cloud-config
      configDrive: true
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlane
metadata:
  name: capi-ovh-cp
spec:
  version: v1.36.1
  replicas: 3
  machineTemplate:
    spec:
      infrastructureRef:
        apiGroup: infrastructure.cluster.x-k8s.io
        kind: OpenStackMachineTemplate
        name: capi-ovh-cp
  kubeadmConfigSpec:
    clusterConfiguration:
      controllerManager:
        extraArgs:
          - { name: cloud-provider, value: external }
    initConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          - { name: cloud-provider, value: external }
        name: '{{ local_hostname }}'
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackMachineTemplate
metadata:
  name: capi-ovh-md-0
spec:
  template:
    spec:
      flavor: c3-4
      image:
        filter:
          name: ubuntu-2404-noble
      identityRef:
        cloudName: ovh
        name: capi-ovh-cloud-config
      configDrive: true
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
metadata:
  name: capi-ovh-md-0
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            - { name: cloud-provider, value: external }
          name: '{{ local_hostname }}'
---
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  name: capi-ovh-md-0
spec:
  clusterName: capi-ovh
  replicas: 1
  selector:
    matchLabels: null
  template:
    spec:
      clusterName: capi-ovh
      version: v1.36.1
      bootstrap:
        configRef:
          apiGroup: bootstrap.cluster.x-k8s.io
          kind: KubeadmConfigTemplate
          name: capi-ovh-md-0
      infrastructureRef:
        apiGroup: infrastructure.cluster.x-k8s.io
        kind: OpenStackMachineTemplate
        name: capi-ovh-md-0

Regarding the child cluster’s nodes, I’m going to use the official Ubuntu cloud image. Staying on a bare image is simpler to reproduce and test; on the other hand it’s a bit slower at startup than the official images since we have to install a lot of dependencies. In a real production context, I would have gone through the official image-builder.

and sorry to those who wanted to see some Talos, it complicated the demo too much, I stuck with a classic Ubuntu for the PoC.

Before applying this manifest, we still need to give CAPO the credentials, which is exactly what the manifest’s identityRef points to (name: capi-ovh-cloud-config). The clouds.yaml is retrieved from the Horizon interface (directly on OVH’s OpenStack).

kubectl -n capi-ovh create secret generic capi-ovh-cloud-config \
  --from-file=clouds.yaml=clouds.yaml

A few minutes later, we have a control-plane (3 nodes) and a worker (via the capi-ovh-md-0 MachineDeployment) running on OVH:

$ kubectl --kubeconfig capi-ovh.kubeconfig get nodes -o wide
NAME                        STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP
capi-ovh-cp-jz4h9           Ready    control-plane   4m14s   v1.36.1   10.6.0.214    <none>
capi-ovh-cp-p8xq2           Ready    control-plane   3m52s   v1.36.1   10.6.0.61     <none>
capi-ovh-cp-t7v6d           Ready    control-plane   3m39s   v1.36.1   10.6.0.128    <none>
capi-ovh-md-0-7wkzs-lws49   Ready    <none>          2m48s   v1.36.1   10.6.0.180    <none>

For the nodes to become Ready, we had to install the usual addons: a CNI (Cilium, deployed via Helm) and above all the OpenStack Cloud Controller Manager, which takes care of reconciling the nodes’ providerID (the same spec.providerID story as in the previous article).

Installing Karpenter (cluster-api version)#

Now that we have our sandbox, we can graft Karpenter onto CAPI.

On the workload cluster side, we lay down the two CRDs seen above: the NodePool (the constraints) and the ClusterAPINodeClass (the “how”, almost empty here since CAPI handles most of the work). We deliberately stay permissive on the requirements (just amd64 + on-demand), we cap at 20 vCPU with limits, and we configure WhenEmpty consolidation (Karpenter will only delete empty nodes):

apiVersion: karpenter.cluster.x-k8s.io/v1alpha1
kind: ClusterAPINodeClass
metadata:
  name: default
spec: {}
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"] # either spot or on-demand. OVH doesn't offer spot, so on-demand in our case.
      nodeClassRef:
        group: karpenter.cluster.x-k8s.io
        kind: ClusterAPINodeClass
        name: default
      expireAfter: Never
  limits:
    cpu: "20"
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s

On the management side, Karpenter needs a MachineDeployment it can drive, with a little subtlety. The label node.cluster.x-k8s.io/karpenter-member: "" designates this MachineDeployment as “drivable” by Karpenter: it’s the one the provider is allowed to scale (and not the capi-ovh-md-0 we used to deploy our current worker).

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  name: capi-ovh-karpenter
  labels:
    node.cluster.x-k8s.io/karpenter-member: "" 
  annotations:
    capacity.cluster-autoscaler.kubernetes.io/cpu: "2"
    capacity.cluster-autoscaler.kubernetes.io/memory: "4G"
    capacity.cluster-autoscaler.kubernetes.io/maxPods: "110"
    capacity.cluster-autoscaler.kubernetes.io/labels: "kubernetes.io/arch=amd64,topology.kubernetes.io/zone=nova,node.kubernetes.io/instance-type=c3-4,karpenter.sh/capacity-type=on-demand"
spec:
  clusterName: capi-ovh
  replicas: 0
  # ...

The controller still needs to be deployed. It’s a single binary that talks to two clusters: Karpenter’s core watches the NodePool/NodeClaim/pods (on the workload side), while the provider scales the MachineDeployment (on the management side, MKS). So we need a kubeconfig that links the two.

The provider’s official docs recommend running Karpenter in the workload cluster: it then uses the in-cluster config (like standard operators) to see its cluster’s resources, and can also reach the management cluster via the --cluster-api-kubeconfig flag (managed by the chart’s values).

But, bad surprise: the provider has no container image published in a pullable way (the chart points to gcr.io/k8s-staging-... but the registry returns a 401). So we build our own. The repo provides a multi-arch Dockerfile, a docker buildx is enough:

git clone https://github.com/kubernetes-sigs/karpenter-provider-cluster-api
cd karpenter-provider-cluster-api
docker buildx build --platform linux/amd64 \
  -t ghcr.io/qjoly/karpenter-clusterapi-controller:poc --push .

needless to say you should not use this image, it is not maintained and you have no security guarantee whatsoever.

Once we have a valid image, we then install the charts/karpenter chart. In my case, I mounted the mgmt-kubeconfig secret containing the management cluster’s kubeconfig.

# kubeconfig of the MANAGEMENT cluster (MKS), mounted in the workload
kubectl -n karpenter create secret generic mgmt-kubeconfig --from-file=mgmt.kubeconfig=mks.kubeconfig

helm upgrade --install karpenter ./charts/karpenter -n karpenter --skip-crds \
  --set image.repo=ghcr.io/qjoly/karpenter-clusterapi-controller --set image.tag=poc \
  --set env.clusterAPIKubeconfig=/etc/mgmt/mgmt.kubeconfig \
  --set 'volumes[0].name=mgmt-kubeconfig,volumes[0].secret.secretName=mgmt-kubeconfig' \
  --set 'volumeMounts[0].name=mgmt-kubeconfig,volumeMounts[0].mountPath=/etc/mgmt'

And in production, this management kubeconfig must absolutely not be admin: it’s an access from the workload to the hub (we’ll come back to it in the security part), to be reduced to the strict minimum via RBAC (Machine/MachineDeployment of the relevant namespace).

kubectl -n karpenter set env deployment/karpenter \
  SYSTEM_NAMESPACE=karpenter DISABLE_WEBHOOK=true

Just to make it clear, here’s what runs and where:

How we test the scale-up#

To prove that scale-up works, we just need to deploy something that will stay pending because the current nodes are not available, for example with an AntiAffinity (normally we scale due to lack of resources, but nothing stops us from excluding the current nodes to simulate the lack of resources).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 2
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: inflate
              topologyKey: kubernetes.io/hostname
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.10
          resources:
            requests:
              cpu: "500m"
              memory: "256Mi"

First attempt: it starts, then it disappears#

I drop this deployment, and at first everything seems to work: Karpenter detects the Pending pod, scales the capi-ovh-karpenter MachineDeployment from 0 to 1, CAPO provisions an OVH instance… which boots fine and joins the cluster. Except the pod stays hopelessly in Pending, and after a while the freshly created node disappears.

Looking closer, the corresponding NodeClaim stayed stuck in Registered=False. My node was booting fine, joining the cluster fine… but Karpenter refused to consider it “its own”. After a while, it ends up seeing this as a deployment failure and deletes the machine.

Back to debugging, especially since the logs aren’t very talkative by default.

The expected taint#

Reading the code of Karpenter’s registration reconciler (the nodeclaim/lifecycle controller in kubernetes-sigs/karpenter), it becomes clearer: Karpenter requires every new node to arrive with a karpenter.sh/unregistered taint. To distinguish a node it provisioned itself from manual scaling. Once it recognizes the machine it ordered, it removes the taint (and otherwise, it waits until timeout).

To fix this, we put this taint directly in the KubeadmConfigTemplate that Karpenter will use:

apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
metadata:
  name: capi-ovh-karpenter
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          # ...
          taints:
            - key: karpenter.sh/unregistered
              effect: NoExecute

We relaunch the inflate deployment and cross our fingers. 🤞

Second attempt: the complete scale-up#

With the taint in place, we replay the inflate deployment and this time it works: Karpenter detects the Pending pod, scales the capi-ovh-karpenter MachineDeployment from 0 to 1, CAPO provisions an OVH instance, the node registers, Karpenter removes the taint, and the pod schedules onto it. On this run (bare Ubuntu image, c3-4), we can observe the steps via the NodeClaim’s status:

• Launched: the provider triggered the creation of the machine;
• Registered: the Kubernetes Node appeared and was “adopted” by Karpenter. This is where the famous karpenter.sh/unregistered taint comes in: Karpenter validates it, then removes it itself. This is precisely the step that failed on the first attempt;
• Initialized: the node is fully ready (Ready, startup taints removed). Pods can finally land on it.

On the adoption speed side, here’s what it looks like:

T+0s     kubectl apply -f inflate.yaml
T+2s     Karpenter creates the NodeClaim default-mrnnj (instance-type c3-4)
T+66s    CAPO scaled the MD to 1, OpenStack boots the VM
T+74s    Node Registered, taint karpenter.sh/unregistered removed
T+141s   Node Initialized (Ready=True), finally schedulable
T+153s   Both inflate pods are Running

It’s also worth remembering that the node becomes Registered at T+74s but is only really Initialized (thus schedulable) at T+141s. Those ~67 seconds are the bare Ubuntu image installing containerd, kubelet and the Kubernetes packages after registration. We wouldn’t have had this delay with a pre-packaged image in production, but it’s a PoC, so we stick with bare.

And scale-to-zero?#

An autoscaler that only knows how to grow isn’t much use apart from emptying the wallet. I delete the inflate deployment and watch Karpenter do the cleanup:

$ kubectl delete deploy inflate
# in the controller logs:
"disrupting node(s)" reason="empty" decision="delete" \
  disrupted-node-count=1 replacement-node-count=0 pod-count=0 \
  disrupted-nodes=[{"Node":"capi-ovh-karpenter-w6gt5-286tq", ...}]
"tainted node" taint.Key="karpenter.sh/disrupted"

WhenEmpty consolidation does its job well: reason=empty, decision=delete, zero replacement. About 3 minutes after removing the load, all NodeClaims have disappeared and the MachineDeployment is back to replicas: 0. Scale-to-zero is therefore demonstrable.

A big difference between Karpenter and CAPI: Karpenter reasons “delete this specific node”, whereas the cluster-api provider materializes this by decrementing the number of replicas of the MachineDeployment, and it’s then CAPI that chooses which Machine to remove (and sometimes, it’s the machine carrying an essential pod, which implies costs to move pods during the “Drain” step).

In short, Karpenter is a good product that fully handles scale-up and scale-down.. but we’re not going to stop there!

Karpenter vs cluster-autoscaler: the benchmark#

While I had the infra on hand, I wanted to pit Karpenter against its great rival: the cluster-autoscaler. It too knows how to drive CAPI MachineDeployments, it just moves the number of replicas, whereas Karpenter reasons in terms of NodeClaims.

The protocol that will test them, identical for both:

• three flavors offered to the autoscaler: c3-4 (2 vCPU), c3-8 (4 vCPU) and c3-16 (8 vCPU);
• an inflate load of 8 pods × 1 vCPU, dropped all at once on a cold pool (the base worker is cordoned, the control-plane is tainted): the 8 pods are Pending, the autoscaler must provision everything from scratch;
• we measure the time until the 8 pods are Running, the number of nodes created, and the flavors chosen.

The cluster-autoscaler, I deployed it directly in the management cluster (knowing that both modes are possible).

The results (one run per autoscaler, bare image):

Surprise compared to what I expected: no clear winner on time. Both reach 8 pods Running in ~135 s, because the limiting factor is neither Karpenter nor the CA, it’s the boot of the bare Ubuntu image (~2 min to install containerd + kubelet). Karpenter’s NodeClaim → Machine handshake isn’t that penalizing after all.

More interesting: Karpenter provisioned less than the cluster-autoscaler (2 nodes / 10 vCPU versus 3 / 12). Its bin-packing crammed 7 pods on a c3-16 + 1 on a c3-4, where the CA in least-waste preferred 3× c3-8. But beware of the trap: it’s not that Karpenter “optimizes better”. It took c3-16 a bit by accident 😅, we’re going to dig into how it chooses its flavor.

The cluster-api provider for Karpenter in this bit of code hard-codes Price: 0.0 for all instance types:

offerings := cloudprovider.Offerings{
    cloudprovider.Offering{ Requirements: ..., Price: 0.0, Available: true },
}

Result: it has no idea of the flavors’ prices, so no way to distinguish them by cost. So how does it decide between two flavors that both fit? Let’s run a new test to understand better!

The demonstration: three sizes, the wrong choice#

Second test, more targeted this time. To make it concrete, I still have the three eligible MachineDeployments (c3-4 (2 vCPU), c3-8 (4 vCPU) and c3-16 (8 vCPU)), then I drop a single pod requesting 3 vCPU: it doesn’t fit on c3-4, the optimal choice is c3-8, and c3-16 is overkill.

On the Karpenter side, the NodeClaim correctly computes the compatible types.. then the provider chooses one:

created nodeclaim   NodePool="default" instance-types="c3-16, c3-8"  requests cpu=3100m
launched nodeclaim  instance-type="c3-16"  allocatable cpu=8

It took c3-16 (8 vCPU for a need of 3). Why the biggest one? Because in the absence of a price to distinguish them, the provider sorts the compatible types by name and takes the first ("c3-16" < "c3-8" in lexicographic order). The code comment is quite clear: “if multiple instance types are found to be compatible we need to select one. for now, we sort by resource name and take the first”.

On the cluster-autoscaler side (--expander=least-waste), the same pod, the same choice of sizes:

Expanding Node Group .../capi-ovh-karpenter-c3-16 would waste 62.50% CPU ...
Expanding Node Group .../capi-ovh-karpenter-c3-8  would waste 25.00% CPU ...
Best option to resize: .../capi-ovh-karpenter-c3-8
Final scale-up plan: [capi-ovh-karpenter-c3-8 0->1]

It explicitly computes the perfect size and chooses c3-8. Same load, same catalog of flavors: cluster-autoscaler provisions 4 vCPU, Karpenter provisions 8. I think we have found a winner.

Conclusion#

Being able to autoscale nodes on Kubernetes remains a fairly complex need; the coordination between Kubernetes resources and cloud resources brings plenty of subtleties and constraints, not even mentioning scale-down which adds its own.

I had the chance to see Karpenter and its official provider in action during the live with Rémi Verchère about GPUs in Kubernetes and it was very impressive. In a CAPI context, the conclusion is more nuanced than I imagined: in practice, the two CAPI autoscalers are neck and neck (equivalent scale-up times).

For now, my opinion is that Karpenter is a very promising project, but as soon as it comes to going through the CAPI provider, it’s less efficient than Cluster-Autoscaler. It lacks the notion of pricing, and the workload talking to management direction doesn’t sit well with me, even though Karpenter does seem faster at creating nodes. The day these points are settled, Karpenter will finally be able to play its true role on CAPI. We’ll talk about it again :D.

A huge thanks to OVH for letting me test all this on their platform 🫶 It’s thanks to their help that I was able to write this article in good conditions (also, there’s a €200 voucher on the “Public Cloud” part, enough to test the CAPO provider in depth).

Enjoy your coffee ☕ !