Kubernetes Worker Node Pools

Reference for KubernetesNodes, the resource that adds a pool of worker nodes to a managed Kubernetes cluster.

kubernetes-nodes manages a single worker node pool for a managed Kubernetes cluster. It is the worker half of the Kubernetes-app split (see cozystack/community#8): the control plane lives in the kubernetes chart, and each pool is a separate kubernetes-nodes release attached to a parent cluster by name.

A KubernetesNodes CR named <cluster>-<pool> (e.g. myk8s-md0) provisions a MachineDeployment + KubevirtMachineTemplate + MachineHealthCheck wired to the parent cluster kubernetes-<cluster> in the same namespace, plus a talos-reconcile Job that produces the pool’s TalosConfigTemplate.

Parameters

Linkage

NameDescriptionTypeValue
clusterName of the parent Kubernetes cluster (the Kubernetes CR) in the same namespace this node pool attaches to. Required. The pool’s CAPI objects are wired to the parent cluster kubernetes-<cluster>; the KubernetesNodes CR must be named <cluster>-<pool>. The pool name (the part after <cluster>-) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default md0), or the render fails on ownership conflict.string""

Common Parameters

NameDescriptionTypeValue
storageClassStorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass’s CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an osImage.builtin pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden’s. Leaving it empty resolves to the cluster’s default StorageClass, and the same comparison is made against that. On a builtin pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.stringreplicated

Pool Sizing

NameDescriptionTypeValue
minReplicasMinimum number of replicas in this pool. Used as the cluster-autoscaler floor.int0
maxReplicasMaximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.int10
instanceTypeVirtual machine instance type.stringu1.medium
diskSizeSystem disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool’s osImage: the default and osImage.factory stream the Image Factory raw artifact in over HTTP, while osImage.builtin makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A builtin pool must keep diskSize at or above the golden’s storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source – the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a builtin pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden’s size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus diskSize + golden storage for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer’s own events.quantity20Gi
rolesList of node roles. Each role r labels nodes with node-role.kubernetes.io/<r>. Use [ingress-nginx] for a pool that hosts the tenant ingress-nginx controller.[]string[]
resourcesExplicit CPU and memory for each worker node, as an alternative to instanceType sizing. Optional: when omitted, the node is sized by instanceType. When both cpu and memory are set, they take precedence and instanceType is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype’s CPU/memory). Set both cpu and memory together or neither; setting only one is rejected at render time.object{}
resources.cpuNumber of vCPU cores for each worker VM, as a whole number such as 2. On the kubevirt substrate the value becomes the VM’s integer domain.cpu.cores, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted 0 is treated as unset. On the proxmox substrate a millicore or fractional quantity is rounded up to whole cores.quantity""
resources.memoryMemory (RAM) available.quantity""
podCpuLimitCPU limit for the worker VM’s virt-launcher compute container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from resources.cpu, which sizes the guest itself. Cannot be combined with instanceType sizing: KubeVirt refuses a VM that references an instancetype and also sets domain.resources.limits.cpu, so the chart rejects that combination at render time. Sizing the pool by resources (both cpu and memory) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on Insufficient cpu. Pair it with podCpuRequest to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with podCpuRequest: 0, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker’s Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.quantity""
podCpuRequestCPU request for the worker VM’s virt-launcher compute container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from resources.cpu, which sizes the guest itself. It exists to be paired with podCpuLimit: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with instanceType sizing, for the same reason as podCpuLimit: KubeVirt refuses a VM that references an instancetype and also sets domain.resources.requests.cpu, so the chart rejects that combination at render time, as it does a value above podCpuLimit. Usable without a limit, where it only changes the scheduler’s weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker’s Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a podCpuLimit is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.quantity""
gpusList of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).[]object[]
gpus[i].nameName of GPU, such as “nvidia.com/AD102GL_L40S”.string""
kernelModulesKernel modules loaded on every worker in this pool, emitted as Talos machine.kernel.modules. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one nvidia.com/* GPU gets nvidia (with NVreg_NvLinkDisable=1, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), nvidia_uvm, nvidia_drm, nvidia_modeset (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart’s choice entirely, or [] to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as []: a generated parameter table that shows [] in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing [] disables it. The module still has to be in the image, and which extensions the image carries is the pool’s schematic: a GPU pool points osImage.factory.schematicID (or osImage.builtin.schematicID) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the nvidia.com/ resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an nvidia.com/* GPU and leaves the field unset: that pool rolls.[]object[]
kernelModules[i].nameModule name as modprobe takes it, e.g. nvidia_uvm.string""
kernelModules[i].parametersModule parameters, each as a bare key=value string.[]string[]
kubeletKubelet resource reservations for this pool.object{}
kubelet.systemReservedMemoryMemory reserved for host OS. Auto-computed from instanceType if empty.string""
kubelet.kubeReservedMemoryMemory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.string""
kubelet.systemReservedCpuCPU reserved for host OS. Auto-computed from instanceType if empty.string""
kubelet.kubeReservedCpuCPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.string""
kubelet.evictionHardMemoryHard eviction threshold for memory (absolute like 200Mi or percentage like 7%).string7%
kubelet.evictionSoftMemorySoft eviction threshold for memory (absolute like 1Gi or percentage like 10%).string10%
logSerialConsoleStream each worker VM’s guest serial console into a guest-console-log container beside virt-launcher, readable with kubectl logs. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content – including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform’s cluster-wide disableSerialConsoleLog, set because that container has been seen holding virt-launcher in PodInitializing (kubevirt/kubevirt#15989), which would take the pool down rather than explain it – so check that virt-launcher Pods reach Running after enabling it.boolfalse
maxUnhealthyMaximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer (“0”, “1”, …) or a percentage (“0%”, “50%”); bare numeric strings are rejected, so the safer default is a percentage. Drop to “0%” once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At “50%” a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.string50%
nodeStartupTimeoutMaximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine’s creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.string20m
readyUnknownTimeoutHow long a worker Node may report Ready=Unknown before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. Ready=Unknown is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.string5m
readyFalseTimeoutHow long a worker Node may report Ready=False before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports Ready=False, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller’s taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.string10m
maxNodeProvisionTimeHow long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool’s MachineDeployment as the cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime annotation, which overrides the autoscaler’s built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool’s upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (“30m”, “1h30m”), which is narrower than Go’s duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (“20m0s”, “1h0m0s”) is accepted. Keep in sync with the parent kubernetes chart.string30m

Kubernetes Version

NameDescriptionTypeValue
versionKubernetes major.minor version the pool joins. Must not be ahead of the parent cluster’s minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade – bump the parent Kubernetes CR first, then each pool – and must satisfy the Talos<->Kubernetes support matrix against talos.version.stringv1.35

Talos Worker Image

NameDescriptionTypeValue
osImageWorker OS image source for this pool. Default (omitted): import over HTTP from the talos.* Image Factory. Set osImage.builtin to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool.object{}
osImage.builtinClone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes – the golden binds where CDI imported it, a worker disk where its VM is scheduled.*objectnull
osImage.builtin.schematicIDTalos schematic ID of the golden to clone. Defaults to talos.schematicID.string""
osImage.builtin.versionTalos version of the golden to clone. Defaults to talos.version.string""
osImage.factoryImport from an Image Factory / mirror (default).*objectnull
osImage.factory.versionTalos version. Defaults to talos.version.string""
osImage.factory.schematicIDTalos schematic ID. Defaults to talos.schematicID.string""
osImage.factory.imageFactoryURLBase URL of the Image Factory serving the boot raw image. Defaults to talos.imageFactoryURL. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from talos.installerRepository, so an air-gapped pool must also have talos.installerRepository pointed at a reachable registry.string""
talosTalos worker image configuration. Keep in sync with the parent cluster’s talos.object{}
talos.versionDefault Talos release for this pool’s worker OS image and in-guest installer. The pool can pin its own release via osImage.builtin.version or osImage.factory.version; when osImage omits it, this value applies. Both ways in are checked against the chart’s Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check talos.version would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator’s call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full vMAJOR.MINOR.PATCH release, optionally with a pre-release suffix – the render refuses anything else, a minor-only v1.13 included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool’s HelmRelease instead.stringv1.13.6
talos.schematicIDTalos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via osImage.builtin.schematicID or osImage.factory.schematicID; when osImage omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier – the render refuses anything else. A custom schematic on osImage.builtin also needs its own catalog entry, since the golden is named from the schematic ID.stringce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515
talos.imageFactoryURLDefault base URL of the Talos Image Factory that serves the worker OS disk image (the openstack-amd64.raw.xz raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and osImage.factory, which overrides it via osImage.factory.imageFactoryURL. A pool on osImage.builtin clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain http(s) URL and the render refuses anything else; a pool on osImage.builtin builds no such URL and is not held to it, so blanking this field there is legal.stringhttps://factory.talos.dev
talos.installerRepositoryOCI repository prefix for the Talos installer image used by the in-guest talos-reconcile upgrade Job. Resolved as <installerRepository>/<schematicID>:<version> against whatever this pool’s osImage pins, falling back to talos.schematicID and talos.version — so the installer matches the OS the pool actually booted. The prefix itself has no osImage.* override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory’s installer path. Override for air-gapped or mirrored registries. No trailing slash.stringfactory.talos.dev/installer
talos.registryMirrorsTalos machine.registries.mirrors passthrough for worker nodes: a map of upstream registry host to { endpoints: [ ... ] }. Empty by default, so workers pull container images (the Talos kubelet image included) directly from the upstream registry. Point a host such as ghcr.io at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker’s boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets skipFallback: true, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.object{}
talos.imageFactoryCAComplete PEM certificate block(s) CDI must trust when importing this pool’s worker OS disk image over HTTPS — the no-osImage default and osImage.factory path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (imageFactoryURL or osImage.factory.imageFactoryURL) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with x509: certificate signed by unknown authority. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume’s source.http.certConfigMap; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on osImage.builtin clones a golden image and builds no HTTP source, so this field has no effect there.string""

Substrate

NameDescriptionTypeValue
substrateWhich infrastructure provider backs this pool’s worker VMs. kubevirt runs them inside this cluster on KubeVirt, sized by instanceType and booted from a Talos disk image CDI streams from the image factory. proxmox runs them on an external Proxmox VE cluster through capmox, sized by resources, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment’s template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down.stringkubevirt
proxmoxProxmox substrate settings.object{}
proxmox.templateTagsTags identifying the Talos VM template to clone. capmox matches a template when its tag set is EQUAL to this list, not when it contains it, so every tag on the template must be listed here — a missing one yields found 0 VM templates with tags ... while the tags are plainly present. The template has to carry this pool’s talos.schematicID at talos.version in the nocloud flavour, with the Proxmox guest agent enabled for the qemu-guest-agent extension of the default schematic; hack/proxmox-talos-template.sh builds it from these values and prints the tags to list here. Do not build it from the platform release’s nocloud-amd64.raw.xz: that is the bare-metal node image, whose zfs extension keeps a VM from finishing its boot sequence and reboots it at the Talos boot timeout. Required when substrate is proxmox.[]string[]
proxmox.fullFull clone rather than linked. Default is a linked clone: it costs kilobytes at creation instead of the whole disk, because a ZFS-backed linked clone writes only its own increment (measured: 8K against 10.2G for the same worker full-cloned onto a sparse 0 pool). The cost is that the clone holds the template’s base snapshot, so the template cannot be rotated while any linked clone still references it — which is what the consolidation CronJob resolves 24h after creation. Set true for a pool that must be independent of the template from the first second.boolfalse
proxmox.networkNIC configuration.object{}
proxmox.network.bridgeProxmox bridge the VM NIC attaches to (e.g. vmbr0). Required when substrate is proxmox.string""
proxmox.network.vlanL2 VLAN tag. Omitted when unset.int0
proxmox.network.mtuNIC MTU. Omitted when unset.int0
proxmox.storageProxmox storage for the cloned disk; needs full: true. A target storage is a full-clone parameter — a linked clone always lives on the template’s storage, and Proxmox refuses the pair — so the render refuses it too. Empty keeps the template’s storage.string""
proxmox.poolProxmox resource pool the worker VMs are added to. Set the same pool on every node pool of a tenant cluster and grant the tenant’s tokens their privileges on /pool/<name>: that is the one ACL path that follows every VM capmox creates, replacements included. The pool must exist on the hypervisor; capmox does not create it. Empty adds the workers to no pool.string""
proxmox.dnsServersNameservers written into the worker machineconfig. On kubevirt the workers use the management cluster’s CoreDNS, which they reach over the pod network; an off-cluster Proxmox worker cannot, so it needs reachable resolvers of its own. Required when substrate is proxmox. Keep in sync with the parent kubernetes chart’s proxmox.dnsServers.[]string[]