The LeaderWorkerSet and DisaggregatedSet APIs
A Kubernetes API for deploying a group of pods as one replication unit, aimed at multi-host AI and ML inference workloads that shard a large model across multiple nodes.
LeaderWorkerSet (LWS) is a Kubernetes API for deploying a tightly managed group of pods, a single leader plus a set of workers, as one unit of replication, addressing deployment patterns common to multi-host inference where a large language model is sharded across multiple devices and nodes. Each pod in a group gets a unique index, pods in a group are created in parallel with optional gang scheduling, and a group can use separate templates for its leader and workers.
Multiple replicas of a group can be created with a scale subresource for HPA-driven autoscaling, rolling updates happen at the group level so all pods in a group upgrade together, and pods in the same group can be placed in the same topology. A companion DisaggregatedSet API builds on LWS to add autoscaling, rollouts and failure handling for more advanced disaggregated multi-node inference deployments, co-designed with the llm-d project.