Kubernetes v1.36: Smarter Scaling with Server-Side Sharded Watches
As Kubernetes clusters expand to thousands of nodes, controllers that monitor high-cardinality resources like Pods encounter a scalability bottleneck. Each replica of a horizontally scaled controller receives the entire event stream from the API server, incurring significant CPU, memory, and network costs—only to discard the objects it doesn't manage. Scaling out the controller doesn't ease per-replica load; it compounds the problem. To address this, Kubernetes v1.36 introduces server-side sharded list and watch as an alpha feature, empowering controllers to receive only the slice of resources they own.
What is the scaling problem that server-side sharded list and watch solves?
In large Kubernetes environments, controllers often watch many objects, such as Pods or Nodes. When these controllers are scaled horizontally—for example, in a deployment with multiple replicas—each replica must process the full stream of events from the API server. This means every replica deserializes and examines every object, even those it is not responsible for. The overhead includes CPU time for parsing, memory for buffering, and network bandwidth for data transfer. As more replicas are added, the total load on the system multiplies rather than being distributed. This inefficiency limits the ability to scale controllers beyond a certain cluster size, leading to degraded performance and increased operational costs. Server-side sharded list and watch solves this by having the API server filter events at the source, so each replica only receives the objects it needs.
How does client-side sharding fall short?
Some controllers, such as kube-state-metrics, already support client-side sharding. In this approach, each replica is assigned a portion of the key space and discards objects that don't belong to it. While this works functionally, it doesn't reduce the data volume flowing from the API server. Every replica still receives the complete event stream, leading to wasted resources: CPU cycles spent on deserializing irrelevant objects, memory consumed by discarded data, and network bandwidth that scales linearly with the number of replicas. Essentially, the cost per replica remains constant, and scaling out increases the aggregate load. Server-side sharding moves the filtering logic into the API server, eliminating these inefficiencies by sending only the relevant events to each replica.
What is new in Kubernetes v1.36 for sharding?
Kubernetes v1.36 introduces server-side sharded list and watch as an alpha feature, as defined in KEP-5866. This feature adds a shardSelector field to ListOptions, allowing clients to specify a hash range. The API server computes a deterministic 64-bit FNV-1a hash of a specified field (such as the object's UID or namespace) and returns only objects whose hash falls within the requested range. This applies to both initial list responses and subsequent watch events. The feature is designed to seamlessly integrate with existing controllers and informers, enabling efficient sharding without code rewrites. By filtering events on the server side, it dramatically reduces the data sent to each controller replica.
How does the shardSelector and hash range work?
The shardSelector is a string in the format shardRange(field, start, end), where field is the metadata path (e.g., object.metadata.uid) and start/end are hexadecimal strings defining the hash range (inclusive start, exclusive end). The API server computes an FNV-1a 64-bit hash of the specified field value and checks if the hash lies within [start, end). Objects outside the range are excluded from both list responses and watch streams. This computation is deterministic and consistent across all API server instances, making the feature safe for multi-master setups. For example, to assign a replica the lower half of the UID space, you would specify shardRange(object.metadata.uid, '0x0000000000000000', '0x8000000000000000').
How can controllers implement server-side sharded watches?
Controllers typically use informers to list and watch resources. To implement server-side sharding, each replica injects the shardSelector into the ListOptions via the WithTweakListOptions function when creating the shared informer factory. For instance, a Go controller would provide a callback that sets opts.ShardSelector to the appropriate hash range string. For a 2-replica deployment, replica 0 might use the lower half and replica 1 the upper half. The key is to ensure each replica uses a non-overlapping range that covers the entire hash space collectively. This approach requires no changes to the informer logic itself, only the configuration of list options at creation time. The provided code snippet in the original text demonstrates this pattern.
What are the benefits of server-side sharding?
Server-side sharding offers several advantages over client-side approaches. It reduces network bandwidth by ensuring only relevant data is sent from the API server to each controller replica. It lowers CPU and memory usage because replicas no longer waste resources deserializing and discarding unnecessary objects. The total system load scales with the number of shards, not with the number of replicas, enabling true horizontal scaling. Additionally, since the filtering is deterministic and consistent across API server instances, the feature works seamlessly in clusters with multiple API servers. This improved efficiency allows controllers to handle larger clusters without performance degradation, ultimately reducing operational costs and simplifying cluster management.