As Kubernetes adoption has grown, the conversation has shifted beyond running containers to managing increasingly complex application lifecycles. Modern platforms support stateless web services, stateful databases, batch processing, AI workloads, and platform services. At the same time, they must remain reliable during upgrades, scaling events, and infrastructure failures.

Every Kubernetes user relies on SIG Apps, whether they realize it or not. Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs form the foundation of how applications are deployed, updated, scaled, and operated across the Kubernetes ecosystem.

SIG Apps is focused on improving workload resilience, refining application lifecycle management, and addressing the operational challenges that emerge when applications encounter node failures, rollout disruptions, and increasingly complex infrastructure environments.

In this spotlight, we sit down with two of the three SIG Apps chairs Janet Kuo and Maciej Szulik to discuss the evolution of Kubernetes workload management, the challenges of balancing application reliability with operational simplicity, and the future of application lifecycle management within one of Kubernetes’ most influential Special Interest Groups.

Introducing SIG Apps

Natalie Fisher: Can you introduce yourself, your role, and how you got involved in SIG Apps?

Janet Kuo: I'm a Senior Staff Software Engineer at Google and have been a Kubernetes maintainer since 2015, joining the community just as we were racing toward the 1.0 launch. In those early days, my focus was on building the core Workloads API, specifically developing controllers like Deployment, ReplicaSet, StatefulSet, and DaemonSet, defining their rollout behaviors, and bringing them from initial designs to GA. That hands-on work was my entry point into SIG Apps.

Since then, I've stayed deeply involved in both the technical and community sides of Kubernetes. I have led SIG Apps as Co-Chair and Tech Lead since 2019. Currently, in addition to maintaining the workloads API, I am driving new subprojects like the Agent Sandbox to ensure Kubernetes is ready for next-generation agentic and AI workloads.

Maciej Szulik: I started contributing to Kubernetes all the way back in 2014. Since then, I've worked across various areas of the project: controllers, kubectl, and apimachinery, which eventually led me to become one of the Chairs and Tech Leads for SIG Apps. My current focus is reliability of the workload controllers under the SIG Apps umbrella and stability and ease of use of kubectl as part of my SIG CLI Tech Lead role. I also care about overall community health and growth as part of my Steering Committee role. Outside of Kubernetes, I work as a Staff Platform Engineer at Defense Unicorns, where I'm helping make Kubernetes more airgap-native with a project called zarf.

The problem and the solution

SIG Apps is responsible for the core workload APIs that power how applications run on Kubernetes. From Deployments and StatefulSets to Jobs and CronJobs, these controllers determine how workloads are created, updated, scaled, and recovered when things go wrong.

As Kubernetes expands to support increasingly diverse workloads – including AI, batch processing, and large-scale distributed applications – SIG Apps continues to evolve these APIs while balancing reliability, backward compatibility, and operational simplicity.

NF: For readers who may not be familiar, what is SIG Apps, and what role does it play within the broader Kubernetes ecosystem?

MS: SIG Apps is the Kubernetes Special Interest Group responsible for the workloads APIs. CronJob and Job help running batch workloads, whereas DaemonSet, Deployment, ReplicaSet, and StatefulSet serve the majority of other applications. More broadly, SIG Apps owns the layer most developers actually touch day-to-day: the controllers that turn a workload specification into running, self-healing pods. It's the group deciding how Deployments roll out, how Jobs retry, how DaemonSets place a pod per node.

JK: Adding to what Maciej described, as the industry shifts, we are seeing a massive demand to run complex, non-traditional workloads like distributed AI training, batch computing, and dynamic agent environments. Our role is expanding: we aren't just maintaining the classic workloads API, but we are actively evolving it and establishing new patterns (like the Agent Sandbox) to make sure Kubernetes remains the best platform for the next generation of workloads, such as AI.

NF: Looking at the workload APIs owned by SIG Apps (Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs), which areas are receiving the most attention from maintainers and contributors?

MS: After a long stretch focused on making batch workloads run smoothly on Kubernetes, we’ve shifted attention to make sure serving workloads (DaemonSets, StatefulSets, etc) aren’t left behind. This means performance and high-scale improvements to rollout and scaling behavior, plus working through our backlog of user-reported issues, prioritizing the ones with the strongest support from the user base.

Current focus areas

As Kubernetes workloads grow in scale and complexity, the challenges facing workload controllers evolve as well. We asked the SIG Apps chairs where contributors are focusing their efforts today and which resilience problems they believe are the highest priorities.

NF: From your perspective, what are the most important workload resilience problems SIG Apps is trying to solve today?

MS: Node lifecycle challenges have come up repeatedly across SIG Apps, SIG Node, and SIG Autoscaling discussions. DaemonSets and Jobs are just where the pain is most visible, since they're the workloads most directly bound to node state. Rather than solve it piecemeal within one SIG, we've settled on spinning up a dedicated Node Lifecycle Working Group to focus on this properly and hopefully land long-term solutions instead of one-off patches.

JK: From an AI perspective, resilience is critical. When you are running a massive distributed LLM training job that spans hundreds of GPUs, a single node failure can halt the entire pipeline. Similarly, if a DaemonSet that runs your logging or GPU monitoring agent gets stuck on a bad node, it impacts the entire cluster's health.

In addition to the work in the Node Lifecycle WG to handle infrastructure-level degradation, SIG Apps is addressing this at the orchestration layer through subprojects like JobSet (for distributed training) and LeaderWorkerSet (LWS) (for sharded LLM inference). These APIs introduce patterns like "all-or-nothing" failure handling, where a single pod or job failure triggers a coordinated group-level restart to resume from the last clean checkpoint, rather than letting stuck workloads hang in an inconsistent state.

Real-world impact

The work happening within SIG Apps extends far beyond controller implementations and API design. We wanted to understand what these improvements mean in practice for platform teams operating Kubernetes clusters in production.

NF: For platform teams operating Kubernetes in production, what practical improvements would they notice if the node lifecycle and workload resilience work currently under discussion is successfully delivered?

MS: I’m mostly looking from the sidelines, the folks actually in the Node Lifecycle Working Group would give you a sharper answer. But from where I sit, I’m hoping their work translates into fewer 3am pages that turn out to be “a DaemonSet rollout got stuck because node X was flaky, and someone had to manually cordon/delete/restart to unstick it.”

JK: +1 to what Maciej said, and beyond reducing manual intervention, platform teams will also see much better resource predictability and cost efficiency. For example, in AI workloads where GPU idle time is extremely expensive, having Kubernetes automatically detect a degraded node and reschedule the training coordinator or agent before the job crashes means less wasted compute and more stable job execution.

Challenges and trade-offs

Evolving APIs that millions of workloads rely on requires careful engineering and even more careful decision-making. We asked the SIG Apps chairs about the technical and operational trade-offs they weigh when introducing changes to Kubernetes’ core workload controllers.

NF: What are some of the hardest technical or operational trade-offs SIG Apps encounters when evolving core workload controllers?

MS: Honestly, a few tensions keep coming up: how aggressively a controller should give up on stuck pods, and what signals it actually needs to make that call correctly. At the same time, we always have to think about backward compatibility. Deployment, DaemonSet, and Job behavior has been depended on for a decade [by Kubernetes users, tooling, automation, and higher-level controllers], so even a change that’s clearly “more correct” can break automation people built around the old behavior without meaning to.

JK: One of our hardest trade-offs is resisting the urge to make "elegant" design changes that break backward compatibility. Instead, we have to design opt-in features that let users adopt new behaviors without forcing them on legacy workloads. When we need to support completely new paradigms, we prefer introducing them as CRDs first rather than bloating the core APIs, like we are doing with Agent Sandbox, JobSet, and LWS.

Looking ahead

While much of SIG Apps’ work focuses on maintaining the stability of existing workload APIs, the group is also shaping the future of Kubernetes through new enhancements and proposals. We concluded by asking about one proposal that recently returned to active development and what it represents for the future of workload management.

NF: The SIG recently discussed reviving KEP-4443 with a target release of Kubernetes 1.38. What opportunities or challenges does this proposal aim to address, and why is now the right time to revisit it?

KEP-4443 addresses a small but real gap in the Job API: a PodFailurePolicy can be configured to add a condition reason to the JobFailed condition, but different pod failure policy rules targeting different container exit codes all produce that same generic reason. The proposal is simple: an optional Name field on each PodFailurePolicyRule, which gets appended to the JobFailed condition reason, so higher-level tools like JobSet can finally react differently depending on which rule triggered the failure.

As for timing, the answer is as simple as it always is in open source: we lost the original contributor who was driving this. Now we’ve got someone new interested in picking it up, that’s why we’re targeting the next release.

Getting Involved

NF: For someone interested in contributing to SIG Apps, where would you recommend they start, especially if they are not yet a Kubernetes maintainer?

MS: The best place to start is the #sig-apps slack channel and our regular SIG Apps meetings. We’ve all started there, and if it feels intimidating, or nobody replies right away, that’s completely normal. Everyone's busy. It's not personal.

JK: In addition to what Maciej answered, I'd suggest looking at our newer subprojects and initiatives. Contributing to stable APIs like Deployment or StatefulSet can be daunting because the barrier for making changes is very high due to backward compatibility, and there is much less low-hanging fruit.

If you are new to the community, projects like the Agent Sandbox are fantastic entry points. They are actively evolving, have a friendly group of maintainers, and offer plenty of greenfield development opportunities where you can make a significant impact quickly.

Summary

SIG Apps has shaped how Kubernetes applications are deployed and operated since the project’s earliest days. While users often interact with Deployments, StatefulSets, Jobs, and DaemonSets without thinking about the controllers behind them, the work within SIG Apps continues to shape the reliability and scalability of workloads across the Kubernetes ecosystem.

From improving workload resilience and node lifecycle behavior to enabling new patterns for AI and distributed computing, the SIG is evolving Kubernetes while remaining committed to one of the project’s core principles: preserving the stability and backward compatibility that users depend on. Whether you’re interested in core workload APIs, emerging projects like Agent Sandbox, or helping improve the operational experience of Kubernetes users everywhere, SIG Apps offers many opportunities to get involved.

Kubernetes v1.37 promotes the PersistentVolumeClaimUnusedSinceTime feature gate to Beta (enabled by default). With this feature, the PersistentVolumeClaim (PVC) protection controller adds an Unused condition to each PVC, telling you whether any running pod currently references it — no custom tooling or cross-referencing required.

For the API definition of PVC conditions, see the PersistentVolumeClaim API reference. Read on to learn how the Unused condition works and how to use it.

Why track PVC usage?

In large-scale Kubernetes clusters, it is common for users to create PVCs and then delete the associated pods without cleaning up the storage, because Kubernetes does not automatically delete PVCs when their pods are removed (to protect against accidental data loss). Over time, these orphaned PVCs may accumulate, silently consuming storage capacity and driving up cloud costs.

Before Kubernetes v1.37, it was easy to identify an unused PersistentVolume, but much harder to determine whether a PVC was still being used. Doing so required cross-referencing pods, PersistentVolumes, and PVCs over a potentially large window of time. Administrators often resorted to custom monitoring pipelines or scripts to answer a seemingly simple question: "Is anything actually using this volume?"

The PersistentVolumeClaimUnusedSinceTime feature solves this by making the answer available natively in the PVC status. Once the feature is enabled, every PVC gets an Unused condition managed by the PVC protection controller.

User stories

• Storage administrator: "I want to know which PVCs in my cluster are not being used by any pod so I can safely identify orphaned volumes and schedule them for deletion."
• DevOps engineer: "I want to list PVCs that have the Unused condition set to True so I can automate cleanup in development environments."

How does it work?

The PVC protection controller — which already watches pods to enforce the storage object in use protection — now also manages a new Unused condition on PVCs.

The condition works as follows:

A few details worth noting:

• Terminated pods don't count: A pod that has completed (phase Succeeded or Failed) does not keep the PVC marked as in use. This means batch jobs with restartPolicy: Never won't prevent the PVC from becoming Unused=True after they finish.
• Pending pods do count: Even an unschedulable pod (for example, one with an impossible node selector) still counts as using the PVC. The intent to use the volume is enough.
• Multiple pods: If several pods reference the same PVC, the condition transitions to Unused=True only after the *last" non-terminated pod is removed or terminates.

Using lastTransitionTime to find when a PVC became idle

Like every Kubernetes condition, the Unused condition carries a standard lastTransitionTime field. This means you get a useful bonus for free: when the condition transitions from False to True, the lastTransitionTime records exactly when the PVC became idle. You can use this timestamp to answer questions like "how long has this PVC been sitting unused?" — for example, to find PVCs that have been idle for more than 30 days (see the example query below).

What changed from Alpha to Beta?

Kubernetes v1.36 introduced this feature as Alpha, where you had to enable the PersistentVolumeClaimUnusedSinceTime feature gate explicitly. For Beta in v1.37, the feature gate is enabled by default, and the feature has full end-to-end test coverage.

How to use it

Since the feature is Beta and enabled by default in Kubernetes v1.37, the Unused condition will appear on PVCs automatically. Here is a walkthrough to see it in action:

1. Create a PVC:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
    name: my-data
   spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 1Gi
   

2. After a short time, inspect the PVC conditions:

   kubectl get pvc my-data -o jsonpath='{.status.conditions[*]}' | jq .
   

   You should see an Unused condition with status True and reason NoPodsUsingPVC:

   {
    "lastProbeTime": null,
    "lastTransitionTime": "2026-09-14T12:03:11Z",
    "message": "No pods are currently referencing this PVC",
    "reason": "NoPodsUsingPVC",
    "status": "True",
    "type": "Unused"
   }
   

3. Create a pod that uses the PVC:

   apiVersion: v1
   kind: Pod
   metadata:
    name: my-app
   spec:
    containers:
    - name: app
    image: busybox
    command: ["sleep", "3600"]
    volumeMounts:
    - name: data
    mountPath: /data
    volumes:
    - name: data
    persistentVolumeClaim:
    claimName: my-data
   

4. Check the condition again — it should now show Unused=False:

   kubectl get pvc my-data -o jsonpath='{.status.conditions[?(@.type=="Unused")].status}'
   

   Output:

   False
   

5. Delete the pod and wait for the condition to transition back to Unused=True:

   kubectl delete pod my-app
   kubectl get pvc my-data -o jsonpath='{.status.conditions[?(@.type=="Unused")]}'
   

   The condition should show Unused=True with reason NoPodsUsingPVC again.

Finding unused PVCs across the cluster

To list all PVCs that have been unused for more than 30 days, you can use a command like:

kubectl get pvc -A -o json | jq -r '
 .items[]
 | select(.status.conditions[]? | select(.type=="Unused" and .status=="True"))
 | select(
 (.status.conditions[] | select(.type=="Unused") | .lastTransitionTime) as $t
 | (now - ($t | fromdateiso8601)) > (30 * 86400)
 )
 | "\(.metadata.namespace)/\(.metadata.name) unused since \(.status.conditions[] | select(.type=="Unused") | .lastTransitionTime)"
'

What's next?

Depending on feedback and adoption, the Kubernetes project intends to graduate this feature to General Availability (GA) in a future release. If you have feedback on this feature, please open an issue in the kubernetes/kubernetes repository.

To learn more about this enhancement, refer to KEP-5541: PersistentVolumeClaim last used time.

Getting involved

The Kubernetes project always welcomes new contributors. If you would like to get involved, you can join us at SIG Storage.

If you would like to share feedback, you can do so on our public Slack channel (visit https://slack.k8s.io/ for an invitation if you need one).

Special thanks to the contributors who helped design and implement this feature (alphabetical order):

• Arvind Parekh (ArvindParekh)
• Hemant Kumar (gnufied)
• Jan Šafránek (jsafrane)
• Kevin Hannon (kannon92)
• Roman Bednář (RomanBednar)

Kubernetes v1.37 brings important storage security features: emptyDir permission modes and bind mount options. They help application programmers and security professionals implement rigorous security policies, for example, prohibiting deletion of files across containers or execution of arbitrary binaries from writable volumes, directly in Kubernetes without any complicated circumvention.

Linux storage and permission fundamentals

Before diving into the new Kubernetes features, let us briefly review the low-level Linux security mechanisms that make them possible.

Bind mount flags

When Linux mounts or remounts a directory, Virtual File System (VFS) flags control what actions are permitted on that filesystem:

• noexec: Do not permit direct execution of any binaries on the mounted filesystem.
• nosuid: Do not allow set-user-identifier or set-group-identifier bits to take effect.
• nodev: Do not interpret character or block special devices on the file system.

Directory permissions and the sticky bit

Standard Unix permissions regulate access across three scopes: Owner, Group, and Others (e.g., 0755 or 0777).

Beyond standard read, write, and execute bits, Linux supports the sticky bit (as in mode 01777). When applied to a directory, the sticky bit ensures that a file inside that directory can only be deleted or renamed by the file's owner or root. This is essential for shared writable directories like /tmp.

Motivation for the improvements

Why does Kubernetes need bind mount options and emptyDir permissions?

The primary goal of these features is to increase the security of Kubernetes workloads by allowing security-related bind mount options on volume mounts. By default, volumes are bind-mounted into containers by the container runtime and kubelet without noexec, nosuid, or nodev flags. This default can undermine security. For example, with noexec missing, a compromised process can use any writable volume (emptyDir, PersistentVolume, etc.) to download, chmod +x, and execute arbitrary binaries even when the container has a read-only root filesystem (readOnlyRootFilesystem: true). Supporting noexec, nodev, and nosuid gives users a native way to harden volume mounts to match security benchmarks and policy.

The gap is most visible with emptyDir volumes, which are the most common writable volume type and have been the subject of multiple security findings:

• Issue #48912: Recognized security gap - the inability to set mount options on emptyDir was flagged in an audit but remained unresolved until now.
• Issue #119627: Kubernetes 1.24 Security Audit (Finding NCC-E003660-7HM) - external auditors specifically noted that the inability to mount emptyDir with noexec represents a security failure.

However, the same gap applies to all volume types. PersistentVolumes have a mountOptions field, but those options are filesystem-level flags applied by the CSI driver at the node, so they do not reliably translate into bind mount flags inside the container. Previously, there was no mechanism to set noexec, nosuid, or nodev on the bind mount that the container runtime creates for any volume type.

Additionally, the emptyDir volume type defaults to creating directories with a hardcoded mode of 0777. This previously meant that any process that can discover the volume could read, write, and delete anything in the volume, regardless of who created it.

You could - and still can - use an initial container to set a different access mode, but this is more complex, and hard to verify for compliance.

This causes real problems:

• Multi-container pods sharing an emptyDir could not prevent one container from deleting another's files. The sticky bit (01777) solves this, but there was no native way to set it.
• Some applications and security frameworks expect /tmp directories to have the sticky bit set (mode 01777). Without native support for setting the emptyDir mode, users had to use init containers or alternative volume types to meet this requirement.
• Platform engineers who want tighter permissions (e.g., 0750 for owner and group only) have to use init containers running chmod, which adds unnecessary complexity.

The emptyDir volume type was a notable gap. As one of the most common writable volume types in Kubernetes, it had no way to control its creation permissions.

Real-world use cases

Application developers, working closely with security engineers, are responsible for maintaining the security posture of their applications and ensuring workloads do not pose risks to the wider infrastructure. These features allow development teams to confidently address critical security scenarios:

Preventing Privilege Escalation on Writable Mounts: An application developer configuring temporary workspace volumes (like emptyDir or /tmp mounts) can ensure they are mounted with nosuid and noexec. This guarantees that even if the application is compromised and a malicious payload is downloaded, the workload cannot execute the payload or use it to escalate privileges on the node.

Securing Shared Scratch Space in Multi-Container Pods: A developer configuring CI/CD pipeline pods often needs multiple containers (e.g., a builder container and a sidecar logger) to share a workspace. By setting mode: 01777 on an emptyDir, the developer ensures the shared workspace behaves like a traditional Unix /tmp directory. Each container can write files independently, but a compromised process in one container cannot delete the build artifacts produced by another.

Enforcing Principle of Least Privilege for Application Data: An application developer deploying a database pod can lock down access to the database's temporary storage. By setting mode: 0750 on the emptyDir, the developer ensures that only the specific database user and group can read or write to the volume, explicitly denying access to any other processes or sidecars in the same pod.

Note: Both features are behind Alpha feature gates in Kubernetes v1.37. To use them, enable VolumeBindMountOptions and EmptyDirVolumeMode on the API server and kubelet.

Example 1: Enforcing bind mount options

This full Pod manifest mounts an emptyDir volume at /tmp with bindMountOptions: [noexec, nosuid].

apiVersion: v1
kind: Pod
metadata:
 name: hardened-bindmount-pod
 namespace: default
spec:
 os:
 name: linux
 containers:
 - name: hardened-app
 image: alpine:latest
 command: ["sleep", "3600"]
 securityContext:
 readOnlyRootFilesystem: true
 volumeMounts:
 - name: temp-storage
 mountPath: /tmp
 bindMountOptions:
 - noexec
 - nosuid
 volumes:
 - name: temp-storage
 emptyDir: {}

Example 2: emptyDir volume permission mode with sticky bit

This full Pod manifest creates an emptyDir volume using mode: 01777 to enforce standard Unix /tmp sticky bit protections across containers.

apiVersion: v1
kind: Pod
metadata:
 name: hardened-emptydir-pod
 namespace: default
spec:
 os:
 name: linux
 containers:
 - name: app-container
 image: alpine:latest
 command: ["sleep", "3600"]
 volumeMounts:
 - name: shared-tmp
 mountPath: /tmp
 volumes:
 - name: shared-tmp
 emptyDir:
 mode: 01777

Verifying the features in Linux

To verify that these features are actively enforcing restrictions, you can run kubectl exec into the container. The following examples simulate attempts to perform actions that are successfully blocked by these features.

Verifying noexec

Attempt to write and run a script on a volume mounted with noexec:

# 1. Exec into the pod
kubectl exec -it hardened-bindmount-pod -- sh

# 2. Create an executable script on the mounted volume
cd /tmp
echo '#!/bin/sh' > test.sh
echo 'echo "Executing untrusted code..."' >> test.sh
chmod +x test.sh

# 3. Attempt to run the script
./test.sh

Expected result:

sh: ./test.sh: Permission denied

Even if an executable file is created, the Linux kernel refuses execution because MS_NOEXEC is enforced at the bind mount level.

Verifying the sticky bit

Attempt to delete another user's file in an emptyDir with 01777 permission mode:

# 1. Exec into the pod
kubectl exec -it hardened-emptydir-pod -- sh

# 2. Verify directory permissions on /tmp
ls -ld /tmp
# Output: drwxrwxrwt 2 root root ... /tmp (Notice the 't' indicating sticky bit)

# 3. Create a file as the guest user
su -s /bin/sh -c "touch /tmp/guest_file" guest

# 4. Attempt to delete that file as nobody
su -s /bin/sh -c "rm /tmp/guest_file" nobody

Expected result:

rm: can't remove '/tmp/guest_file': Operation not permitted

The kernel blocks deletion because the sticky bit (01777) restricts file removal strictly to the owner of the file.

Things to know

Keep these key details in mind as you begin using these features. Full details are available in the official documentation for bind mount options, emptyDir volume mode, and emptyDir volumes.

• Default unchanged: If you omit bindMountOptions or do not set an emptyDir mode, you get standard default behaviors (like 0777 permissions) exactly as before.
• Broad volume support: bindMountOptions works with emptyDir, PersistentVolumes, CSI volumes, projected volumes, ConfigMaps, Secrets, and more. The only exception is image volumes, which are explicitly unsupported. The mode field works with all emptyDir medium types: default (disk-backed), Memory (tmpfs), and HugePages.
• Runtime capabilities matter (for bindMountOptions): The container runtime must support the CRI mount_options field and advertise it via runtimeFeatures. The scheduler uses node declared features to avoid placing pods on incompatible nodes. If a pod reaches such a node anyway, the kubelet rejects it. There is no silent degradation. However, using mode for an emptyDir does not require runtime support.
• Not the same as PV mountOptions: PersistentVolume mountOptions apply at the storage layer via the CSI driver. The new bindMountOptions controls bind mount flags applied inside the container by the runtime. They operate at different layers and do not conflict.
• fsGroup interaction: If fsGroup is set in the pod's security context, the group permissions applied by fsGroup will override the mode specified for the emptyDir volume. This is the same behavior that exists for defaultMode on Secret and ConfigMap volumes.
• Linux only: Flags like noexec, nosuid, nodev, and Unix permission modes are Linux concepts. bindMountOptions has no effect on Windows nodes. On Windows, the mode field is also skipped for emptyDir volumes, since Windows does not support Unix-style file permissions.
• Version skew safety: Both features are additive. For emptyDir mode: if the API server has the gate enabled but the kubelet does not, the field is accepted but ignored - the kubelet falls back to 0777. For bindMountOptions: the scheduler uses Node Declared Features to prevent placing pods on nodes without runtime support; if a pod reaches such a node, the kubelet rejects it rather than silently ignoring the options.
• Feature Gates: Both capabilities are available as Alpha features in Kubernetes v1.37:
  • VolumeBindMountOptions: Controls bind mount flags on volume mounts.
  • EmptyDirVolumeMode: Controls creation permission modes on emptyDir volumes.

How do I get involved?

These new features are driven by SIG Node and SIG Storage. You can find more details in the KEPs for these enhancements: KEP-5855 (bind mount options) and KEP-5502 (emptyDir permission mode).

Reach out to SIG Node:

• Slack: #sig-node
• Mailing list

Reach out to SIG Storage:

• Slack: #sig-storage
• Mailing list

With the release of Kubernetes v1.37, the Pod-Level Resource Managers feature has graduated to Beta status (disabled by default)!

First introduced as an Alpha feature in Kubernetes v1.36, this enhancement builds on Pod-Level Resources by equipping Kubelet's Topology Manager, CPU Manager, and Memory Manager to use Pod-level resource declarations (.spec.resources) directly when making hardware placement decisions.

Bringing pod-level resources to node managers

Before this feature, obtaining exclusive NUMA-aligned CPU cores or memory for latency-critical applications forced cluster operators into an all-or-nothing choice: assign integer resource requests to every container in the Pod, or forfeit exclusive NUMA alignment entirely. For modern workloads running lightweight sidecars (such as logging agents or telemetry exporters), allocating dedicated physical cores to auxiliary containers was wasteful.

Pod-Level Resource Managers solves this challenge by enabling hybrid allocation models. The Kubelet can reserve exclusive NUMA-aligned resources for primary application containers while placing non-Guaranteed sidecars into a pod-isolated shared pool. This ensures primary workloads get unthrottled, NUMA-local performance while sidecars benefit from running in a pod-isolated shared pool, enjoying local NUMA alignment and protection from external node interference without consuming dedicated physical cores.

What's new in Beta

Graduating to Beta brings key operational and API enhancements:

• Graduation to Beta: Controlled by the PodLevelResourceManagers feature gate, available to opt in (disabled by default) in Kubernetes v1.37.
• PodResources API Reporting: The v1 PodResources gRPC service (PodResourcesLister) introduces top-level cpu_ids and memory fields on PodResources responses. Monitoring tools and device plugins can query pod-level exclusive assignments directly without double-counting container allocations.

Getting started and providing feedback

For a deep dive into the technical details and configuration of this feature, check out the official documentation:

• Pod-level resource managers
• Pod-level resource managers reference

To follow a step-by-step tutorial on configuring and deploying workloads:

• Use pod-level resources with Kubelet resource managers

To learn more about how to assign resources to pods:

• Assign Pod-level CPU and memory resources

As this feature moves through Beta toward GA, your feedback is invaluable. Please report any issues or share your experiences via the standard Kubernetes communication channels:

• Slack: #sig-node
• Mailing list
• Open Community Issues/PRs

Changed Block Tracking (CBT) support for CSI drivers shipped as Alpha in September 2025. With the March 2026 v1.0.0 release of the external-snapshot-metadata project, the feature moved to Beta.

If you aren't yet familiar with changed block tracking for storage in Kubernetes, the Alpha announcement covers the motivation, the three primary components (the CSI SnapshotMetadata gRPC service, the SnapshotMetadataService CRD, and the external-snapshot-metadata sidecar), and a walkthrough of how to use the API. CBT currently applies to block volumes; file-volume and network file-share changed-list tracking is not covered by this feature. This post focuses on what is different in Beta.

What's new in Beta

The main change in that release was the promotion of the SnapshotMetadataService CRD from v1alpha1 to v1beta1. The CRD used to advertise a driver's metadata service now serves cbt.storage.k8s.io/v1beta1. The schema itself is unchanged, but this release removed v1alpha1 (rather than serving it alongside the new version). If you are upgrading from Alpha, you need to:

• Re-apply the CRD definition shipped with v1.0.0.
• Update SnapshotMetadataService manifests to use apiVersion: cbt.storage.k8s.io/v1beta1.
• Update any client or controller code that talks to the CRD.

This is a one-time change. There is no automatic conversion between the two versions.

Compatibility

• Minimum Kubernetes version: 1.33
• CSI spec: 1.10 or newer
• Container image: registry.k8s.io/sig-storage/csi-snapshot-metadata:v1.0.0

Trying it out

The Getting Started section in the Alpha blog still applies. In short:

1. Make sure your CSI driver supports volume snapshots and ships the external-snapshot-metadata sidecar.
2. Install the SnapshotMetadataService CRD (the v1beta1 definition from the v1.0.0 release).
3. Create a SnapshotMetadataService resource for your driver.
4. Use a client — snapshot-metadata-lister, or your own implementation — to call GetMetadataAllocated and GetMetadataDelta.

If you want to see the full flow end-to-end, the hostpath driver example is a good starting point.

What's next?

The focus for the rest of the Beta cycle is wider CSI driver adoption and operational feedback before the feature moves towards GA. If you maintain a CSI driver, this is a good time to evaluate adding support. If you are building a backup application on top of the API, feedback on the streaming clients and the iterator package is very welcome.

Where can I learn more?

• The CSI developer documentation for snapshot metadata.
• KEP-3314.
• The external-snapshot-metadata repository.
• The gRPC schema.
• The snapshot-metadata-lister example client.

How do I get involved?

This work is the result of contributions from many people across SIG Storage. A big thank you to everyone who helped review, code, and test the feature through Alpha and into Beta:

• Ben Swartzlander (bswartz)
• Carl Braganza (carlbraganza)
• Daniil Fedotov (hairyhum)
• Ivan Sim (ihcsim)
• Nikhil Ladha (Nikhil-Ladha)
• Praveen M (iPraveenParihar)
• Rakshith R (Rakshith-R)
• Xing Yang (xing-yang)

If you would like to get involved with CSI or storage in Kubernetes, SIG Storage is the place to start. The Data Protection Working Group also holds regular meetings, and new attendees are always welcome.

Memory QoS has graduated to Beta in Kubernetes v1.37 and is now enabled by default. On Linux nodes running cgroup v2, the feature uses the memory controller to give the kernel better guidance on how to treat container memory. It was first introduced as Alpha in v1.22, and expanded in v1.36 with tiered memory reservation.

This post covers what changed in v1.37, what the Beta promotion means for cluster operators, and how to configure the feature.

What changed in v1.37

Memory QoS is Beta and enabled by default

The MemoryQoS feature gate is now Beta in v1.37. This means every v1.37 kubelet has the feature gate turned on without any configuration change. Turning on the feature by default is safe because the default kubelet configuration does not enable memory throttling or memory reservation. No memory.high, memory.min, or memory.low values are written to cgroups unless you explicitly configure them.

You can opt into specific behaviors through kubelet configuration fields:

1. Set memoryThrottlingFactor (for example, 0.9) to enable memory.high throttling on Burstable and BestEffort containers. The default is null, which means no throttling.
2. Set memoryReservationPolicy to TieredReservation to enable tiered memory protection via memory.min and memory.low. The default is None, which means no memory reservation.

Default memoryThrottlingFactor changed to null

In earlier Alpha releases, memoryThrottlingFactor defaulted to 0.9, which meant enabling the feature gate caused the kubelet to set memory.high on containers. In v1.37, the default is null, so the kubelet does not set memory.high unless you configure a value.

This change was made because, with the feature gate now on by default, an automatic memory.high could throttle workloads that were previously running without throttling. Making it null ensures that upgrading to v1.37 does not change runtime behavior for existing clusters.

If your kubelet configuration file already contains an explicit memoryThrottlingFactor value, that value is preserved during the upgrade and throttling continues to work as before. If your configuration file does not include memoryThrottlingFactor, the kubelet uses the new null default and stops setting memory.high. To keep throttling in that case, add memoryThrottlingFactor explicitly:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

How to configure MemoryQoS in v1.37

For full details on configuring Memory QoS, see Memory QoS with cgroup v2, Configuring memory reservation, and System requirements

Enable memory throttling only

Set memoryThrottlingFactor to a value between 0 and 1. The kubelet uses this factor to calculate memory.high for Burstable and BestEffort containers. See Memory throttling for how memory.high is calculated for each QoS class.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

Enable memory throttling and tiered reservation

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9
memoryReservationPolicy: TieredReservation

Enable tiered reservation without throttling

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryReservationPolicy: TieredReservation

Disable Memory QoS entirely

To disable the feature after upgrading, set the feature gate to false and ensure a compatible kubelet configuration. The kubelet rejects the configuration if memoryThrottlingFactor is set to anything other than the former default of 0.9, or if memoryReservationPolicy is TieredReservation, so remove or adjust those fields if you set them.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
 MemoryQoS: false

When the feature gate is off, or memoryReservationPolicy is not TieredReservation, the kubelet resets stale protection at startup on cgroup v2 nodes: memory.min=0 and memory.low=0 on the root kubepods cgroup, and memory.low=0 on the Burstable QoS cgroup. For containers, stale memory.high values are reset to max on reconciliation paths such as restart or resize.

Known limitation: memory reservation is node-wide

memoryReservationPolicy applies to every pod on the node. With TieredReservation, every Guaranteed pod gets memory.min and every Burstable pod gets memory.low; there is no way to opt individual pods in or out. A node that mixes workloads needing hard reservation with workloads that should stay reclaimable has to choose one policy for all of them.

Hard reservation also covers everything charged to the container's cgroup, including page cache, so a pod that reads large files can hold memory the kernel would otherwise reclaim to serve its neighbors.

SIG Node is tracking both in kubernetes/kubernetes#140246. If this affects you, that issue is the best place to describe your workload.

What to expect next

The next milestone for Memory QoS is graduation to GA. Feedback from Beta users will shape any remaining adjustments before that step. If you run into issues, please file bugs at kubernetes/kubernetes.

How can I learn more?

• KEP-2570: Memory QoS
• Pod Quality of Service Classes
• Memory QoS with cgroup v2
• Managing Resources for Containers
• Kubernetes cgroups v2 support
• Linux kernel cgroups v2 documentation

Getting involved

This feature is driven by SIG Node. If you are interested in contributing or have feedback, you can reach out through:

• Slack: #sig-node
• Mailing list
• SIG Node meetings

I'm excited to announce that native histogram support for Kubernetes metrics is graduating to Beta and is enabled by default in Kubernetes v1.37!

Native histograms (previously introduced as Alpha in Kubernetes v1.36 under KEP-5808) bring high-resolution, low-cardinality observability to Kubernetes metrics. By adopting Prometheus Native Histograms, Kubernetes components now expose latency and duration metrics with far greater accuracy while significantly reducing telemetry storage and scraping overhead.

Why move beyond classic histograms?

Since the early days of Kubernetes observability, duration and latency metrics (such as API server request latencies or scheduling durations) have relied on classic Prometheus histograms.

Classic histograms require metric authors to define a static list of cumulative bucket boundaries (le labels), such as 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10. While familiar, this approach introduces three major challenges:

1. The Bucket Guessing Game: If a workload's latency profile changes, for example, shifting into microsecond ranges or experiencing long-tail tail latencies beyond the highest bucket, the histogram loses visibility. Specifying bucket boundaries upfront requires knowing the distribution before observing it
2. High Cardinality & Storage Cost: With classic histograms, each bucket boundary is exported as a separate time series (_bucket{le="..."}). A histogram with 10 buckets across multiple labels multiplies the number of time series by 10, increasing memory consumption in Prometheus and inflating time series database (TSDB) storage costs
3. Interpolation Error in Quantiles: Calculating percentiles using histogram_quantile() relies on linear interpolation between static bucket boundaries. When bucket spans are coarse, quantile calculations can suffer from significant estimation error

What are Prometheus native histograms?

Prometheus Native Histograms replace static user-defined buckets with dynamic, exponential buckets.

Instead of emitting a separate time series for every single bucket boundary, a native histogram is stored as a single time series containing a rich schema of positive and negative spans, zero thresholds, and exponential scaling factors.

• High Resolution Automatically: Exponential buckets dynamically adjust to any value range — from nanoseconds to hours — without requiring pre-configured bucket boundaries
• Up to 90% Fewer Time Series: By consolidating buckets into structured spans within a single time series, scraping and storage overhead are dramatically reduced
• Accurate Quantile Calculation: Quantiles can be calculated with mathematical bounds on error (≃5% worst-case relative error under default settings) across the entire spectrum of observations

How native histograms work in Kubernetes

In Kubernetes, native histogram support is implemented directly inside the shared metrics subsystem (k8s.io/component-base/metrics).

Figure 1 illustrates how native histogram metrics are processed and exposed across Kubernetes components.

1. Dual exposition for zero breaking changes

A primary design requirement for KEP-5808 was zero disruption for existing observability stacks. When the NativeHistograms feature gate is enabled, Kubernetes components use dual exposition:

• Classic buckets (h.Bucket) are still emitted alongside native spans. Existing Prometheus servers, dashboards, and alerting rules that rely on traditional text scraping or classic bucket labels continue to work unmodified
• Native spans (h.Schema, h.PositiveSpan) are included in the same Protobuf payload for collectors that understand native histograms

2. Tuned default exponential configuration

When NativeHistograms is enabled, the k8s.io/component-base/metrics package automatically applies standardized exponential options to all histogram metrics:

• BucketFactor: 1.1: Configures exponential buckets where each bucket is at most 10% wider than the preceding one. This guarantees a mathematically bounded worst-case relative error of at most ~5% for quantile calculations regardless of whether an operation takes 1 millisecond or 10 seconds.
• MaxBucketNumber: 160: Caps the maximum number of buckets per histogram to 160. Following OpenTelemetry SDK recommendations for base-2 exponential histogram aggregation, this limit protects component memory usage even under extreme outlier distributions.

3. Broad component support

Because native histograms are integrated into component-base/metrics, all major Kubernetes control plane and node components inherit support automatically, including:

• kube-apiserver (e.g., apiserver_request_duration_seconds, authentication/authorization metrics, validation latencies)
• kube-scheduler (e.g., scheduler_plugin_execution_duration_seconds, scheduler_scheduling_algorithm_duration_seconds)
• kubelet (node-level container runtime and pod lifecycle metrics)
• kube-controller-manager and kube-proxy

How to scrape native histograms

The simple answer: upgrade to Kubernetes v1.37, and it works.

Because Kubernetes v1.37 enables NativeHistograms by default, your cluster is already emitting dual-exposition metrics. How you configure Prometheus to scrape native histograms depends on your Prometheus version:

1. Prometheus scrape configuration by version

• Prometheus 3.0+ (Recommended): Use explicit per-job configuration in your scrape_configs rather than global flags (the global --enable-feature=native-histograms flag is deprecated in Prometheus 3.9+):

  scrape_configs:
   - job_name: 'kubernetes-apiservers'
   scrape_native_histograms: true
   always_scrape_classic_histograms: true # Recommended during transition
  

  You must read the caution in Migrating dashboards and alerts in the Native Histograms documentation. In summary: always set always_scrape_classic_histograms: true during your transition period. Without this setting, Prometheus will only ingest the native format and stop ingesting classic _bucket, _count, and _sum series. Setting always_scrape_classic_histograms: true ensures existing dashboards (histogram_quantile(..._bucket...)) and alerts continue to work while you migrate them to native histograms.

• Prometheus 2.40 – 2.x: Enable Native Histograms globally by starting Prometheus with the feature flag:

  prometheus --enable-feature=native-histograms
  

  Note that in Prometheus 2.x, this is an all-or-nothing setting for all scrape targets.

2. Verify Protobuf dual exposition

Standard Prometheus text scraping (application/openmetrics-text or plain text format) only transfers classic buckets. When scrape_native_histograms is enabled, Prometheus automatically negotiates Protobuf format with Kubernetes endpoints.

You can verify that a Kubernetes component is exporting native histograms using curl with an Accept header specifying Protobuf. For example:

## THIS IS NOT SECURE. ONLY DO THIS IN A TEST CONTEXT.
curl --insecure \
 -H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited" \
 --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
 https://localhost:6443/metrics

When decoded, the returned MetricFamily for histogram metrics (like apiserver_request_duration_seconds) will contain both traditional bucket entries and populated schema / positive_span fields.

Querying native histograms in PromQL

Once native histograms are ingested into Prometheus, you can query them using standard PromQL histogram functions without needing static le bucket labels or _bucket suffixes:

# 1. Calculating P99 latency for a single target:
# Classic histogram (requires _bucket suffix):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))

# Native histogram (operates directly on the metric name):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))

# 2. Aggregating across multiple instances (e.g., all API servers):
# Classic histogram (requires sum by (le) to preserve bucket boundaries):
histogram_quantile(0.99, sum by (le) (rate(apiserver_request_duration_seconds_bucket[5m])))

# Native histogram (no grouping by le required!):
histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds[5m])))

With native histograms, functions like histogram_quantile() operate directly on the dynamic exponential spans inside the time series, producing highly accurate quantiles without static bucket interpolation error.

For official documentation on querying Native Histograms in PromQL, see:

• PromQL histogram_quantile documentation
• PromQL histogram_fraction documentation
• PromQL histogram_sum documentation
• PromQL histogram_count documentation
• PromQL histogram_count documentation
• PromQL histogram_stddev and histogram_stdvar documentation

Dashboard migration & rollback strategy

Recommended migration workflow

To safely transition your monitoring infrastructure to Native Histograms without breaking existing alerts or dashboards, I recommend a four-step migration workflow:

1. Enable Both Formats: In your Prometheus 3.x scrape config, set scrape_native_histograms: true AND always_scrape_classic_histograms: true so both formats are collected safely during transition
2. Migrate Queries: Update your Grafana dashboards and Prometheus alerting rules from classic quantile queries (histogram_quantile(..._bucket...)) to native histogram queries (histogram_quantile(...)), and replace references to classic _count and _sum series with histogram_count(...) and histogram_sum(...)
3. Verify in Staging/Production: Validate that all dashboards and SLO alerts fire and graph correctly using the new native histogram queries
4. Unlock ~10x Storage Savings: Once migration is complete, set always_scrape_classic_histograms: false. Prometheus will stop ingesting the static _bucket, _count, and _sum time series, reducing your histogram time series count by up to 90%!

Opt-out and rollback flexibility

Because native histograms are dual-exposed, using them is entirely opt-in from a collector perspective:

• Instant Collector Rollback: If you need to stop ingesting native histograms, simply set scrape_native_histograms: false in your Prometheus job configuration. No Kubernetes restart is required, and Prometheus will immediately resume scraping only the classic format without data loss
• Component Feature Gate Rollback: Administrators can also disable the feature gate on Kubernetes components using --feature-gates=NativeHistograms=false (requires component restart)

What's next & how to get involved

As native histograms progress toward General Availability (GA) in future Kubernetes releases, SIG Instrumentation will continue evaluating ecosystem readiness, performance characteristics, and long-term plans for eventually deprecating static classic buckets once native histogram adoption becomes ubiquitous across the monitoring community.

• Read the KEP-5808 page or the KEP GitHub issue to learn more.
• Read the Prometheus Native Histograms specification and PromQL querying functions documentation
• Get involved with SIG Instrumentation on Slack in #sig-instrumentation or join the weekly SIG meetings

Acknowledgements

A huge thank you to contributors across SIG Instrumentation and component owners who collaborated on the design, implementation, testing, and review of native histograms in Kubernetes!

In Kubernetes, resource allocation has historically been a static decision made during a Pod's initial scheduling and placement. With the graduation of the core in-Place Pod resize feature to General Availability in v1.35, application developers and cluster operators gained the powerful ability to dynamically adjust CPU and memory allocations of running containers without incurring disruptive restarts or application downtime.

However, in-place resizing introduced a unique resource scheduling gap: if a running Pod requested a resource scale-up that exceeded the host node's allocatable headroom, the Kubelet was forced to mark the request as Deferred. The Pod would remain parked in this state indefinitely, waiting for resources on the node to naturally free up.

To bridge this scheduling gap, Kubernetes v1.37 introduces scheduler preemption for in-place Pod resize (Alpha), behind the InPlacePodVerticalScalingSchedulerPreemption feature gate. This feature allows the Kubernetes scheduler to actively free up capacity on a fully-utilized node by preempting lower-priority workloads, enabling the pending in-place resizes of critical, higher-priority applications to succeed.

The "deferred" resize challenge

To understand why this preemption mechanism is needed, it is helpful to look at how Kubernetes handles running Pod resizing. When a user or controller (such as the Vertical Pod Autoscaler) updates the resource requests of an active container, the Kubelet evaluates whether the underlying node has enough spare allocatable capacity to fulfill the increase.

If the node's resources are fully utilized and cannot satisfy the new limits, the Kubelet sets the container's resizeStatus (reported in the Pod's status.containerStatuses[]) to Deferred. Unlike an Infeasible resize request (which is immediately rejected because it exceeds physical machine boundaries, namespace limit ranges, or admission quotas) a Deferred status indicates that the request is valid but is temporarily unable to be actuated, waiting until node capacity becomes available.

Before the introduction of this preemption mechanism, a Pod's in-place resize scale-up request could become permanently blocked if the node was heavily utilized. Even when a critical application (such as an in-memory database or a real-time web server) required more memory to prevent an imminent out-of-memory (OOM) crash, and the node lacked free capacity, the resize remained Deferred.

In this scenario, cluster administrators had limited choices:

1. Manually evict lower-priority Pods from the node to clear resource headroom.
2. Rely on the cluster autoscaler to eventually spin up a larger node and reschedule the Pod. However, this is an operation that is highly disruptive and violates the core "no restart" value proposition of in-place scaling.
3. Rely on a custom autoscaling solution, for example a cluster autoscaler that can trigger dynamic node resizing operations itself.

Because the kube-scheduler was unaware of deferred resizes on running Pods, it could not leverage standard priority-based preemption to evict lower-priority workloads and make room for the higher-priority running Pod's resource growth.

Why this matters

In production Kubernetes environments, cluster administrators strive to maximize resource utilization and efficiency. A common strategy is to bin-pack unused capacity on not-yet-full nodes with lower-priority workloads, such as batch jobs, background data processing, or best-effort tasks.

Without scheduler preemption for in-place resizing, this created a major operational dilemma. If lower-priority workloads consumed the remaining headroom on a node, higher-priority applications running on that same node would become blocked (Deferred) when they needed to scale up to handle sudden traffic surges or memory spikes. Operators were forced to choose between running low-utilization clusters with idle buffer capacity or risking that critical workloads could not resize when needed.

With scheduler preemption for in-place Pod resize, you can confidently bin-pack unused space across your clusters with lower-priority workloads without worrying about them degrading higher-priority Pods or blocking their scale-up requests. If a high-priority workload requires an in-place resize that exceeds available node capacity, the scheduler automatically preempts the lower-priority Pods to clear headroom. You achieve high cluster utilization and cost efficiency while preserving the responsiveness and reliability of critical services.

Architectural mechanics: How it works

Scheduler preemption for in-place Pod resize integrates directly into the core scheduling cycle to coordinate resources dynamically and safely.

Centralized scheduler tracking

The kube-scheduler monitors the cluster for running Pods with a Deferred resize status condition. Normally, Pods with spec.nodeName populated are considered successfully placed and bypass the active scheduling queue. Under this feature gate, the scheduler intercepts Pods carrying the Deferred condition, permitting them to remain in active scheduling evaluations specifically to trigger preemption. The scheduler maintains continuous tracking of these Pods until the Kubelet successfully completes the resize actuation.

Single-node preemption boundary

Unlike placement preemption, which evaluates all nodes in a cluster to find the best scheduling fit, preemption for in-place resizing is strictly localized to the Pod's currently assigned node. The scheduler identifies eligible lower-priority "victim" Pods on the same host and initiates their graceful eviction, freeing up local capacity. Preemption is strictly scoped to the same node where the deferred Pod is running; if a node cannot accommodate the resize even after evicting all eligible lower-priority workloads, the resize remains in the Deferred state.

Resource reservation safety

To prevent scheduling races and double-allocation, the scheduler treats resources requested for a resize as already consumed. This enables the Kubelet to actuate the resize once the preemption takes effect.

Separation of concerns & critical admission

When a node is under resource pressure, the Kubelet includes a local mechanism known as the critical Pod admission handler. During initial Pod admission, if a critical system Pod arrives on a node that lacks spare capacity, this local handler can directly evict lower-priority Pods on that node to guarantee admission for the critical workload.

A significant architectural benefit of this new feature is the strict separation of concerns between the Kubelet and the scheduler. Under the InPlacePodVerticalScalingSchedulerPreemption feature gate, the Kubelet's critical Pod admission handler does not perform local preemption checks or trigger local evictions for in-place resizing operations. Instead, the Kubelet defers the request and delegates the preemption decision entirely to the scheduler. This guarantees that a single, centralized orchestrator manages all resize-related preemption logic, respecting global priorities, Pod disruption budgets (PDBs), and graceful termination policies.

Managing competing updates & races

If a competing, higher-priority resize request is submitted for another running Pod on the same node during an active preemption cycle, the Kubelet prioritizes the higher-priority request. The scheduler is designed to observe these updates and will dynamically trigger a new round of preemption if more capacity is required to fulfill the new state.

Node-level preemption configuration

Administrators and automated controllers (such as a cluster autoscaler) can disable preemption specifically for in-place resizes on particular nodes. This is configured using the new spec.podPreemptionPolicy field in the Node Spec:

apiVersion: v1
kind: Node
metadata:
 name: batch-workload-node
spec:
 podPreemptionPolicy:
 disableResizePreemption:
 - "cluster-autoscaler.kubernetes.io/disable-preemption"
 - "operator.example.com/policy-override"

An example use case for this policy is when a controller would prefer to size down other pods or dynamically adjust the node capacity itself when possible, only enabling scheduler preemption as a last resort.

Try it out!

To utilize scheduler preemption for in-place Pod resize:

• Your cluster must be running Kubernetes v1.37 or later across both the control plane and all worker nodes.
• The InPlacePodVerticalScalingSchedulerPreemption feature gate must be enabled across all control plane components (kube-apiserver, kube-scheduler) and the kubelet.

Mini-tutorial: Observe resize preemption in action

To see this feature in action locally, you can test scheduler preemption on a single-node kind cluster with constrained CPU headroom.

1. Create a kind cluster with scheduler resize preemption enabled

Create a kind cluster configuration file named kind-config.yaml with the InPlacePodVerticalScalingSchedulerPreemption feature gate enabled:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
 InPlacePodVerticalScalingSchedulerPreemption: true

Create the cluster using this configuration, passing the --image flag to ensure the cluster is running Kubernetes v1.37 (or later):

kind create cluster --config kind-config.yaml --image kindest/node:v1.37.0

Once your cluster is ready, inspect the node to check how many allocatable CPU cores it has:

kubectl get nodes -o custom-columns=NAME:.metadata.name,ALLOCATABLE_CPU:.status.allocatable.cpu

In a standard local kind environment, the output shows 8 allocatable CPU cores:

NAME ALLOCATABLE_CPU
kind-control-plane 8

2. Create PriorityClasses and deploy Pods

Create two PriorityClasses and deploy a low-priority Pod (requesting 3 CPU) alongside a high-priority Pod (requesting 4 CPU). Together, these workloads consume 7 of the 8 available CPU cores, leaving 1 CPU of free allocatable headroom on the node.

# preemption-demo.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: high-priority
value: 1000000
globalDefault: false
description: "High priority workload"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: low-priority
value: 1000
globalDefault: false
description: "Low priority workload"
---
apiVersion: v1
kind: Pod
metadata:
 name: low-priority-pod
spec:
 priorityClassName: low-priority
 containers:
 - name: worker
 image: nginx
 resources:
 requests:
 cpu: "3"
 memory: "500Mi"
 limits:
 cpu: "3"
 memory: "500Mi"
---
apiVersion: v1
kind: Pod
metadata:
 name: high-priority-pod
spec:
 priorityClassName: high-priority
 containers:
 - name: app
 image: nginx
 resources:
 requests:
 cpu: "4"
 memory: "1Gi"
 limits:
 cpu: "4"
 memory: "1Gi"

Save this manifest to preemption-demo.yaml and apply it:

kubectl apply -f preemption-demo.yaml

Wait until both Pods are running on the node:

kubectl get pods

Output:

NAME READY STATUS RESTARTS AGE
high-priority-pod 1/1 Running 0 9s
low-priority-pod 1/1 Running 0 9s

3. Request an in-place scale-up

Patch the high-priority Pod to increase its CPU request from 4 to 6 (+2 CPU delta). Because only 1 CPU of headroom is free on the node, this resize request exceeds remaining allocatable capacity:

kubectl patch pod high-priority-pod --subresource resize --patch \
 '{"spec":{"containers":[{"name":"app", "resources":{"requests":{"cpu":"6"}, "limits":{"cpu":"6"}}}]}}'

4. Inspect the preemption event on the low-priority Pod

With InPlacePodVerticalScalingSchedulerPreemption enabled, the scheduler intercepts the Deferred resize condition on high-priority-pod and targets low-priority-pod for preemption.

To verify that the scheduler actively preempted the low-priority Pod, inspect its events:

kubectl get events --field-selector involvedObject.name=low-priority-pod

In the event stream (or via kubectl describe pod low-priority-pod), you will see a Preempted event emitted by the scheduler:

LAST SEEN TYPE REASON OBJECT MESSAGE
5s Normal Preempted pod/low-priority-pod Preempted by pod 97dba925-6b5f-4e2f-99f9-d51c30016586 on node kind-control-plane
5s Normal Killing pod/low-priority-pod Stopping container worker

5. Trace the resize event lifecycle on the high-priority Pod

Next, inspect the event history on high-priority-pod to observe how the resize progressed from being deferred to successfully completed:

kubectl get events --field-selector involvedObject.name=high-priority-pod

You will observe a sequence of events as the Kubelet coordinates with the scheduler:

LAST SEEN TYPE REASON OBJECT MESSAGE
33s Warning ResizeDeferred pod/high-priority-pod Pod resize OutOfcpu: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2,"error":"Node didn't have enough resource: cpu, requested: 6000, used: 3950, capacity: 8000"}
32s Normal ResizeStarted pod/high-priority-pod Pod resize started: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2}
32s Normal ResizeCompleted pod/high-priority-pod Pod resize completed: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2}

1. ResizeDeferred: The Kubelet initially marks the resize request as deferred (Warning) due to insufficient CPU headroom on the node (OutOfcpu).
2. ResizeStarted: Once the scheduler preempts low-priority-pod and capacity is released, the Kubelet accepts the new allocation and begins actuating the resize.
3. ResizeCompleted: The Kubelet successfully updates container cgroup limits via the container runtime without restarting the Pod.

Finally, verify that the allocated CPU on the container reflects the new request (appending {"\n"} to the JSONPath query ensures a trailing newline in your terminal):

kubectl get pod high-priority-pod -o jsonpath='{.status.containerStatuses[0].allocatedResources.cpu}{"\n"}'

Output:

6

This confirms that the in-place resize succeeded.

Getting involved

This feature represents a major step forward for resource scheduling, bringing enterprise-grade density control and workload prioritization to dynamic resource scaling. We invite cluster operators, platform architects, and developers to enable the InPlacePodVerticalScalingSchedulerPreemption feature gate in their testing environments and share feedback.

If you want to share your experience with this feature, please get in touch with the community via SIG Scheduling or SIG Node channels!

Kubernetes has many ways to describe what is happening on a Node. Readiness, taints, Pod state, labels, annotations, and provider-specific APIs each expose part of the picture. What has been missing is a shared, Kubernetes-owned way to say that a Node is draining, undergoing maintenance, or undergoing Graceful Node Shutdown.

Kubernetes v1.37 introduces five well-known Node conditions that provide that description:

• DrainInProgress
• Drained
• MaintenancePlanned
• MaintenanceInProgress
• GracefulNodeShutdownInProgress

The new Node lifecycle conditions

Maintenance can include hardware or software rollout, remediation, decommissioning, or debugging. Whether maintenance requires a drain depends on its impact. A Kubernetes upgrade usually should follow a drain, while a kernel live patch might not need one.

Like other Node conditions, each lifecycle condition uses status to report whether the observation is active:

• True: the lifecycle state is currently observed.
• False: the lifecycle state is not currently observed.
• Unknown: Kubernetes cannot determine whether the lifecycle state is active.

The reason provides a stable, machine-readable cause for the current status, and message can provide additional human-readable detail.

For example, an authorized maintenance controller could publish:

# Node .status excerpt
status:
 conditions:
 - type: MaintenancePlanned
 status: "True"
 reason: MaintenanceWindow
 lastTransitionTime: "2026-12-09T12:00:00Z"
 message: "Hardware maintenance is scheduled for this Node"

What changes in Kubernetes v1.37

The v1.37 release reserves these names as well-known NodeConditionType constants and introduces the Alpha NodeLifecycleConditions feature gate, which is disabled by default. In v1.37 the gate is effectively a no-op: it does not restrict who can set these conditions, and no core component reads them. It exists so that the built-in behavior planned for future releases — controllers that consume these conditions — can be opted into when it arrives. You do not need to enable it to start publishing the conditions today.

For this release, an administrator or an administrator-authorized controller is responsible for setting and clearing the lifecycle conditions.

In this first release, no core workload controller changes its behavior based on these conditions, but an administrator can publish them to communicate maintenance and drains to cluster users.

How to use lifecycle conditions today

The immediate value is operational clarity. Administrators and lifecycle automation can use these conditions as a common status channel for Node lifecycle work that already happens today.

For example, maintenance automation can set MaintenancePlanned when a future maintenance window is scheduled, then set MaintenanceInProgress when work starts. Drain automation can set DrainInProgress when it begins evicting Pods and Drained when the administrator's selected drain criteria have been met. The GracefulNodeShutdownInProgress condition can report that Graceful Node Shutdown is in progress on the Node.

The recommended pattern is to use lifecycle conditions to report status, while lifecycle operations are managed through other mechanisms. Continue to use existing Kubernetes mechanisms such as kubectl cordon, kubectl drain, taints, and workload-specific controls to change scheduling or eviction behavior. Use lifecycle conditions to make the state of that work visible to people, dashboards, alerts, and automation that choose to consume the signal.

When setting a condition, use True while the lifecycle state is active. Set the condition to False, or remove it, when the state is no longer active. Use a stable reason value and a clear message so that both people and automation can understand why the condition changed. Cluster administrators should also decide which component owns each lifecycle condition to avoid conflicting writes.

Why a shared signal matters

Node lifecycle affects components across the cluster. The kubelet, node lifecycle controller, workload controllers, scheduler, autoscalers, storage operators, and external maintenance systems all need some understanding of what is happening to a Node.

Today, each component has to reconstruct that understanding from indirect signals. One controller might look at Node readiness, another at taints, and another at Pods that are terminating or missing. Infrastructure providers and operators often add their own labels or annotations.

Those signals remain useful for their intended purposes, but they do not answer the same question. A taint can influence scheduling or eviction, for example, but it does not attest that a drain is in progress or that an administrator's drain criteria have been met. A NotReady Node does not explain whether the cause is an unexpected failure, a graceful shutdown, or planned maintenance.

Without shared lifecycle context, independently correct components can make conflicting decisions. A DaemonSet controller can replace a Pod that the kubelet intentionally terminated during graceful shutdown. A Job controller can wait indefinitely for a terminal Pod phase on a Node that an administrator is removing. A storage operator might learn about maintenance only after drain has already started.

The new conditions provide a stable place on the Node for that missing context, as part of the larger effort to enhance Node Lifecycle management.

The foundation for lifecycle-aware Kubernetes

The value of a shared signal comes from what can consume it — core controllers, administrators, or the ecosystem of lifecycle projects. Follow-up enhancements can build on the conditions without every component inventing a different way to infer Node lifecycle state.

Consider a long-standing DaemonSet rollout edge case. A Node that is broken or undergoing maintenance can remain unavailable for reasons unrelated to the new DaemonSet revision. That Node still consumes the rollout's availability budget, which can slow or block the controller from progressing the rollout on healthy Nodes.

The DaemonSet controller knows that a Pod is unavailable, but it cannot tell whether the new revision failed or an administrator intentionally took the Node out of service. Readiness, taints, and Pod state expose pieces of the situation, but none provides authoritative maintenance context.

The MaintenanceInProgress condition creates a Kubernetes-owned place to publish that context. Future work can define how the DaemonSet controller uses it for rollout ordering, availability accounting, and status reporting. Those behaviors still require careful design, but the goal is for administrators to no longer have to manually adjust the rollout.

Future expansions and getting involved

Node lifecycle is a cross-cutting problem. Solving it starts with components sharing enough context to make compatible decisions. The next stage is to build on Node Lifecycle Conditions to improve scenarios such as Graceful Node Shutdown, drain, and maintenance. Longer-term lifecycle coordination may require explicit ownership, locking, and potentially a dedicated API.

The Kubernetes ecosystem already includes many solutions for Node maintenance, remediation, drain, autoscaling, and fleet management. The experience behind those projects is essential to building a foundation that works across different environments and operational models. The Node Lifecycle Working Group, SIG Node, and SIG Apps invite maintainers and users to share their use cases and ideas to shape the future work.

Follow the work through KEP-5683: Node Lifecycle Conditions. To participate in our discussions, join one of our groups:

• Node Lifecycle Working Group
• SIG Node
• SIG Apps

AI/ML and complex batch workloads continue to push the boundaries of Kubernetes scheduling. Following the foundational workload-centric enhancements introduced in previous releases, Kubernetes v1.37 delivers the next major milestone in the Workload-Aware Scheduling (WAS) journey. In this release, the core Workload and PodGroup APIs—enabling gang scheduling—along with Workload-Aware Preemption (WAP) and shared DRA ResourceClaims for PodGroups, all graduate to Beta, solidifying their role in the Kubernetes ecosystem.

To address the hierarchical scheduling requirements of modern high-performance distributed workloads, v1.37 introduces the new CompositePodGroup API. This new API allows expressing multi-level topology constraints, gang scheduling, and preemption policies for complex, heterogeneous groups of Pods. Crucially, this architectural expansion unlocks native scheduling support for advanced workload structures commonly managed by higher-order extension APIs such as JobSet and LeaderWorkerSet (LWS).

Alongside these API additions, v1.37 focuses on streamlining adoption by introducing a new set of controller integration APIs and the workloadbuilder Go library. These provide standardized building blocks that significantly simplify how out-of-tree controllers can integrate with WAS capabilities. Utilizing these new tools, the native Job controller integration has been upgraded to fully consume the expanded WAS APIs—enabling advanced scheduling policies, flexible disruption modes, and topology-aware scheduling for standard batch workloads.

Gang scheduling and Workload / PodGroup APIs

Kubernetes v1.37 delivers a major milestone: Workload / PodGroup APIs and gang scheduling are officially graduating to Beta. This graduation signals that native, "all-or-nothing" scheduling for workloads is solidifying for wider adoption.

Key updates to the API and gang scheduling algorithm in this release include:

Beta graduation and API versioning changes

The core Workload and PodGroup APIs have been promoted to v1beta1, meaning they are now one step away from General Availability (GA). For early adopters who have been testing these features, take note of the alpha versioning transition: v1alpha2 has been entirely replaced by v1alpha3. This transition introduces breaking changes designed to clean up the API structure around disruptionMode.

Native PodGroup queueing

A significant under-the-hood improvement in v1.37 makes the PodGroup a first-class citizen in the scheduling queue. Previously, even if belonging to a PodGroup, all member Pods were queued individually. Now, only the top-level PodGroup object is queued. This ensures all Pods share the same queueing behavior and lays the groundwork for more advanced PodGroup queueing strategies in the future.

Dynamic elasticity with minCount mutability

In earlier iterations, the minCount field, which dictates the minimum number of Pods required to successfully schedule a PodGroup, was strictly immutable. In v1.37, minCount is now mutable. This API change unlocks flexibility for elastic workloads. Controllers can now dynamically adjust the minimum required size of a gang on the fly, allowing workloads to gracefully degrade or expand without interrupting already-scheduled Pods.

Workload-aware preemption

In Kubernetes v1.37 the separate WorkloadAwarePreemption feature gate for workload-aware preemption was merged into the GenericWorkload feature gate, becoming a core part of the gang scheduling effort.

While the core concepts of workload-aware preemption stay the same, there are some differences between the v1.36 and v1.37 releases:

Performance and optimality

To check whether a preemptor can fit in the cluster thanks to preemption, the scheduler simulates the removal of all potential victims and re-runs the scheduling algorithm. After that it tries to reprieve as many victims as possible. In the v1.36 release, the scheduling algorithm was run for each victim reprieval, verifying whether with the victim reprieved, the algorithm can still find a valid placement for the preemptor. In v1.37, the scheduling algorithm is run only once and the preemptor Pods are assumed based on its output. Later, the reprieval checks whether a victim can still run in its place with the preemptor assumed.

PodGroup as a victim

One of the limitations of v1.36 was the fact that the default preemption for single Pods was not aware of PodGroups and was not respecting their disruptionMode fields, allowing for disruption of single Pods even when the PodGroup had disruptionMode: {all: {}} set. Kubernetes v1.37 removes this limitation; the default preemption now respects the PodGroup disruptionMode field.

Rename of the disruptionMode fields

During the promotion of the API to Beta, the disruptionMode field was changed to decouple its naming from the PodGroup object, allowing consistent naming across PodGroups and CompositePodGroups. The modes changed as follows: PodGroup became all, and Pod became single.

Support for preemptionPolicy

In v1.36, the PodGroup does not have a preemptionPolicy field. The PodGroup can perform preemption as long as none of the Pods forming it has preemptionPolicy: Never set. In v1.37, when the PodGroupPreemptionPolicy feature gate is enabled, a PodGroup also has a preemptionPolicy field. It serves as an authoritative field for whether a PodGroup can perform preemption.

CompositePodGroup API

In Kubernetes v1.36, workload-aware scheduling established a clean separation between static workload templates (Workload) and runtime group state (PodGroup), but the supported scheduling policies were limited to a single, flat group. The CompositePodGroup API, introduced in Kubernetes v1.37, extends this model to support hierarchical scheduling requirements.

This API allows its consumers to express multi-level scheduling requirements by organizing a workload in a tree-shaped hierarchy consisting of CompositePodGroup and PodGroup objects. Each CompositePodGroup carries policies and constraints that apply to other groups (CompositePodGroups and/or PodGroups), similar to how PodGroups govern scheduling behavior for a flat group of Pods. The scheduler treats such a hierarchy as a single scheduling unit and aims to satisfy the requirements specified by every group within that hierarchy.

Defining a workload hierarchy

To express multi-level scheduling requirements, you define a hierarchy of templates in a Workload object. Controllers then create the corresponding CompositePodGroup and PodGroup objects from that hierarchy.

To support this, the Workload API is extended with the spec.compositePodGroupTemplates field. Each CompositePodGroupTemplate defines a template for a parent CompositePodGroup and directly nests the templates (podGroupTemplates and/or compositePodGroupTemplates) from which its child groups derive.

Below is a sample Workload object that defines a two-level template hierarchy:

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
 name: example-workload
 annotations:
 kubernetes.io/description: "Two-level workload hierarchy requiring 4 worker Pods and 1 driver Pod to schedule together."
spec:
 compositePodGroupTemplates:
 - name: workload-root
 schedulingPolicy:
 gang:
 minGroupCount: 2
 podGroupTemplates:
 - name: workers
 schedulingPolicy:
 gang:
 minCount: 4
 - name: driver
 schedulingPolicy:
 gang:
 minCount: 1

After creating example-workload, a controller can stamp out the corresponding runtime group objects from these templates:

1. A root CompositePodGroup that references the workload-root template in example-workload and carries its group-level scheduling policy (gang scheduling with minGroupCount: 2):

   apiVersion: scheduling.k8s.io/v1alpha3
   kind: CompositePodGroup
   metadata:
    name: example-root-group
    annotations:
    kubernetes.io/description: "Root group coordinating gang scheduling across child worker and driver PodGroups."
   spec:
    workloadRef:
    workloadName: example-workload
    templateName: workload-root
    schedulingPolicy:
    gang:
    minGroupCount: 2
   

2. Two child PodGroup objects (example-workload-workers and example-workload-driver) that reference their respective leaf templates in example-workload and link to the root group via parentCompositePodGroupName:

   apiVersion: scheduling.k8s.io/v1beta1
   kind: PodGroup
   metadata:
    name: example-workload-workers
    annotations:
    kubernetes.io/description: "Worker group requiring at least 4 Pods to be scheduled together."
   spec:
    parentCompositePodGroupName: example-root-group
    workloadRef:
    workloadName: example-workload
    templateName: workers
    schedulingPolicy:
    gang:
    minCount: 4
   ---
   apiVersion: scheduling.k8s.io/v1beta1
   kind: PodGroup
   metadata:
    name: example-workload-driver
    annotations:
    kubernetes.io/description: "Driver group requiring 1 Pod to schedule alongside the workers."
   spec:
    parentCompositePodGroupName: example-root-group
    workloadRef:
    workloadName: example-workload
    templateName: driver
    schedulingPolicy:
    gang:
    minCount: 1
   

How multi-level gang scheduling works

To schedule a hierarchical workload, kube-scheduler evaluates the entire group tree as a unified scheduling unit:

• Recursive evaluation: The scheduler traverses the hierarchy from the root CompositePodGroup down to the leaf PodGroup objects. At each level, a parent CompositePodGroup is considered schedulable only when its child groups satisfy its scheduling policy (for example, placing at least minGroupCount of child groups when using the gang policy), while each leaf PodGroup must satisfy its own Pod-level policy (for example, placing at least minCount of member Pods when using the gang policy).
• All-or-nothing scheduling: Once a valid combination of child groups is found that satisfies the requirements of the root CompositePodGroup, the Pods across the entire hierarchy are scheduled and bound atomically. If the root group cannot satisfy its policy constraints, the entire hierarchy remains unschedulable and no Pods are bound, preventing partial deployments and deadlocks.

Workload-aware preemption for the CompositePodGroup API

Kubernetes v1.37 extends workload-aware preemption to support CompositePodGroup hierarchies as well. Specifically, if a CompositePodGroup cannot be scheduled due to insufficient capacity in the cluster, the scheduler can invoke preemption to evict lower-priority workloads in order to fit the Pods belonging to that CompositePodGroup.

A CompositePodGroup can be selected for preemption as well. To specify the desired behavior during preemption, workload owners can specify an appropriate disruptionMode in the CompositePodGroup spec:

• single: Allows individual child groups within the CompositePodGroup to be preempted and disrupted independently. This is the behavior when disruptionMode is not set.
• all: Enforces "all-or-nothing" disruption semantics across the entire CompositePodGroup hierarchy. If any Pod within the descendant subtree must be preempted, the scheduler evicts all Pods across the entire hierarchy together.

Topology-aware scheduling

In Kubernetes v1.37, topology-aware scheduling expands to support complex, multi-level workload hierarchies and delivers performance improvements for existing single-level deployments.

Multi-level topology-aware scheduling

In Kubernetes v1.36, we introduced foundational topology-aware scheduling, allowing you to define co-location constraints directly on a PodGroup. While effective for single-level groupings, complex distributed workloads—such as large-scale AI/ML training, JobSet deployments, or disaggregated inference via LeaderWorkerSet (LWS)—often require co-location across multiple levels of cluster infrastructure simultaneously.

For example, an entire workload may need to run within a single availability zone, while different parts of that workload (such as specific worker groups or driver processes) require strict co-location within specific server racks.

In Kubernetes v1.37, alongside the new CompositePodGroup API (scheduling.k8s.io/v1alpha3), topology-aware scheduling expands to support multi-level topology-aware scheduling. You can now express complex co-location requirements by specifying topology constraints at different levels of a group hierarchy.

Top-down topology constraint resolution

During hierarchical scheduling, the kube-scheduler resolves multi-level topology constraints in a top-down manner. Specifically, topology domains that are considered during the scheduling of a child group are confined within a topology domain that corresponds to the placement assumed by the parent group.

Configuration and runtime execution

Using the updated Workload API (scheduling.k8s.io/v1beta1), you can configure multi-level topology constraints directly within compositePodGroupTemplates. In the example below, the parent template constrains the overall workload to a single availability zone (topology.kubernetes.io/zone), while child templates for workers and driver constrain their respective Pods to server racks (topology.example.com/rack) within that selected zone:

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
 name: multi-level-tas-workload
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Workload defining zone-level co-location for the root group and rack-level co-location for child groups."
spec:
 compositePodGroupTemplates:
 - name: root
 schedulingPolicy:
 gang:
 minGroupCount: 2
 schedulingConstraints:
 topology:
 - key: topology.kubernetes.io/zone
 podGroupTemplates:
 - name: workers
 schedulingPolicy:
 gang:
 minCount: 8
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack
 - name: driver
 schedulingPolicy:
 gang:
 minCount: 1
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack

When a controller creates an instance of this workload at runtime, it spawns the corresponding runtime objects from these templates:

1. The root CompositePodGroup referencing the root template, carrying the availability zone topology constraint and the hierarchical gang scheduling policy.
2. The two child PodGroup objects (tas-workload-workers and tas-workload-driver), each referencing the root CompositePodGroup as their parent group via the parentCompositePodGroupName spec field:

apiVersion: scheduling.k8s.io/v1alpha3
kind: CompositePodGroup
metadata:
 name: tas-workload-root
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Root group constraining the entire workload to a single availability zone."
spec:
 workloadRef:
 workloadName: multi-level-tas-workload
 templateName: root
 schedulingPolicy:
 gang:
 minGroupCount: 2
 schedulingConstraints:
 topology:
 - key: topology.kubernetes.io/zone
---
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
 name: tas-workload-workers
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Worker group requiring 8 Pods co-located within a single rack in the selected zone."
spec:
 parentCompositePodGroupName: tas-workload-root
 workloadRef:
 workloadName: multi-level-tas-workload
 templateName: workers
 schedulingPolicy:
 gang:
 minCount: 8
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack
---
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
 name: tas-workload-driver
 namespace: job-ns
 annotations:
 kubernetes.io/description: "Driver group requiring 1 Pod placed in a rack within the selected zone."
spec:
 parentCompositePodGroupName: tas-workload-root
 workloadRef:
 workloadName: multi-level-tas-workload
 templateName: driver
 schedulingPolicy:
 gang:
 minCount: 1
 schedulingConstraints:
 topology:
 - key: topology.example.com/rack

During scheduling, the scheduler evaluates multiple candidate availability zones across the cluster for tas-workload-root. For each candidate zone, it subdivides the nodes by rack topology to explore feasible rack placements for tas-workload-workers and tas-workload-driver strictly within that zone, systematically evaluating multiple combinations across available zones and racks before making a scheduling decision.

By allowing topology constraints to be modeled hierarchically, Kubernetes v1.37 provides a structured way to express multi-level co-location requirements across complex cluster infrastructures.

Performance improvements for single-level TAS

Alongside the Alpha introduction of multi-level hierarchies, Kubernetes v1.37 reduces the cost of placement evaluation for existing single-level topology-aware scheduling. We are continuously working to optimize the efficiency of placement evaluation algorithms in kube-scheduler and plan to deliver further performance improvements in future releases.

Controller Integration APIs

Kubernetes v1.37 introduces new standard building blocks so that every controller can expose the same scheduling primitives in their own APIs, and share the same logic for translating them into scheduling objects. These primitives express specific scheduling behaviors — such as policies or disruption logic — while leaving the field naming flexible for each controller. A prime example of this is the native Job controller, which we detail in the next section.

Types prefixed with WorkloadPodGroup describe a leaf group of Pods; types prefixed with WorkloadCompositePodGroup describe a group of groups. A controller embeds them verbatim into its own API, under whatever field name fits its domain:

• WorkloadPodGroupSchedulingPolicy — either basic, meaning standard Pod-by-Pod scheduling, or gang with a minCount. The composite variant takes a minGroupCount instead.
• WorkloadPodGroupSchedulingConstraints — the topology constraints (topology[].key) the group's Pods must be co-located within.
• WorkloadPodGroupDisruptionMode — single or all, with the preemption semantics described earlier in this post.
• WorkloadPodGroupResourceClaim — the ResourceClaims shared across the group.

Only the shapes are shared, so controllers retain full autonomy over how they name and nest these fields in their own APIs.

The workloadbuilder library turns that intent into the scheduling objects. A controller describes its workload as a tree of WorkloadItem nodes — a node with children compiles to a CompositePodGroupTemplate, a node without children to a PodGroupTemplate — and attaches its own defaults plus the user-supplied building blocks to each node. From there, Validate() reports problems back at the exact field path within the controller's own API, BuildWorkload() compiles the tree into a Workload, and NewPodGroup() and NewCompositePodGroup() stamp out the runtime group objects.

Validation is deny-by-default: a controller declares the policies and disruption modes it actually supports through AllowedPolicies and AllowedDisruptionModes, and anything outside those lists is rejected. Building blocks added in future releases therefore stay unavailable until a controller explicitly opts into them.

For hierarchical workloads where a parent controller owns the Workload and delegates group creation to its children, NewBuilderFromExistingWorkload lets a child materialize only its own PodGroup from the parent's Workload.

Neither the building blocks nor the library have a feature gate of their own; they become user-visible through whichever controller adopts them. The native Job controller is the first to do so, and we detail it in the next section.

Integration with the Job controller

Building upon the new controller integration APIs, the Job API now features an explicit .spec.scheduling field, so you can declare how a Job should be scheduled instead of relying on the Job controller to infer it from the Job's shape. This expands support well beyond static, indexed, and fully-parallel Jobs.

.spec.scheduling is composed of the building blocks described above:

• schedulingPolicy — basic for standard Pod-by-Pod scheduling, or gang for all-or-nothing scheduling.
• schedulingConstraints — the topology domain the Job's Pods must be co-located within.
• disruptionMode — whether the Job's Pods can be preempted individually (single) or only as a whole (all).
• resourceClaims — the ResourceClaims shared by all of the Job's Pods.

For example:

apiVersion: batch/v1
kind: Job
metadata:
 name: distributed-training-job
 annotations:
 kubernetes.io/description: "Distributed Job using explicit WAS scheduling with gang policy and zone topology constraints."
spec:
 parallelism: 8
 completions: 8
 scheduling:
 schedulingPolicy:
 gang: {} # minCount omitted → defaults to parallelism (8)
 schedulingConstraints:
 topology:
 - key: topology.kubernetes.io/zone
 disruptionMode:
 all: {}
 template:
 spec:
 containers:
 ...

Omitting .spec.scheduling, or omitting schedulingPolicy within it, selects the basic policy, which behaves exactly like standard Job scheduling today.

For every Job it manages, the controller compiles this configuration into a Workload and a PodGroup owned by the Job, and sets .spec.schedulingGroup.podGroupName on each Pod it creates so the scheduler treats them as one group. Once created, .spec.scheduling is immutable, with one exception: schedulingPolicy.gang.minCount can be updated, which lets you resize a running gang.

DRA ResourceClaim support for workloads

As the core WAS APIs mature, so do their integrations with Dynamic Resource Allocation (DRA). Kubernetes v1.36 introduced the DRAWorkloadResourceClaims feature gate. The associated feature allows ResourceClaims to be replicated and reserved for entire PodGroups and shared by all their member Pods:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
 name: training-job-workers-pg
spec:
 ...
 resourceClaims:
 - name: pg-claim
 resourceClaimTemplateName: my-claim-template
---
apiVersion: v1
kind: Pod
metadata:
 name: topology-aware-workers-pg-pod
spec:
 ...
 schedulingGroup:
 podGroupName: training-job-workers-pg
 resourceClaims:
 - name: pg-claim
 resourceClaimTemplateName: my-claim-template

In Kubernetes v1.37, the DRAWorkloadResourceClaims feature gate graduated to Beta.

While the API and core functionality of the feature remain unchanged, one change eliminates some potentially surprising behavior when disabling the feature. Previously when one of a Pod's spec.resourceClaims referenced a ResourceClaimTemplate and matched one of its PodGroup's spec.resourceClaims and the DRAWorkloadResourceClaims feature gate was disabled, a ResourceClaim was created for the Pod instead of the PodGroup. In that scenario in v1.37, no ResourceClaim is created at all. This change prevents Kubernetes from creating a flood of ResourceClaims from a ResourceClaimTemplate and potentially exhausting DRA resources when a claim intended to be shared by a whole PodGroup is replicated for each and every Pod in the group.

For more information, see the feature documentation.

What's next?

The Workload-Aware Scheduling Working Group (WG WAS) is currently finalizing its plans for the Kubernetes v1.38 release cycle. While the roadmap is still taking shape (stay tuned!), the following key initiatives are already planned:

• Graduation of Workload and PodGroup APIs to GA: Solidifying the core foundation of workload-aware scheduling as a stable Kubernetes API.
• Graduation of Topology-Aware Scheduling (TAS) and CompositePodGroup (CPG) to Beta: Bringing these advanced placement and hierarchical scheduling features to Beta stability.
• Graduation of controller integration building blocks to Beta: Further refining the integration APIs to ensure a robust developer experience.
• Increased adoption and integration: Expanding the ecosystem by integrating workload-aware scheduling with other controllers, with a particular focus on hierarchical orchestrators such as JobSet.
• Kueue Integration: Fostering closer alignment between WAS and Kueue. In the near term, we aim to ensure Kueue is fully aware of WAS features for seamless interoperability. In the long term, we envision Kueue leveraging WAS as its underlying engine for capabilities like gang-scheduling and topology-aware placement.

Getting started

Many of the workload-aware scheduling improvements are now available as Beta features in v1.37, while new advanced capabilities are introduced in Alpha. Both Beta and Alpha features here are disabled by default and require manual enablement.

Beta features:

• Workload API, gang scheduling, and preemption: The GenericWorkload feature gate (which now integrates gang scheduling and workload-aware preemption) is Beta and disabled by default on the kube-apiserver, kube-controller-manager and kube-scheduler. Ensure your manifests are updated to use the scheduling.k8s.io/v1beta1 API group.
• DRA ResourceClaim support for workloads: Enable the DRAWorkloadResourceClaims feature gate on the kube-apiserver, kube-controller-manager, kube-scheduler and kubelet.

Alpha features:

• Topology-aware scheduling: Enable the TopologyAwareWorkloadScheduling feature gate on the kube-apiserver and kube-scheduler.

• CompositePodGroup API: Enable the CompositePodGroup feature gate on the kube-apiserver, kube-controller-manager and kube-scheduler, and ensure the scheduling.k8s.io/v1alpha3 API version is enabled. Note that enabling CompositePodGroup on the kube-controller-manager also requires the TopologyAwareWorkloadScheduling feature gate to be enabled.

• Workload API integration with the Job controller: Enable the WorkloadWithJob feature gate on the kube-apiserver and kube-controller-manager.

• PodGroup preemptionPolicy: Enable the PodGroupPreemptionPolicy feature gate on the kube-apiserver and kube-scheduler.

Controller integration APIs:

The new workloadbuilder library is available to developers building both out-of-tree and in-tree controllers who want to integrate with WAS. It does not require a feature gate. You can explore the library and find usage examples directly in the kubernetes/component-helpers repository.

We encourage you to try out workload-aware scheduling in your test clusters and share your experiences to help shape the future of Kubernetes scheduling. You can send your feedback by:

• Reaching out via Slack (#wg-workload-aware-scheduling).
• Joining the WG Workload-Aware Scheduling or SIG Scheduling meetings.
• Filing a new issue in the Kubernetes repository.

Learn more

To dive deeper into the architecture and design of these features, read the KEPs:

• KEP-4671: Gang Scheduling Support in Kubernetes
• KEP-5710: Workload-aware preemption
• KEP-5732: Topology-aware workload scheduling
• KEP-6012: CompositePodGroup API
• KEP-6089: WAS: Controller Integration APIs
• KEP-5547: WAS: Integrate Workload APIs with Job controller
• KEP-5729: DRA: ResourceClaim Support for Workloads

Kubernetes v1.37 promotes the KubeletInUserNamespace feature gate to beta. With this feature enabled, all of the node components (kubelet, CRI and OCI runtimes, CNI plugins, and kube-proxy) can run as a non-root user on the host, using a Linux user namespace. This technique is also known as rootless mode. The work started as an experiment in 2018, and was merged into Kubernetes v1.22 (2021) as an alpha feature (Kubernetes Enhancement Proposal KEP-2033).

This feature should not be confused with user namespaces for pods (hostUsers: false with the UserNamespacesSupport feature gate, GA since v1.36), which puts pods in user namespaces but still runs the node components as root. These two features do not conflict. Moreover, they can be combined to nest Kubernetes inside Kubernetes without resorting to the full privileged: true.

Why run the node components in a user namespace?

Because the node components have historically had container-breakout vulnerabilities that could compromise full root privileges on the host.

Examples of such vulnerabilities include:

• CVE-2022-0811 ("cr8escape"): CRI-O could be tricked into setting arbitrary sysctls, such as kernel.core_pattern, resulting in arbitrary code execution as root on the host
• CVE-2023-27561: runc could be tricked into bypassing the masked paths of a container via a volume mount race, exposing the host's procfs files (a regression of CVE-2019-19921)
• CVE-2024-10220: the kubelet could be made to execute arbitrary commands as root via gitRepo volumes (gitRepo volumes had a similar vulnerability, CVE-2018-11235, back in 2018 too)
• CVE-2025-31133: runc could be tricked into bind-mounting attacker-controlled paths and writing to the host's procfs files, such as /proc/sysrq-trigger and /proc/sys/kernel/core_pattern
• CVE-2026-53488: containerd could be tricked into executing arbitrary commands on the host, via crafted labels in a container image

By running the node components in a user namespace, the potential damage is confined to the non-root user's account. Notably, an attacker cannot conceal their intrusion by modifying the kernel, the boot loader, or the firmware.

It should still be noted that user namespaces are not effective for mitigating vulnerabilities in the kernel itself. User namespaces should be used in conjunction with traditional hardening measures such as seccomp to prevent containers from invoking unnecessary system calls.

Use cases

• Production clusters: mitigate potential container-breakout vulnerabilities.
• Shared machines (e.g., HPC): users can deploy Kubernetes without asking the machine administrator for root privileges, and without the risk of accidentally breaking other users' environments.
• Laptops: prevent a local cluster from accidentally breaking the host system configuration, e.g., the host iptables rules used for VPNs.
• AI sandbox: a Kubernetes application developer may create a dedicated local user account for running an AI coding agent and a test Kubernetes cluster. This setup is useful for preventing the AI agent from breaking the host when it is deceived by malicious information on the Internet.
• Kubernetes-in-Kubernetes: a nested cluster can run inside a parent cluster as a user-namespaced pod (hostUsers: false), isolating workloads more strictly than Kubernetes API namespaces do.
• Bootstrapping: a temporary unprivileged cluster can be used to bootstrap an actual cluster, e.g., with Cluster API.

How does it work?

A Linux kernel user namespace maps a host level non-root user (e.g., UID 1000) to a fake root user inside the namespace. The UID 0 privileges are limited to the inside of the namespace. The fake root is enough for most of the node components' tasks: mounting volumes, creating cgroups, and configuring the network namespaces of pods. It still comes with some caveats that may break compatibility with specific CNI and CSI drivers, though.

The user namespace has to be created outside of Kubernetes. For example, Rootless Docker can be used to prepare the user namespace in which Kubernetes runs.

The KubeletInUserNamespace feature gate itself is quite "boring": basically it just lets the kubelet ignore permission errors that occur when setting some sysctl values (e.g., vm.overcommit_memory and kernel.panic) and when watching kernel messages via /dev/kmsg.

See Running Kubernetes Node Components as a Non-root User for further information.

What changed from Alpha to Beta?

• The KubeletInUserNamespace feature gate is now enabled by default. Enabling the gate does not put the kubelet into a user namespace automatically, so nothing changes for existing "rootful" clusters.
• kubectl get nodes -o yaml now reports whether nodes are running in a user namespace via the runningInUserNamespace property. A cluster administrator can use this property to set node labels or taints, to avoid scheduling workloads that need real root privileges (e.g., some CNI plugin installers) onto rootless nodes.
• For Kubernetes' own CI/CD testing, the node conformance end to end tests now run on a rootless cluster (ci-kubernetes-e2e-kind-rootless).

Several related improvements have also happened outside the promotion of the feature gate itself:

• Linux kernel v6.3 (2023): added support for idmapped tmpfs.
• Kubernetes v1.33 (2025): enabled the UserNamespacesSupport feature gate by default, allowing user-namespaced pods (hostUsers: false) to be created without extra configuration.
• containerd v2.1 (2025): added support for writable cgroups.

With these improvements, a Kubernetes cluster with KubeletInUserNamespace can now also be nested inside Kubernetes pods with hostUsers: false (UserNamespacesSupport).

How to use it

kind

The easiest way is to use kind (a Kubernetes SIG Testing project) to run a Kubernetes cluster in rootless Docker, rootless nerdctl, or rootless Podman:

# Example using Docker
dockerd-rootless-setuptool.sh install
kind create cluster

Depending on the host configuration, you may need additional configuration for systemd, kernel modules, sysctl, etc.

See the Docker documentation and the kind documentation for further information.

minikube

minikube (a Kubernetes SIG Cluster Lifecycle project) also supports running a Kubernetes cluster in rootless Docker or rootless Podman:

dockerd-rootless-setuptool.sh install
minikube start --driver=docker

See the minikube documentation for further information.

Usernetes

Usernetes (a third-party project) is a distribution of rootless Kubernetes, maintained by the author of this article. The project began in 2018, and it is where the KubeletInUserNamespace feature gate originally came from.

Unlike kind and minikube, Usernetes supports creating a cluster with multiple rootless Docker / Podman / nerdctl nodes, connected using VXLAN via the Flannel CNI plugin.

Usernetes also experimentally supports a Kubernetes-in-Kubernetes mode.

k3s

k3s (a CNCF Sandbox project) also supports rootless mode. Unlike kind, minikube, and the current generation of Usernetes, rootless k3s does not rely on an external runtime such as rootless Docker.

What's next?

Depending on feedback and adoption, the Kubernetes project plans to graduate this feature to General Availability (GA) in a future release. If you have feedback on this feature, please open an issue in the kubernetes/kubernetes repository.

The project is also discussing several Kubernetes Enhancement Proposals that may contribute to simplifying Kubernetes-in-Kubernetes with this feature:

• KEP-5474: Enable Writable cgroups for unprivileged containers
• KEP-5714: Allow specifying whether to unshare cgroup namespaces

Getting involved

We always welcome new contributors. If you would like to get involved, you can join the Node Special Interest Group (SIG Node).

If you would like to share feedback, you can do so on our public Slack channel (visit https://slack.k8s.io/ for an invitation if you need one).

Special thanks to everyone who helped design and implement this feature, including but not limited to (in alphabetical order):

• Bing Hongtao (HirazawaUi)
• Jordan Liggitt (liggitt)
• Sergey Kanzhelev (SergeyKanzhelev)
• Tim Hockin (thockin)

Kubernetes 1.37 is here and Dynamic Resource Allocation (DRA) keeps pushing past where it started! This release brings DRA Extended Resource support to GA, a milestone the team has been building toward for three straight releases. Several more features graduate to Beta or GA. A fresh batch of alpha features rounds out the release.

I'll dive into what's new for DRA in Kubernetes 1.37!

What's stable in 1.37

DRA Extended Resource support has graduated to GA. This is the mechanism that lets DRA drivers satisfy requests made through the traditional extended resource API, think example.com/gpu in a Pod spec, without requiring a separate device plugin alongside the DRA driver. An extended resource name can be set directly on a DeviceClass, and Pods requesting it get matched to a device through DRA with no ResourceClaim needed on the workload's part.

It's been on a steady path since KEP acceptance in 1.34. Alpha landed in 1.35, Beta in 1.36, and now it's Stable. For cluster operators, this is what makes DRA adoption gradual. Existing workloads written against extended resources keep working unmodified while the backend allocation logic moves over to DRA.

ResourceClaims status with possible standardized network interface data adds a devices field to ResourceClaim .status, letting DRA drivers report per-device status, including, for network devices, the interface name, MAC address, and IP addresses. This gives users and controllers visibility into device state that was previously invisible once a device was configured in a Pod, and makes it possible to build things like network services that rely on a device's reported IPs.

DRA: device taints and tolerations is now Stable; DRA drivers can mark devices as tainted so they're skipped for new Pod scheduling, and cluster admins can apply the same taints cluster-wide via a DeviceTaintRule, without reconfiguring drivers. Pods already using a tainted device can be evicted automatically, unless their ResourceClaim explicitly tolerates the taint. This mirrors node taints and tolerations, letting operators take a single device offline for maintenance or mark it degraded, without disrupting the rest of the cluster.

Standard numaNode device attribute standardizes resource.kubernetes.io/numaNode as a shared attribute name, so devices from different drivers can be compared on the same NUMA node instead of each driver inventing its own name for it. It landed directly as stable in 1.37, since it's a naming/registration KEP with no feature gate or in-tree behavior change.

Feature promoted to Beta

ResourceClaim support for workloads graduates to Beta behind the DRAWorkloadResourceClaims feature gate, which stays disabled by default. In a cluster that has the feature enabled, Workloads and PodGroups can reference ResourceClaims directly, so a single claim can be shared across an entire group of Pods. This is instead of claims being capped at 256 Pods through the old per-Pod reservation limit.

The DRA Device Attributes Downward API is aimed at supporting device injection into KubeVirt VMs. Drivers populate a Metadata field when preparing a claim, and the framework writes it to a JSON file mounted into the container via CDI, letting workloads read a device's PCI bus address, MAC address, and other attributes directly instead of requiring custom controllers to watch and translate ResourceClaims and ResourceSlices.

Alpha features

List types for attributes moved into a second Alpha in 1.37, letting a device attribute hold more than one value instead of a single scalar, such as a CPU that's adjacent to more than one PCIe root. This makes it possible to match or distinguish devices based on overlapping or non-overlapping sets of values, while single-value attributes keep working as they do today.

Node allocatable resource requests moved into Alpha 2. It lets the scheduler and kubelet treat DRA-managed CPU, memory, and similar node resources the same way they treat ordinary resource requests, so a node doesn't get oversubscribed and users no longer have to duplicate the same request in both a ResourceClaim and the pod spec.

Resource availability visibility moved to a second Alpha in Kubernetes 1.37. Users create a ResourcePoolStatusRequest to get a point-in-time availability snapshot. To refresh it, delete and recreate the request; it is not a continuous monitoring API.

DRA: Optional Node Operations lets a driver skip kubelet's prepare and unprepare calls for allocations that don't need any setup on the node. This makes it possible to avoid an unnecessary dependency on the driver for allocations where there's genuinely nothing for it to do locally.

Derived Attributes is a new feature that lets you use CEL expressions to match up devices based on your own custom rules. Before this, pairing devices from different vendors (like a GPU/TPU and a NIC on the same NUMA node) only worked if both drivers used the exact same attribute name. If one used numa and the other used numaNode, the scheduler couldn't pair them together. Now, you can easily bridge these differences yourself inside your manifest, meaning you don't have to wait for hardware vendors to agree on standardized attribute names. Beyond just fixing naming differences, you can also use CEL to handle more complex scenarios like slicing a specific ID out of a long, monolithic topology string, or grouping devices into custom performance tiers based on their available capacity.

DRA Device Compatibility Groups lets drivers tag partitions of a device, like MIG vs vGPU profiles on the same GPU, with compatibility groups, so the scheduler rejects incompatible combinations up front instead of the driver failing at node preparation time. It's controlled by the DRADeviceCompatibilityGroups feature gate, disabled by default.

PreQueueingHint extension point is new as Alpha in 1.37. DRA ResourceClaim events used to trigger a full scan of every unschedulable pod, an O(N²) cost during large scale-ups. The DRA plugin now uses a pod informer index to narrow that to just the pods actually affected, cutting the requeue path to O(1) and roughly doubling scheduling throughput in early benchmarks. Controlled by the SchedulerPreQueueingHints feature gate.

DRA Consumable Capacity now supports fractional values in CapacityRequestPolicyRange, enabling more precise capacity requests and allocation for devices with fractional resources. This improves flexibility for workloads that require fine-grained resource allocation. The enhancement is gated by the DRAFractionalCapacityRange feature gate, which is in Beta in 1.37.

What’s next

DRA continues to mature with every release. Several features currently in Alpha and Beta are on track to progress in the coming releases, and the community keeps working on DRA's performance, scalability, and reliability. Expect another ambitious set of DRA features in Kubernetes 1.38.

Getting involved

A good starting point is joining the WG Device Management Slack channel and meetings which happens at US/EU and EU/APAC friendly time slots.

Not all enhancement ideas are tracked as issues yet, so come talk to us if you want to help or have some ideas yourself! We have work to do at all levels, from difficult core changes to usability enhancements in kubectl which could be picked up by newcomers.

Acknowledgments

The following KEP owners added or promoted a feature in the 1.37 release (in alphabetic order):

• Alay Patel (alaypatel07)
• Byonggon Chun(bg-chun)
• Gaurav Ghildiyal (gauravkghildiyal)
• Jiefeng Xu (jiefeng-xu)
• John A. Hull (johnahull)
• Jon Huhn (nojnhuh)
• Lionel Jouin (LionelJouin)
• Patrick Ohly (pohly)
• Praveen Krishna (pravk03)
• Shingo Omura (everpeace)
• Troy Chiu (troychiu)

This would not have been possible without the help of the reviewers and approvers. So a huge thanks to everyone else who helped shape this release, in ways big and small. Given enough eyeballs, all bugs are shallow and this release had plenty of them, watching closely and caring enough to make things better. DRA got better this cycle because of all of you.

Kubernetes v1.37 includes API support for horizontal autoscaling of workloads down to zero replicas. This feature is now Beta and enabled by default. A HorizontalPodAutoscaler (HPA) that uses a suitable object metric or external metric can now scale a workload to zero replicas, then bring it back when the metric changes.

Before v1.37, you needed an add-on or external component, or you had to enable the Alpha feature gate, to scale from zero. It is now part of core Kubernetes.

Scaling to zero removes the last idle Pod from workloads such as queue consumers and batch processors. The savings are largest when each Pod reserves expensive resources, including dedicated CPUs or GPUs.

The trade-off is cold-start time: the HPA must observe the metric, schedule a Pod, and start the application. This works well when work can wait in a durable queue.

Kubernetes Services do not buffer requests while no Pods are ready, so HTTP and other request-driven workloads need a separate buffering layer.

Why scaling from zero needs a different metric

The HPA commonly scales on CPU or memory usage. Both metrics come from running Pods. Once the replica count reaches zero, there are no Pods left to measure and no signal that can tell the HPA to scale back up.

Object and external metrics do not have that limitation. A queue length, for example, exists independently of the workers that consume it. The HPA can continue reading the queue length while no workers are running.

The following example scales a queue consumer to and from zero using an external metric.

Configure an external metric

The following example uses a Prometheus metric named queue_consumer_lag. It assumes that Prometheus already collects a series similar to this one:

queue_consumer_lag{namespace="default",name="worker_tasks"}

Kubernetes needs a metrics adapter to make that value available through the External Metrics API. One implementation is the Prometheus Adapter, which can expose the series using an externalRules entry:

externalRules:
- seriesQuery: '{__name__="queue_consumer_lag",name!=""}'
 metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (name)
 resources:
 overrides:
 namespace:
 resource: namespace

The exact adapter installation and discovery rules depend on your monitoring setup. See the Prometheus Adapter guide to external metrics for the full configuration options.

Before creating the HPA, you can verify that Kubernetes can read the metric:

kubectl get --raw \
 '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'

The request should return the current value for worker_tasks. If it does not, fix the metrics pipeline before configuring the HPA. An HPA cannot scale from zero when its metric is unavailable.

Configure the HPA

The following HPA targets a Deployment named queue-worker. It allows between zero and ten replicas, with one replica requested for each 30 queued tasks:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: queue-worker
 annotations:
 kubernetes.io/description: "Scales queue-worker based on the number of queued tasks"
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: queue-worker
 minReplicas: 0
 maxReplicas: 10
 metrics:
 - type: External
 external:
 metric:
 name: queue_consumer_lag
 selector:
 matchLabels:
 name: worker_tasks
 target:
 type: Value
 value: "30"

When the queue is empty, the HPA can reduce the Deployment to zero replicas. When tasks arrive, the external metric remains available and the HPA calculates a new replica count, capped at ten by maxReplicas.

Start the Deployment with at least one replica. Manually setting a Deployment to zero has always paused autoscaling. The HPA preserves that behavior and will not wake a workload that it did not scale down itself.

Normal HPA behavior still applies. In particular, the default downscale stabilization window is five minutes. The window prevents a short drop in queue length from immediately removing all workers. You can configure the window through spec.behavior.scaleDown if your workload needs different behavior.

How the HPA distinguishes zero from paused

Scaling from zero creates an ambiguity. A replica count of zero can mean that the HPA scaled the workload down, or that an operator manually paused it.

The controller resolves this with a ScaledToZero status condition. When the HPA scales a workload from one or more replicas to zero, it records ScaledToZero=True. The condition tells later reconciliation loops that the controller owns the zero state and should continue evaluating object or external metrics.

After scaling the workload back up, the controller changes the condition to ScaledToZero=False with the reason NotScaledToZero. A workload at zero without the ScaledToZero=True condition remains paused.

You can inspect the conditions with:

kubectl describe hpa queue-worker

If the adapter cannot return the configured metric, the HPA reports ScalingActive=False with a reason such as FailedGetExternalMetric. Restore the metric or manually scale the workload to recover capacity.

Before upgrading or rolling back

In Kubernetes v1.37, the HPAScaleToZero feature gate is enabled by default on both the kube-apiserver and kube-controller-manager. The API server accepts minReplicas: 0; the controller manager performs the condition-based scaling.

During a version-skewed control plane upgrade, wait until both components support the feature and have it enabled before creating HPAs with minReplicas: 0. A controller manager with the feature disabled treats replicas: 0 as a manual pause and may leave a workload at zero.

Before disabling the feature gate or downgrading to a version without the condition-based implementation:

• Change affected HPAs to minReplicas: 1 or higher.
• Scale any workload currently at zero to at least one replica.

minReplicas: 0 also requires at least one object or external metric. The API server rejects an HPA that only contains resource metrics such as CPU or memory.

From Alpha to Beta

The first Alpha implementation shipped in Kubernetes v1.16. Kubernetes v1.36 added the ScaledToZero condition and the controller behavior needed to distinguish an automatic scale-down from a manual pause.

Kubernetes v1.37 enables the feature by default after adding integration and end-to-end coverage for scaling down to zero and back up from an external metric. The next step is to gather operational feedback before considering graduation to GA.

How can I learn more?

• Read the documentation for scaling to and from zero.
• Read KEP-2021: HPA supports scaling to and from zero pods for object and external metrics.
• Learn how to configure the Prometheus Adapter for external metrics.

How to get involved

This feature is owned by SIG Autoscaling. Join Kubernetes Slack and the #sig-autoscaling channel to share feedback from Beta usage.

Acknowledgements

Thanks to the SIG Autoscaling contributors who took this feature from the original v1.16 implementation to the condition-based redesign and Beta graduation. Thanks also to Guy Templeton and Adrian Moisey for reviewing the KEP, and to the release, documentation, and production-readiness reviewers who helped prepare it for Kubernetes v1.37.

I am excited to announce that etcd RangeStream is graduating to beta in Kubernetes v1.37. Paired with etcd v3.7, it reduces the memory the API server and etcd need to read a large collection, and makes peak usage more predictable.

The cost of large reads

The API server serves most list and watch requests from its in-memory watch cache. Populating that cache requires reading a resource's full state from etcd, at startup and on every re-initialization. For a resource with many objects, or large ones, such as Pods, that read is expensive.

The API server already paginated these reads, asking etcd for a fixed number of keys at a time rather than the whole collection at once. But a page bounded by key count has no awareness of object size, so a page of large objects can still be very large. That makes memory usage hard to predict, and a bad combination of object size and concurrent reads can be enough to trigger an OOM. etcd's unary Range assembles each page in full before sending it, and the API server holds it while decoding, so the same payload sits in memory on both sides at once. Most of that cost lands on etcd, which is also where streaming helps most.

Streaming reads with RangeStream

etcd v3.7 adds a streaming version of that read, the RangeStream RPC. It takes the same RangeRequest as Range and returns the same result set, but instead of building the whole response up front, etcd splits it into chunks and streams them. Chunk size is tuned adaptively to the values being returned, so a collection of large objects is bounded by bytes rather than by a key count, and memory is freed as the stream progresses instead of being held until a whole page is assembled.

When the feature is enabled, the API server uses RangeStream wherever it reads a whole collection out of etcd. This includes watch cache initialization, and the fallback paths where a list request cannot be served from the cache and reads etcd directly. In either case the API server decodes each chunk as it arrives and releases it before pulling the next one, so neither side ever holds the whole collection.

Requirements

• Kubernetes v1.37 or later
• etcd v3.7 or later

RangeStream is used when the EtcdRangeStream feature gate is enabled on the kube-apiserver, which is beta and on by default in v1.37, and etcd is v3.7 or later. The API server resolves etcd's support at startup and also falls back at runtime if a call returns Unimplemented, so an API server paired with an older etcd keeps using the paginated Range path on its own. To turn it off, disable the gate:

--feature-gates=EtcdRangeStream=false

Confirming RangeStream is in use

The API server records streamed reads under their own operation label on its etcd metrics. A non-zero count here means RangeStream is in use:

etcd_request_duration_seconds_count{operation="listStream"}

If it stays at zero, the API server is still using the paginated Range path, most likely because etcd is older than v3.7.

Learn more

• KEP-5966: etcd RangeStream
• RangeStream in the etcd API guide
• Announcing etcd v3.7
• SIG etcd

If you have questions or feedback, join the #sig-etcd channel on Kubernetes Slack.

I am excited that storage version migration (SVM) has graduated to General Availability (GA) in Kubernetes v1.37!

After a number of releases of work and testing, the built-in StorageVersionMigration API (storagemigration.k8s.io/v1) and control plane controller are now fully stable and enabled by default across all v1.37 Kubernetes clusters.

The problem with stale storage versions

In Kubernetes, stored API resources are written using a specific storage version (schema representation). The way Kubernetes interacts with object storage fundamentally requires mutation of a resource in order to ensure that the latest storage version is used for all resources. This creates problems when you want to change the storage version of a resource.

One example of a scenario where you may want to change the storage version of a resource is when you are promoting a CRD to drop an older API version (such as v1alpha1) to a newer version (leaving just v1beta1 and v1). It's a problem to drop the older API version whilst there are still resources stored with the old alpha version.

To avoid problems, you designate v1 as the new storage version; but, on it's own, that's not enough. While new writes are stored as v1, any existing resource could remain stored as v1alpha1 or v1beta1 in storage. You cannot safely remove v1alpha1 from the CRD's .status.storedVersions or drop serving support until every single resource in storage has been re-written to not be serialized and stored with the alpha version.

Another relevant example is encryption at rest and, related, key rotation. When you configure encryption at rest or rotate encryption keys, existing resources in storage remain unencrypted (or encrypted under old keys) until they are actively re-written through the Kubernetes API server.

Historically, cluster administrators and CRD authors had to rely on manual kubectl get / kubectl replace scripts, or to deploy the out-of-tree kube-storage-version-migrator component to force re-writes. These approaches were often tedious, error-prone, and difficult to monitor.

How storage version migration works

Initiating a storage version migration is as simple as creating a declarative StorageVersionMigration object. The built-in StorageVersionMigrator controller in the Kubernetes control plane watches for these objects, and automatically migrates existing resources to the default storage version for that API.

Example: Migrating a custom resource API

Suppose you have updated a CustomResourceDefinition (crontabs.example.com) to use v1 as its storage version. To migrate all existing stored resources off older versions, create a StorageVersionMigration:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
 name: crontabs-migration
spec:
 resource:
 group: example.com
 resource: crontabs

Apply the manifest using kubectl:

kubectl apply -f crontabs-migration.yaml

Monitoring and verifying migrations

The StorageVersionMigrator controller updates the status of the StorageVersionMigration object as migration progresses. You can inspect the migration status using kubectl:

kubectl get storageversionmigration.storagemigration.k8s.io/crontabs-migration -o yaml

A successful migration will report a Succeeded condition set to True:

status:
 conditions:
 - type: Running
 status: "False"
 lastUpdateTime: "2026-08-02T10:05:00Z"
 reason: StorageVersionMigrationInProgress
 - type: Succeeded
 status: "True"
 lastUpdateTime: "2026-08-02T10:05:00Z"
 reason: StorageVersionMigrationSucceeded

Once the migration has succeeded, you can be confident that all instances of the resource in storage are stored in the current storage version. For CRDs, the stored version should be updated in the CRD's .status.storedVersions to only contain the preferred version. If the .status.storedVersions is not updated following a successful migration then that means that the CRD was updated during the migration. In that case, the migration should be retried in order to safely deprecate an older storage version.

Including migrations in your CRD manifests

Because StorageVersionMigration is a standard declarative Kubernetes API, CRD authors can bundle or trigger migrations directly alongside CRD upgrades. For example, you can include the migration in the same manifest as your updated CustomResourceDefinition:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
 name: crontabs.example.com
spec:
 group: example.com
 # Updated versions list where v1 has storage: true
 ...
---
apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
 name: crontabs-migration
spec:
 resource:
 group: example.com
 resource: crontabs

What's next?

• Learn more about the concepts behind Storage Versions.
• Read the step-by-step task guide: Migrate Kubernetes Objects Using Storage Version Migration.

SIG API Machinery would love to hear your feedback as you adopt built-in Storage Version Migration in your clusters. Reach out to us on the #sig-api-machinery Slack channel or participate in our community discussions!

Kubernetes brings a wealth of features that make it easy to run your production workloads securely and reliably. While aspects like scheduling, health checks and resource limits are probably at the front of your mind, one other important feature of Kubernetes is production identity — how your workload can authenticate to other systems in order to do its job.

Up until now, the primary production identity mechanism built into Kubernetes has been service account JWTs (JSON Web Tokens). These are cryptographically-signed tokens, issued by the control plane of your cluster, that let anyone in the world understand who is calling when your workload uses them.

In Kubernetes 1.37, the foundations of a new built-in production identity technology have gone GA. Pod Certificates (and the closely-associated Cluster Trust Bundles) build X.509 certificate issuance for TLS and mTLS directly into core Kubernetes.

Why?

Service account JWTs have a lot going for them:

• They are built directly into Kubelet, and work pretty magically. They are written to your workload container’s filesystem before your workload starts up, and automatically kept up to date.
• The issuance system follows least-privilege principles; the node restriction admission plugin ensures that tokens can only be requested by the Kubelet that is actually currently running your pod.
• They can be federated, allowing you to use them to authenticate to other systems outside of Kubernetes. Service account JWTs underpin the pod-to-cloud authentication store for all of the largest cloud providers, and have widespread support across many additional services and software packages. If it can understand JWTs, you can authenticate to it with a service account token.

However, service account JWTs have one big downside — they are bearer tokens. With bearer tokens, if you have the token, then you are the identity asserted by the token. And since you necessarily have to hand copies of the JWT to all your peers in order to authenticate to them, they can be you, too.

There are partial mitigations for this, and service account tokens make use of them (time-, object-, and audience-binding), but none are complete defences.

A solution to this problem lies in proof-of-possession credentials, where you don’t send your entire credential to your peer, but only a proof that you possess the credential. In practice, these schemes are always built on asymmetric cryptographic signatures (RSA, ECDSA, and friends).

There are few different standard approaches, such as request signing (AWS SigV4, JWT DPoP, RFC 9421), but the most widely-deployed and understood solution is X.509 certificates, as used in TLS. In TLS, your credential is split into two pieces

• A private key, which for maximum security should be generated within your workload (or within a hardware security module), and never leave.
• A certificate, which is a description of your identity and public key, signed by a Certificate Authority.

The goal of Pod Certificates is to make using X.509 certificates from your Kubernetes workload just as easy as using service account JWTs, while maintaining Kubernetes’ high security bar. I think we’ve hit this target.

As I’ll cover in the architecture and example sections below, there are many similarities between the design of service account JWT issuance and Pod Certificates. One significant place they diverge, however, is that Pod Certificates is a much more flexible mechanism. Kubernetes only offers one flavor of service account JWTs, with standardized claims.

The X.509 ecosystem is significantly more varied than the JWT ecosystem, and X.509 certificates used for different purposes contain different extensions and information. For this reason, Pod Certificates has common machinery built into Kubelet, but offers a pluggable interface so that many different types of certificates can be issued within a single cluster, at the same time.

In the fullness of time, I expect Kubernetes to offer at least two built-in certificate providers:

• One that issues server TLS certificates for the DNS names used by Kubernetes services.
• One that offers SPIFFE client certificates, filling the same role that service account JWTs fill today.

In the remainder of this article, I’ll take you through the overall architecture of a Kubernetes workload using Pod Certificates, as well as give you an example of installing and using a real (toy) Pod Certificates signer controller.

Architecture

When you use Pod Certificates and Cluster Trust Bundles, there are the following major components:

• Your application, which requests certificates in its pod spec, and reads the keys, certificates and trust bundles from the container filesystem to use for (m)TLS.
• Kubelet, which issues PodCertificateRequest objects and reads ClusterTrustBundle objects on behalf of your workload.
• The signer controller, which answers PodCertificateRequests and publishes ClusterTrustBundles.

The best way to get a sense of what these components each do is to follow the issuance process chronologically:

1. Once your application pod is scheduled to a node, Kubelet identifies all of the podCertificate and clusterTrustBundle projected volumes sources in its spec.
2. For each podCertificate source:
   1. Kubelet generates a new private key according to the keyType field.
   2. Kubelet creates a PodCertificateRequest addressed to the signer named in the source.
   3. The signer controller sees the PodCertificateRequest and decides whether or not to issue the certificate.
   4. The signer controller issues the certificate by filling out the status.certificateChain field.
   5. The signer controller also fills out the status.beginRefreshAt field to instruct Kubelet when it should begin trying to refresh the certificate.
      certificate to the container filesystem.
3. For each clusterTrustBundle source:
   6) Kubelet retrieves the issued certificate, and writes the private key and
   1. Kubelet collects all the ClusterTrustBundles that match the signer name
   2. Kubelet unifies all of the certificates from all matching ClusterTrustBundles, and (stably) reorders them (to prevent applications from accidentally depending on a particular ordering).
      and label selectors in the source.
   3. Kubelet writes the certificates to the file path named in the source.
      and trust anchors from the filesystem.
4. Your application pod starts up, and the application reads keys, certificates,
5. Kubelet periodically updates the files from clusterTrustBundle sources as the contents of the selected ClusterTrustBundles changes. The application must pick up the changes using inotify or polling.
6. As each certificate’s beginRefreshAt time passes, Kubelet repeats the process in step 2 to refresh the certificates, and write the update private keys and certificate chains to the filesystem. As in step 5, the application must pick up changes using inotify or polling.

Some key takeaways:

• Automatic rotation is built in. Applications must properly handle it. Any signers eventually shipped in core Kubernetes will issue certificates with a max lifetime of 24 hours. The maximum lifetime allowed for other signers is 91 days.
• To make automatic rotation support as simple as possible, Kubelet supports writing the private key and certificate chain to a single file (a credential bundle) This allows the application to simply subscribe to inotify events for (or poll) the single file, read the contents, and use them. Kubelet does support writing the private key and certificate chain to separate files, but then the application needs to carefully manage the potential race conditions of reading the files mid-rotation.
• Wherever possible, security checks are built into kube-apiserver, rather than burdening signer or application developers. As an example, the built-in node restriction admission plugin enforces node isolation, ensuring that one compromised node cannot spread access by requesting certificates for pods that aren’t scheduled to it.

Try it out

Because the Kubernetes project does not yet ship any Pod Certificate signers in core, in order to try these features out, you will need to install a third-party signer into your cluster. To make this easier, I have written Tinycert, which you can install into your cluster (or a Kind cluster).

Tinycert is not a full production solution, but it’s a good starting point for experimenting with Pod Certificates, as well as a base for creating your own signers.

Tinycert provides:

• The ahmedtd.github.io/tinycert-service signer, which issues certificates with DNS SANs for all of the Kubernetes Services your Pod is part of.
• The ahmedtd.github.io/tinycert-spiffe signer, which issues SPIFFE-compatible certificates that identify the namespace and service account of your Pod. These can be used as both client and (with effort) server certificates.
• A Go library, github.com/ahmedtd/tinycert/lib/spiffefsd to help your applications load SPIFFE certificates and trust bundles from a SPIFFE Filesystem Delivery (Draft Standard) folder, as well as configure the Go TLS library for proper client and server authentication.
• An example of a SPIFFE client and server application communicating using mutual TLS and SPIFFE certificates.

What next?

• Take a look at the documentation for Pod Certificates and Cluster Trust Bundles.
• Review and offer feedback on the SPIFFE Filesystem Delivery draft standard, which aims to make it as easy as possible to use SPIFFE certificates directly on native Kubernetes.
• Participate in Kubernetes SIG Auth to help shape the future of signers that are built directly in to core Kubernetes.
• Try building your own signer based on Tinycert.

Happy hacking!

Kubernetes v1.37 promotes the metrics.k8s.io API to stable (v1). This API provides CPU and memory usage for nodes and Pods, and is the API behind commands such as kubectl top and resource-metrics-based autoscaling.

For cluster operators and application developers, this graduation means that the API now has the stability guarantees associated with a Kubernetes stable API. The v1 API has the same resource types and fields as v1beta1; this is an API-version graduation, not a change to the metrics that are collected or returned.

A long-lived API reaches stable

The resource Metrics API was introduced as alpha in Kubernetes v1.6 and became beta in v1.8. It has remained unchanged and has been used in production for years by clients including the HorizontalPodAutoscaler (HPA) and kubectl top. Kubernetes v1.37 formally graduates that proven API to metrics.k8s.io/v1.

The API exposes two resource types:

• NodeMetrics, for CPU and memory usage for a node.
• PodMetrics, for CPU and memory usage for a Pod, with a per-container breakdown in its containers field.

The API remains intentionally small. It provides the resource metrics needed for autoscaling and basic inspection; it is not a replacement for a full monitoring pipeline or the custom metrics (custom.metrics.k8s.io) API.

What changed with the v1.37 release?

The v1 API surface is identical to v1beta1, except for the API version. There are no renamed fields, new fields, or changes to the meaning of the returned CPU and memory values.

For example, a client can retrieve node metrics from the stable endpoint:

kubectl get --raw /apis/metrics.k8s.io/v1/nodes

Likewise, it can retrieve metrics for the pods in a namespace:

kubectl get --raw /apis/metrics.k8s.io/v1/namespaces/default/pods

kubectl top supports both API versions. It prefers v1 when available and automatically falls back to v1beta1 on clusters that do not yet serve v1. The HPA controller currently supports only v1beta1. Support for discovery-based selection between v1 and v1beta1 is planned, but is not available in Kubernetes v1.37.

What you need to do

You don't need to enable any feature gate. The Metrics API is served through the API aggregation layer, by an implementation such as metrics-server. You can choose any implementation of metrics.k8s.io; for the v1 metrics API to be available in your cluster, your chosen implementation must serve the v1.metrics.k8s.io API, and you need to register an associated APIService.

During the transition, implementations should serve both v1 and v1beta1. Keeping both versions available maintains compatibility with older clients. The v1beta1 API remains available in Kubernetes v1.37.

You can see which versions your cluster serves with:

kubectl get --raw /apis/metrics.k8s.io/ | jq .

Once your metrics implementation supports v1, you can also check that its APIService is available:

kubectl get apiservice v1.metrics.k8s.io

Learn more

• Read the Resource metrics pipeline documentation.
• Read KEP-5207, the proposal for (graduating) this API.
• Learn about the Metrics API and its reference implementation, metrics-server.

Get involved

The Metrics API is maintained by SIG Instrumentation. To ask questions, share feedback, or contribute, join the #sig-instrumentation channel on Kubernetes Slack or attend a SIG Instrumentation meeting.

Editors: Arsh Sharma, Christopher Tineo, Kirti Goyal, Sophia Ugochukwu, Swathi Rao, Troy Connor

Similar to previous releases, the release of Kubernetes v1.37 introduces new Stable, Beta, and Alpha features. The consistent delivery of high-quality releases underscores the strength of our development cycle and the vibrant support from our community.

This release consists of 67 enhancements. Of those enhancements, 16 have graduated to Stable, 23 have graduated to Beta, 27 are entering Alpha, and 1 is a deprecation/removal.

Release theme and logo

The theme for Kubernetes v1.37 is Garhwal (गढ़वाल, pronounced gaṛhvāl), a Himalayan region of Uttarakhand, India. The snow-capped peaks of the Garhwal Himalaya, deodar forests, terraced fields, rivers and streams, and mountain paths shape both the region and the logo. Together, these elements reflect a community in which every layer, route, and contribution is connected.

The logo is imagined as a window into Garhwal's landscape.¹ Inside, terraced fields climb towards the snowy peaks, each level supported by the one below, much as every Kubernetes release depends on work carried forward. A river winds through the valley and gathers mountain streams, reflecting contributions from many SIGs and communities flowing into one project.

The deodar forest represents the wider Kubernetes ecosystem, where distinct projects share common ground and grow side by side. Stonework and woodcraft shape the path and mountain house, placing people at the centre and evoking shared foundations maintained for those who follow. Above the river, colourful flags catch the wind and bring the scene to life.

Encircling the scene is a patterned frame inspired by basketry woven from ringaal, a flexible dwarf Himalayan bamboo. Individual strips gain strength when interlaced, just as code, reviews, tests, documentation, and coordination come together to make a release.

Within the frame, the Himalayan monal, Uttarakhand's state bird, lives at high altitudes in the Himalaya. Its iridescent plumage holds many colours at once, much as the Kubernetes community brings many skills and perspectives into one project. Flowers of red buransh (Rhododendron arboreum), Uttarakhand's state tree, carry Kubernetes helms at their centres, linking a familiar bloom of Garhwal with the symbol shared by the community. The house bears १.३७ (1.37 in Devanagari numerals), grounding the release in the landscape.

_{1. Keep looking through the window (the logo). Watch the river flow and the flags catch the wind. In 37 seconds, the landscape reveals its magic. 😉}

Spotlight on key updates

Kubernetes v1.37 is packed with new features and improvements. Here are a few select updates the Release Team would like to highlight!

Stable: Resilient watchcache initialization

Kubernetes v1.37 completes the work on resilient watch cache initialization: the ResilientWatchCacheInitialization feature gate reached Stable back in v1.34, and in v1.37 the remaining WatchCacheInitializationPostStartHook gate graduates to Stable and is locked on. It has defaulted to enabled since v1.36, hardening the API server at startup and during recovery. Watchcache initialization and reinitialization no longer create a traffic spike of requests against etcd, and requests are handled gracefully instead of piling up while the cache warms.

Instead of allowing expensive list and watch requests to overload etcd or exhaust API Priority and Fairness capacity, kube-apiserver now safely delegates bounded requests and rejects others with HTTP 429 responses. This reduces the risk of control plane outages in large clusters. Clients (including custom controllers and operators) should be designed to handle HTTP 429 Too Many Requests responses gracefully by respecting Retry-After headers and implementing exponential backoff.

This work was done as part of KEP #4568 led by SIG API Machinery.

Beta: HorizontalPodAutoscaler scale to zero

In Kubernetes v1.37, HorizontalPodAutoscaler scale to zero support is graduating to Beta. First introduced in Kubernetes v1.16, it is now enabled by default. For workloads that are using object or external metrics, this feature allows HorizontalPodAutoscalers to scale down to zero Pods when idle, then restore them when demand returns. Doing that can reduce costs for queue consumers, batch jobs, and GPU workloads. Setting spec.minReplicas: 0 applies this functionality for workloads.

Scaling to zero based on CPU and memory metrics is not supported because those metrics depend on active Pods. Instead, this feature is for situations such as leaving the replica count at zero until there is queued work to process.

While the HorizontalPodAutoscaler is holding a workload at zero replicas, it records a ScaledToZero condition with True in the HorizontalPodAutoscaler's status. The HorizontalPodAutoscaler controller then uses this condition to distinguish a workload that it scaled to zero (and will scale back up when the metric returns) from one that was manually deactivated by setting its replica count to 0. Once the workload is scaled back up, the condition is set to False with the reason NotScaledToZero.

This work was done as part of KEP #2021 led by SIG Autoscaling.

Beta: Manifest-based admission control configuration

Kubernetes v1.37 graduates manifest-based admission control configuration to Beta. Admission webhooks and CEL-based policies can now be loaded from manifest files on disk, via the staticManifestsDir field in AdmissionConfiguration, instead of living only in the Kubernetes API. Policies loaded this way are enforced from API server startup, keep working while etcd is unavailable, and can protect the API-based admission resources themselves from modification.

This work was done as part of KEP #5793 led by SIG API Machinery.

Alpha: Pod-level checkpoint and restore

Kubernetes v1.37 introduces Alpha support for Pod-level checkpoint and restore, extending the CRI with CheckpointPod and RestorePod RPCs, which allow the kubelet and compatible container runtimes to create a Pod checkpoint and restore a Pod from it. To use this feature, your container runtime(s) must also implement these new RPCs.

This work was done as part of KEP #5823 led by SIG Node.

Features graduating to Stable

This lists all the features that graduated to Stable (also known as General Availability). For a full list of updates including new features and graduations from Alpha to Beta, see the release notes.

This release includes a total of 16 enhancements promoted to Stable:

KYAML

KYAML is a safer and less ambiguous subset of YAML designed specifically for Kubernetes, not a replacement for it. Every KYAML file is valid YAML, so KYAML is a valid input for any version of kubectl, and spec files do not need to be written in KYAML for the input to be parsed. Your existing manifests, tooling, and pipelines don't need to change. Introduced as an Alpha feature in v1.34 and graduating to Beta in v1.35, KYAML graduates to Stable in v1.37 with conformance testing complete, and kubectl get -o kyaml is now Stable.

To learn more about KYAML, check out How to Pretty-Print Your Kubernetes YAML as KYAML and Why You'd Want To.

This work was done as part of KEP #5295 led by SIG CLI.

The metrics.k8s.io API

The metrics.k8s.io API graduates to Stable in Kubernetes v1.37 after spending nearly nine years in Beta. The API provides a standard way to retrieve CPU and memory usage for pods and nodes, powering widely used Kubernetes features such as the HorizontalPodAutoscaler (HPA) and commands like kubectl top.

The graduation follows the Kubernetes project’s goal of avoiding permanent Beta APIs. Now that v1 exists, future Kubernetes releases will move over to it; v1beta1 remains usable throughout the transition, in line with the API deprecation policy, so you can adopt the Stable API without breaking existing workflows.

This work was done as part of KEP #5207 led by SIG Instrumentation.

SELinuxMount and SELinuxChangePolicy

In Kubernetes v1.37, SELinuxMount and SELinuxChangePolicy flags reach Stable and are enabled by default: this means that volumes get mounted with -o context=<label> (the MountOption default) instead of being recursively relabeled, but only when the volume's CSI driver opts in via .spec.seLinuxMount: true for the CSIDriver object.

A mount can only carry one SELinux context, so Pods with different SELinux labels sharing a volume on the same node, which used to coexist under recursive relabeling, can now fail to start. To retain the old behavior for a workload, it is advised to set the .spec.seLinuxChangePolicy to Recursive on a Pod.

This behavior itself also isn't locked until v1.38, so disabling it cluster-wide remains an option for one more release.

Clusters without SELinux enabled see no effect at all. To learn more, check SELinux Volume Label Changes goes GA (and likely implications in v1.37).

This work was done as part of KEP #1710 led by SIG Storage.

DRA features graduating to Stable

DRA: ResourceClaim status with possible standardized network interface data

The ResourceClaim .status.devices reaches Stable in Kubernetes v1.37, which allows drivers to report device-specific device status data for each allocated device in a resource claim. This makes it easier to see how a device is configured, troubleshoot problems, and use the device with other services.

This is particularly useful for network devices; before this field was added, if a Pod requested a network device via DRA, there was no way for any other component in the system to learn the IP address that was assigned to that network device. The new status field provides a standardized way for the DRA driver to export that information to components that need it, making DRA fully usable for attaching secondary network interfaces to Pods.

This work was done as part of KEP #4817 led by SIG Node and SIG Network.

DRA: handle extended resource requests via DRA Driver

DRA Extended Resource support reaches Stable in Kubernetes v1.37. This feature allows DRA drivers to fulfill requests made through the traditional extended resource mechanism, such as abc.example/gpu: 3 in a Pod spec, without requiring a separate device plugin.

With this mechanism, an extended resource name can be assigned directly to a DeviceClass. Pods requesting that resource can then have a device allocated through DRA without needing to define a ResourceClaim in the workload.

This work was done as part of KEP #5004 led by SIG Scheduling.

DRA: device taints and tolerations

Support for taints and tolerations for physical devices managed through DRA is now Stable in Kubernetes v1.37. By default, any available device can be considered for scheduling. This enhancement provides greater control over device scheduling by allowing DRA drivers to mark specific devices as tainted, preventing them from being selected for workloads. Alternatively, cluster administrators can create a DeviceTaintRule to taint devices based on specific selection criteria, such as all devices managed by a particular driver.

This work was done as part of KEP #5055 led by SIG Scheduling.

DRA: standard numaNode device attribute

Kubernetes v1.37 defines a new standard NUMA node device attribute. It standardizes resource.kubernetes.io/numaNode as a shared attribute name for device NUMA node information, allowing devices managed by different DRA drivers to be compared based on the same NUMA node. This avoids each driver defining its own attribute name and provides a consistent way to identify NUMA placement across devices. The enhancement lands directly as Stable because it is a naming and registration KEP with no feature gate or in-tree behavior changes.

This work was done as part of KEP #6072 led by SIG Node.

Node declared features

Node declared features graduate to Stable in Kubernetes v1.37, providing a framework to declare the availability of specific, feature-gated Kubernetes features for Nodes. This would then be used by control plane components (such as the kube-scheduler, admission controllers, or the API server itself) to manage version skew.

The feature introduces a new .status.declaredFeatures field for Nodes, which is used to declare a feature graduating through the Alpha → Beta → Stable stages. The control plane can use this to adopt the correct behavior even in a cluster running a mixture of different node versions.

Once features graduate to Stable and the control plane can assume all nodes support them across the supported version skew window, nodes stop reporting them.

The kubelet determines its declared features when it starts, based only on feature gates and the node’s static configuration (so any changes require a kubelet restart).

This work was done as part of KEP #5328 led by SIG Node.

Storage version migrator

Kubernetes v1.37 sees the StorageVersionMigration API (storagemigration.k8s.io/v1) graduate to Stable and become enabled by default. It helps migrate existing resources, both built-in and custom, from an older storage version to the new storage version after an API upgrade, such as when the preferred storage version changes from v1beta1 to v1. It can also be used to rewrite existing data after a change to encryption at rest, so that stale data is stored using the new encryption settings.

Historically, cluster administrators and CustomResourceDefinition authors had to use manual kubectl get or kubectl replace scripts, or deploy the out-of-tree kube-storage-version-migrator component to rewrite existing resources. These approaches were often tedious, error-prone, and difficult to monitor.

To start a storage version migration, users would need to create a declarative StorageVersionMigration object. The built-in StorageVersionMigrator controller in the Kubernetes control plane watches for these objects and automatically migrates existing resources to the default storage version for that API. Since StorageVersionMigration is a standard Kubernetes API, CRD authors can trigger migrations as part of a CRD upgrade instead of managing the migration separately.

This work was done as part of KEP #4192 led by SIG API Machinery.

Stable: Pod certificates and Cluster Trust Bundles

Pod certificates and the closely related ClusterTrustBundles both graduate to Stable in Kubernetes v1.37, providing first-class support for distributing private keys, X.509 certificates, and trust bundles to Pods.

To use this, the developer or administrator chooses a signer name and deploys a signer controller that watches PodCertificateRequest objects, issues and refreshes certificates for eligible Pods, and maintains the corresponding ClusterTrustBundle objects containing the trust anchors needed to verify those certificates. A workload then opts into this identity by defining a podCertificate projected volume with the chosen signer name. Workloads can also mount a ClusterTrustBundle projected volume to load the trust anchor information.

This work was done as part of two KEPs - KEP #4317 and KEP #3257 led by SIG Auth.

Features graduating to Beta

Gang scheduling support in Kubernetes

As Kubernetes becomes the de facto standard for managing AI/ML workloads at scale, scheduling workloads such as AI/ML training jobs and HPC simulations becomes more important than ever. However, scheduling becomes challenging because the default Kubernetes scheduler schedules Pods individually, which can result in some Pods being scheduled while others remain pending due to insufficient resources. This partial scheduling can lead to deadlocks and inefficient use of cluster resources.

Gang scheduling graduates to Beta in Kubernetes v1.37, improving upon native support for gang scheduling through the Workload API and PodGroup concept. This feature implements an all-or-nothing scheduling strategy, ensuring that a defined group of Pods is scheduled only when the cluster has sufficient resources to accommodate the entire group. The Beta graduation of this enhancement also introduces workload-aware preemption to avoid premature preemptions that do not help a workload make progress, along with PodGroup queueing to better coordinate competing workloads.

Importantly, it addresses livelock scenarios that can occur when multiple workloads are being scheduled simultaneously by the kube-scheduler, preventing them from repeatedly interfering with one another without making progress.

This work was done as part of KEP #4671 led by SIG Scheduling.

Native histogram support for Kubernetes metrics

Kubernetes exposes hundreds of histogram metrics in Prometheus format across its control plane components, which are essential to monitor cluster health and debug performance issues. However, classical Prometheus histograms relied on static, pre-defined buckets that forced a compromise between data accuracy and memory usage. To mitigate this, Prometheus introduced native histograms that use dynamic exponential bucket boundaries instead of fixed boundaries, providing significant storage efficiency, improved query performance, and finer-grained visibility into distributions while maintaining full backward compatibility with existing monitoring infrastructure.

Kubernetes v1.37 graduates native histogram support for Kubernetes metrics to Beta. Building on the Alpha implementation, which introduced the NativeHistograms feature gate, the Beta phase improves the implementation and rollout experience. When enabled, Kubernetes components expose histograms in both classic and native formats when the requested scrape protocol supports Native Histograms, (specifically PrometheusProto), allowing existing dashboards and alerts to continue working while users migrate at their own pace. The implementation also refactored histograms created in init() functions to use lazy initialization, ensuring native histogram options are correctly applied after feature gates are parsed. These changes provide a more reliable implementation while retaining safe rollout and rollback through the feature gate or Prometheus-side configuration for Prometheus 3.x users.

This work was done as part of KEP #5808 led by SIG Instrumentation.

WAS: Features graduating to Beta

Workload-aware preemption

Kubernetes traditionally performs preemption at the Pod level, which can be inefficient for workloads made up of multiple tightly coupled Pods. In Kubernetes v1.37, workload-aware preemption graduates to Beta, allowing the scheduler to consider a PodGroup when making preemption decisions. This helps the scheduler consider the workload as a whole when preempting lower priority workloads, reducing cases where individual Pods are disrupted without providing enough capacity for the workload to make progress.

This work was done as part of KEP #5710 led by SIG Scheduling.

DRA: ResourceClaim support for workloads

Dynamic Resource Allocation (DRA) allows Pods to request specialized resources through ResourceClaims. In Kubernetes v1.37, DRA ResourceClaims support for workloads graduates to Beta, allowing Workload and PodGroup APIs to associate ResourceClaims and ResourceClaimTemplates with the groups of Pods. This allows ResourceClaims to be shared across a workload rather than reserved individually for each Pod, while ResourceClaimTemplates can create claims for PodGroups automatically.

This work was done as part of KEP #5729 led by SIG Scheduling.

cAdvisor-less, CRI-full container and Pod stats

The kubelet has historically obtained container and Pod statistics from cAdvisor, while the Container Runtime Interface (CRI) exposes statistics of its own. Having two sources for the same metrics makes it harder to tell where a particular value came from.

In Kubernetes v1.37, the cAdvisor-less, CRI-full Container and Pod Stats enhancement graduates to Beta. The enhancement expands the CRI to provide the container and pod statistics needed by Kubernetes, allowing the kubelet to get these metrics directly from the container runtime instead of relying on cAdvisor for them.

This moves container and pod metrics toward a single source of truth, while reducing duplicated metric collection and simplifying how the kubelet gathers and exposes these statistics.

This feature is Beta in v1.37 but off by default; enable the PodAndContainerStatsFromCRI feature gate to try it.

This work was done as part of KEP #2371 led by SIG Node.

Support memory QoS with cgroups v2

Kubernetes is improving its quality of service mechanisms to cover memory protection and isolation for Kubernetes workloads. For nodes running Linux, the memory QoS feature uses memory requests and limits to configure cgroup controls that can protect requested memory from reclamation and throttle memory usage before workloads reach their hard limits. This can help reduce the impact of memory pressure on memory-sensitive workloads and improve node stability.

In Kubernetes v1.37, memory QoS support is graduating to Beta. The feature uses cgroups v2 memory controls such as memory.min, memory.low and memory.high to provide different levels of memory protection and throttling. For example, memory requests can be used to protect memory from reclamation, while memory.high can be used to throttle workloads that exceed their configured threshold.

The MemoryQoS feature gate is enabled by default in v1.37. Cluster operators can control memory protection through the kubelet’s memoryReservationPolicy setting and configure memory throttling with memoryThrottlingFactor. The defaults are designed to avoid introducing unexpected memory throttling for existing workloads when upgrading to v1.37, while allowing operators to opt into the additional memory protection capabilities.

This work was done as part of KEP #2570 led by SIG Node.

Pod-level resource managers

In Kubernetes v1.37, Pod-level resource managers graduate to Beta behind the PodLevelResourceManagers feature gate,

which stays disabled by default. Enabling it allows the topology, CPU, and memory resource

managers to use the resources defined for an entire Pod when making

allocation and NUMA alignment decisions. This makes it possible to manage a Pod as a single resource unit while still supporting different resource requirements between the containers within it.

With pod-level resource management, a Pod can reserve a NUMA-aligned pool of CPU and memory based on its overall resource budget. Containers that require dedicated resources can receive exclusive portions of that pool, while other containers, such as sidecars or supporting workloads, can share the remaining resources. This is particularly useful for performance-sensitive workloads such as AI/ML and high-performance computing, where keeping resources close to each other on the same NUMA node can improve performance without requiring every container in the Pod to have dedicated resources.

The feature also supports a container scope, where containers can continue to receive independent NUMA-aligned allocations. This provides more flexibility for workloads that combine a performance-sensitive container with other containers that have different resource requirements.

This work was done as part of KEP #5526 led by SIG Node.

Watch-based route controller reconciliation

The route controller in the cloud-controller-manager library previously reconciled routes on a fixed interval, by default every 10 seconds. This could result in unnecessary requests to infrastructure providers, even when nothing had changed and could also delay route updates when a new Node is added.

Watch-based route controller reconciliation graduated to beta in Kubernetes v1.37. This release also adds observability for this work: the route controller's Alpha route_sync_total metric gains two labels, trigger (periodic or node_change) and outcome (changed, noop, or error), so operators can see whether periodic reconciliation is actually correcting route drift or just running as a no-op, and can track failed reconciles.

With watch-based route controller reconciliation, the route controller can reconcile routes from watch events instead of waiting for the next fixed interval: a reconciliation can start as soon as relevant Node changes occur, such as a Node being added or removed or when its addresses or assigned Pod CIDRs change. A less frequent periodic reconciliation still runs to catch outdated routes and keep the state consistent. This behavior sits behind the CloudControllerManagerWatchBasedRoutesReconciliation feature gate and is disabled by default, so the transition has not changed default behavior.

This reduces unnecessary requests to infrastructure providers while allowing routes for newly added Nodes to be reconciled sooner. The change does not alter the route reconciliation logic itself; it changes when reconciliation is triggered.

This work was done as part of KEP #5237 led by SIG Cloud Provider.

Storage capacity scoring of Nodes

The VolumeBinding scheduler plugin has always been able to score nodes for statically bound PVs based on free capacity, but that scoring never extended to dynamic provisioning.

When a CSI driver provisions a new volume on demand, the scheduler had no way to prefer a node with more or less free space.

This was a gap for local storage, as an admin might want pods landing on the node with the most free capacity to leave room for a later volume expansion or on the node with the least (but still sufficient) free capacity to bin-pack workloads and cut down on the number of nodes a cloud cluster needs to run.

Kubernetes v1.37 graduates storage capacity scoring for dynamic provisioning to Beta behind the StorageCapacityScoring feature gate. First introduced in Alpha in v1.33, this feature consolidates (and deprecates) the older VolumeCapacityPriority gate from KEP #1845. When enabled, the VolumeBinding plugin's Score extension point reads CSIStorageCapacity objects published by a driver's external provisioner sidecar and scores nodes for dynamic provisioning the same way it already does for static bindings. Admins choose the strategy via the Shape setting in VolumeBindingArgs, defaulting to "prefer the node with the maximum allocatable" so there is room for expansion later.

The feature depends solely on the StorageCapacityScoring gate: scoring for statically bound PVs runs as soon as it's enabled, independent of any CSI driver. A driver only needs StorageCapacity: true on its CSIDriver object so that its dynamically-provisioned volumes also get capacity-aware scoring. The feature is fully reversible, and disabling the gate stops all VolumeBinding capacity scoring — static and dynamic alike — without affecting already scheduled pods.

This work was done as part of KEP #4049 led by SIG Storage.

Integrate CSI volume attach limits with Cluster Autoscaler

Kubernetes v1.37 improves Cluster Autoscaler's integration with CSI volume attach limits, so that when it creates new nodes for pending Pods, Cluster Autoscaler can more accurately determine how many new nodes are required to attach all pending Pods that use CSI volumes. Cluster Autoscaler already had visibility into CSI volume attach limits for existing nodes, but not for the nodes it was about to create, which means it could undershoot scale-ups and leave volume-backed Pods pending even after adding capacity. The problem compounds on the scheduling side: the NodeVolumeLimits plugin treats a node with no published CSI driver info as having no limits at all, so a freshly created node that hasn't yet reported its CSINode object can get crowded with more volume-backed pods than it can actually mount, which is a race condition that, until now, cluster admins had no way to close.

Kubernetes v1.37 graduates CSI-aware autoscaling to Beta behind the VolumeLimitScaling feature gate, first introduced in Alpha in v1.35. Cluster autoscaler now runs its scale-up simulations against templated CSINode objects, so it correctly accounts for attach limits whether it's scaling an existing node group or scaling one from zero. On the scheduler side, admins can opt in per CSIDriver, via a new PreventPodSchedulingIfMissing field, to block pod placement on nodes that haven't reported their driver yet, with dedicated CSIDriverMissingOnNode and CSINodeMissing errors making those scheduling failures easier to debug. The Beta phase adds e2e coverage for scale-down behavior and CSI opt-in scenarios, and updates the failed_scale_ups_total and scaled_up_nodes_total metrics to include CSI driver information. Both the autoscaler and scheduler changes stay strictly opt-in: disabling the feature gate restores today's default of unlimited pod placement on nodes without CSINode data, so distros and admins running autoscalers that aren't CSI-aware yet (e.g. Karpenter) aren't forced into the new behavior.

This work was done as part of KEP #5030 led by SIG Autoscaling.

Report last used time on a PVC

PersistentVolumeClaims tend to outlive the workloads that created them. When an app gets deleted or migrated, its PVC remains behind, consuming storage and increasing costs.

Kubernetes v1.37 graduates PVC "last used" tracking to Beta behind the PersistentVolumeClaimUnusedSinceTime feature gate, which shipped disabled by default in Alpha (v1.36) and is now enabled by default. The feature adds a new Unused condition to PersistentVolumeClaimStatus, managed by the existing PVC protection controller: Status=True (Reason=NoPodsUsingPVC) once the last non-terminal Pod referencing the PVC goes away, and back to Status=False (Reason=PodUsingPVC) as soon as a Pod starts referencing it again. The condition's lastTransitionTime doubles as an "unused since" timestamp, so admins can query how long a PVC has actually been idle without Kubernetes tracking which Pod used it last or making any deletion decision itself; that's left entirely to the admin. One thing worth noting is that the timestamp reflects when the controller observed no Pods using the PVC, not the exact moment the volume unmounted at the infrastructure level, so the reported idle time may run a little short of the true figure but should never overstate it.

This work was done as part of KEP #5541 led by SIG Storage.

etcd RangeStream support

etcd's unary Range RPC builds an entire response in memory before sending it back, which becomes a problem at scale. On a large list, say kube-apiserver's watch cache warming up on a big cluster, the raw key-value slice, its serialized protobuf form, and the gRPC send buffer all have to coexist in memory at once, and the resulting spikes ripple through kube-apiserver too. Pagination doesn't really fix the underlying cost either because each paginated page still walks the entire B-tree index to recompute the total result count, turning what should be an O(limit) operation into an O(total_keys) one on every single page.

Kubernetes v1.37 ships etcd RangeStream support directly at Beta, behind the EtcdRangeStream feature gate (kube-apiserver only, on by default). This release adds a new server-streaming RangeStream RPC that reuses the existing RangeRequest but returns chunks instead of one buffered blob: the server paginates internally with adaptive chunk sizing (each chunk's target size adjusts based on MaxRequestBytes and the value sizes observed so far), pins a single MVCC revision so the merged stream stays snapshot-consistent, and derives the total key count from the running tally it builds while streaming, rather than a separate index walk. kube-apiserver's watch cache initialization is the primary consumer, and it now decodes each chunk into synthetic created events inline as they arrive instead of assembling the full list in memory first, with the same treatment applied to direct GetList calls when WatchList is disabled.

The feature requires etcd 3.7+; against older etcd, kube-apiserver detects the Unimplemented response and falls back to unary Range automatically, with zero behavior change. If the pinned revision gets compacted mid-stream, kube-apiserver treats it the same as any other watch cache init failure and retries, which is no worse than the compaction races a paginated List call can already hit today. Beta graduation criteria include a scalability test measuring large-list latency on a 5000-node cluster, and etcdctl get --stream ships alongside it for anyone who wants to poke at the new RPC directly.

This work was done as part of KEP #5966 led by SIG etcd.

Concurrent watch object decode

kube-apiserver decodes and transforms every watch event from etcd one at a time on a single goroutine, so one slow per-event transform, most notably a CRD conversion webhook call, blocks every event queued behind it. That's mostly a nuisance for built-in resources, but for a CRD whose served version differs from its stored version, converting a cold cache serially can take minutes. If that exceeds etcd's default 5-minute compaction interval, the revision the cache started reading from gets compacted before initialization finishes, the watch can't resume, and init just restarts and never converges for a large enough resource, with every client trying to list or watch it getting errors in the meantime.

The ConcurrentWatchObjectDecode gate has actually been in Beta, off by default, since v1.31, and Kubernetes v1.37 flips it on by default. Enabling it moves the decode/transform step onto a bounded pool of worker goroutines (10 by default, tuned from a sweep that showed gains flattening out around 8–12) instead of a single one, with a collector reassembling events back into their original order before delivery, so event ordering is preserved exactly. In benchmarks over 150k pods, concurrent decode alone cuts cache initialization about 40%, and about 55% combined with the new EtcdRangeStream feature also landing this release (see KEP 5966). The main tradeoff to watch is conversion webhook load. With the feature on, up to 10 conversions can now run concurrently against a webhook during cache init instead of one at a time. The total call volume is unchanged, only how many run at once, so this mainly matters for webhooks that cap their own concurrency below 10.

This work was done as part of KEP #6178 led by SIG API Machinery.

Stale controller mitigation

Every controller in kube-controller-manager works off a local cache built from watching the kube-apiserver, and that watch stream is only eventually consistent. A change can show up in milliseconds, or it can take seconds or even minutes under load. Today operators have no visibility into that lag and no way to tell a normal delay from a controller that's fallen dangerously out of sync, so a controller can keep reconciling against a view of the world that's already stale.

Stale controller mitigation has been Beta since v1.36, enabled by default per controller behind a StaleControllerConsistency<Controller> feature gate; Kubernetes v1.37 extends it to the HorizontalPodAutoscaler controller and adds the circuit-breaking variant and extra metrics described below. The core mechanism is a read your writes guarantee: client-go's ResourceEventHandlerFuncs gets a new BookmarkFunc callback so a controller can reliably track the resource version of objects it cares about, even through edge cases the existing add/update/delete callbacks miss. A controller records the resource version of its own writes and, on its next reconcile, skips and requeues until its informer cache has actually caught up to that write. The DaemonSet controller is a good example of this. It tracks DaemonSet → Pod resource versions so it won't re-reconcile against its own stale pod cache. A second, circuit-breaking variant targets latency-sensitive controllers like node-lifecycle, which can otherwise read a stale node lease from cache and wrongly decide it's expired; instead, it does a live GET on the disruptive decision and marks its cache "not ready" until it's caught up, rather than acting on a stale read. StaleControllerConsistency gates the mitigation itself (scoped initially to controllers KCM has flagged as high-scale), MonitorInformerStaleness is a separate, observation-only gate that polls the apiserver directly every 5 seconds purely to surface how far behind an informer's cache actually is, and AtomicFIFO / UnlockWhileProcessingFIFO are the underlying client-go workqueue plumbing the mitigation depends on. None of this changes default reconciler behavior; a paused-and-requeued controller can look stuck when it's really just waiting on its cache, and it rolls back cleanly since nothing it does is irreversible.

This work was done as part of KEP #5647 led by SIG API Machinery.

Manifest-based admission control config

In Kubernetes, admission control is responsible for enforcing policies on resources before they are accepted by the API server. However, admission webhooks and policies configured through the Kubernetes API are dependent on the API server and etcd during cluster startup and cannot protect the admission configuration resources themselves. This creates a gap during cluster bootstrap and allows critical admission policies to be modified or removed by users with sufficient privileged access.

In Kubernetes v1.37, manifest-based admission control configuration graduates to Beta, allowing admission webhooks and CEL-based

policies to be loaded from manifest files on disk and enforced from API server startup. Because the configuration is managed independently of the Kubernetes API, it can also protect API-based admission resources from modification. Manifest files are watched for changes and valid updates are reloaded automatically, while invalid updates leave the previously loaded configuration in place.

This work was done as part of KEP #5793 led by SIG API Machinery

Improved handling for undecryptable resources

Kubernetes stores resources in etcd, where encryption at rest can be used to protect sensitive data. However, when encrypted resources can no longer be decrypted, for example because the encryption key is unavailable, the API server cannot read or manage those resources normally. This can leave resources in the cluster that cannot be accessed through the Kubernetes API, requiring administrators to manually modify the underlying etcd data to recover them.

Kubernetes v1.37 includes Beta support for cluster administrators to identify and remove resources that cannot be decrypted by the API server.

Previously Alpha, and introduced in Kubernetes v1.32, this support allows problem API resources to be removed via

the Kubernetes API rather than directly manipulating the etcd file. This feature also provides safeguards for administrators to verify affected resources before deletion.

This work was done as part of KEP #3926 led by SIG Auth.

New features in Alpha

New Recreate strategy for StatefulSet rollouts

Kubernetes v1.37 introduces the Recreate strategy for StatefulSet rollouts. The StatefulSet API previously only offered two update strategies: OnDelete (manual) and RollingUpdate (automatic, default). Similar to Deployments, the Recreate update strategy deletes all of the StatefulSet's Pods before creating new Pods that reflect modifications made to a StatefulSet's .spec.template. Using this strategy requires the StatefulSetRecreateStrategy feature gate to be enabled.

This work was done as part of KEP #3541 led by SIG Apps.

DRA: Alpha features to look out for

DRA: Node allocatable resource request

Kubernetes v1.37 improves Alpha support for managing node resources such as CPU, memory, and huge pages through DRA. It unifies standard and DRA resource accounting, helping prevent the same node capacity from being counted twice.

This update introduces distinct API fields for mapping (for devices directly modeling core resources, like CPU/memory DRA drivers) and overhead (like auxiliary host memory for accelerator devices). The kubelet now enforces these allocations across pod and container cgroups, integrates them with Memory QoS, OOM score calculations, and in-place pod resizing.

This work was done as part of KEP #5517, led by SIG Scheduling with participation from SIG Node.

DRA: derived attributes

Kubernetes v1.37 introduces Alpha support for derived attributes in DRA. Workloads can use CEL expressions to create virtual attributes from device information and use them when selecting related devices.

This makes it easier to co-locate devices such as GPUs and network interfaces, even when their drivers use different attribute names or formats. For example, a workload can derive a shared NUMA identifier and use it to select devices with matching topology.

This work was done as part of KEP #6080, led by SIG Scheduling with participation from SIG Network.

DRA: device compatibility groups

DRA can be used to manage devices that support different partitioning or virtualization schemes. However, some of these configurations cannot be used together on the same physical device, such as MIG and vGPU on a GPU. Previously, these incompatibilities could only be detected during device preparation, after the scheduler had already made its decision.

In Kubernetes v1.37, DRA adds device compatibility groups, allowing resource drivers to describe which devices can be allocated together. The scheduler can use this information when making allocation decisions, preventing incomplete devices from being assigned together and avoiding Pod startup failures caused by incomplete device configurations.

This work was done as part of KEP #5963, led by SIG Scheduling.

Scheduler preemption for in-place Pod resize

Kubernetes v1.37 introduces scheduler preemption for in-place pod resize, behind the (opt-in, Alpha) InPlacePodVerticalScalingSchedulerPreemption feature gate. This change addresses an important feature gap that remained after the core in-place Pod vertical scaling feature graduated to Stable: if a running pod requested additional resources that exceeded the node's available capacity, kubelet marked the request as Deferred, leaving the pod waiting until sufficient resources became available on the node. With this enhancement, the Kubernetes control plane can actively free up capacity on a fully-utilized node and preempt lower-priority workloads, enabling the pending in-place resizes of critical, higher-priority applications to succeed.

This work was done as part of KEP #5836 led by SIG Scheduling.

Dynamic resize of memory-backed volumes

Also building upon in-place Pod vertical scaling, the Alpha in-place scaling for memory backed volumes feature extends the pod

/resize subresource, which previously only enabled dynamic CPU and memory adjustments without restarting containers, to support updating the sizeLimit of memory-backed (medium: Memory) emptyDir volumes on running pods. When a volume's sizeLimit is explicitly adjusted via the /resize subresource, Kubelet dynamically updates the underlying tmpfs mount without container disruption while safely preventing out-of-memory errors or false-positive eviction triggers. This is particularly useful for stateful and memory-intensive workloads that rely on in-memory ephemeral storage, allowing them to dynamically scale storage limits alongside container memory capacity without incurring Pod restarts or application downtime.

This is an opt-in, off-by-default Alpha feature. To try it out, enable the

InPlacePodVerticalScalingMemoryBackedVolumes feature gate.

This work was done as part of KEP #6030 led by SIG Node and SIG Storage.

Specialized lifecycle management for Nodes

Several Kubernetes components need to understand a Node's lifecycle state, and today each one infers it from a different mix of Node readiness, taints, Pod state, labels, annotations, and provider APIs. This enhancement introduces well-known lifecycle conditions on Nodes, giving administrators a single Kubernetes-owned place to publish lifecycle state that core controllers and ecosystem tooling can consume. These new Node Conditions are: DrainInProgress, Drained, MaintenancePlanned, MaintenanceInProgress, and GracefulNodeShutdownInProgress

This work was done as part of KEP #5683 led by SIG Node.

WAS: Alpha features to look out for

CompositePodGroup API

While previous releases introduced support for gang scheduling of workloads with a flat structure, modern AI/ML workloads are complex and have more sophisticated scheduling requirements. In Kubernetes v1.37, the new Alpha CompositePodGroup API allows Kubernetes to describe complex workloads as a hierarchy of groups instead of a flat set of Pods. This enables multi-level gang scheduling, workload-aware preemption and topology-aware scheduling.

This work was done as part of KEP #6012 led by SIG Scheduling

Workload Aware Scheduling Controller APIs

As an Alpha feature Kubernetes v1.37 provides a common framework for integrating workload controllers (such as JobSet, TrainJob, LWS, and RayJob, along with core workloads such as Job) with Workload-aware Scheduling (WAS).

The framework provides reusable scheduling.k8s.io API primitives, such as topology constraints and disruption policies, along with shared libraries that handle the creation of scheduling resources. This allows controllers to expose WAS features natively within their APIs in a consistent way without implementing the same scheduling logic separately.

This work was done as part of KEP #6089 led by SIG Scheduling.

Integrate workload APIs with the Job controller

Initially introduced in Kubernetes v1.36 with limited functionality, this feature builds on top of the Workload Aware Scheduling Controller APIs adding a new user-facing spec.scheduling field to the batch/v1 Job in Kubernetes v1.37, allowing users to explicitly configure scheduling policies, topology constraints, disruption modes, and resource claims. If spec.scheduling is omitted, the Job defaults to Basic scheduling, preserving existing behavior while still creating a Basic Workload/PodGroup for workload-aware scheduling, without enforcing a minCount gate. Users can explicitly opt into Gang scheduling, where minCount defaults to the Job’s parallelism, and the controller uses the shared workloadbuilder library to translate the scheduling configuration into the corresponding Workload and PodGroup objects instead of implementing custom translation logic.

This work was done as part of KEP #5547 led by SIG Scheduling.

localhost NodePort userspace proxy for nftables

Kubernetes v1.37 adds an opt-in userspace proxy to the nftables kube-proxy backend, allowing NodePort services to be accessed through localhost over IPv4 and IPv6. This closes a gap between the nftables and iptables backends, as nftables could not previously serve localhost NodePorts.

The proxy is enabled when localhost or a loopback address is included in the kube-proxy --nodeport-addresses configuration. This can be useful for workloads such as local container registries that rely on localhost:<NodePort> connections. The existing behavior of the iptables and ipvs backends is unchanged.

This work was done as part of KEP #6032 led by SIG Network.

Other notable changes

maxUnavailable for StatefulSets back on by default

The maxUnavailable field for StatefulSets has been re-enabled by default in Kubernetes v1.37 (after a bug was observed in v1.36).

The bug occurred where a faulty initial StatefulSet revision created a Pod that never became ready, and with MaxUnavailableStatefulSet enabled, the StatefulSet controller failed to update that Pod to the newer, corrected revision. When the bug triggered, the affected Pod could end up stuck in a CrashLoopBackOff state indefinitely (see kubernetes#137409).

Improved nftables performance

kube-proxy now uses the kernel’s netlink interface for nftables rule operations, bypassing the nft command-line tool. This makes kube-proxy more efficient when inspecting and managing its nftables rules, improving rule-management performance.

Context handling and contextual logging in client-go

Support for context propagation and contextual logging in client-go is complete, with the exception of a small number of authentication plugin log calls that still rely on the global klog logger because the underlying APIs do not support context passing.

Graduations, deprecations, and removals in v1.37

Graduations to Stable

This lists all the features that graduated to Stable (also known as general availability). For a full list of updates including new features and graduations from Alpha to Beta, see the release notes.

This release includes a total of 16 enhancements promoted to Stable:

• Speed up recursive SELinux label change
• ClusterTrustBundles
• Pod Certificates
• Allow setting arbitrary FQDN as the pod's hostname
• DRA: Resource Claim Status with possible standardized network interface data
• Configurable tolerance for HorizontalPodAutoscalers
• Relaxed validation for Services names
• Add Resource Health Status to the Pod Status for Device Plugin and DRA
• DRA: device taints and tolerations
• DRA: Handle extended resource requests via DRA Driver
• Node Declared Features
• Add condition for sandbox creation
• Move Storage Version Migrator in-tree
• Resilient Watchcache Initialization
• DRA: Standard numaNode Device Attribute
• metrics.k8s.io API definition
• KYAML

Deprecations, removals and community updates

As Kubernetes develops and matures, features may be deprecated, removed, or replaced with better ones for the project's overall health. See the Kubernetes deprecation and removal policy for more details on this process. Many of these deprecations and removals were announced in the Deprecations and Removals blog

Deprecation of kube-dns

CoreDNS has been the default cluster DNS add-on since Kubernetes v1.13, and kube-dns has not kept pace since then; features like EndpointSlices and dual-stack Services aren't available in it.

Kubernetes has already retired the kube-dns subproject and has split node-local-dns out into its own repository, where it continues to be maintained and works with CoreDNS. It is expected that no new packages will be built for kube-dns after v1.40.

If you still run kube-dns, start planning to migrate your clusters to CoreDNS.

Deprecating kube-proxy's support for ipvs mode

kube-proxy support for ipvs mode was introduced in v1.8 to resolve iptables performance bottlenecks. However, since the kernel ipvs API alone cannot fully implement Kubernetes Services, ipvs mode continues to use iptables underneath (KEP-3866, "The ipvs mode of kube-proxy will not save us").

Clusters running kube-proxy in ipvs mode (or mode: ipvs in KubeProxyConfiguration) now log a deprecation warning on startup. The deprecation timeline looks like this:

• By v1.40, ipvs mode for kube-proxy is expected to be disabled by default (still selectable via the feature gate)
• By v1.43, support for ipvs mode would be removed entirely KEP #5495, Graduation Criteria. To confirm which mode you’re currently running, use:

kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep 'mode:'

To understand the rationale behind this deprecation, see KEP #5495.

kubectl: kubectl run --filename/-f to be deprecated

The --filename (or -f) flag for kubectl run is being deprecated as the generated pod is always built purely from CLI arguments like NAME and --image.

See kubernetes/kubernetes#138671 for the original issue and discussion.

kubelet: static Pods can no longer reference Secrets or ConfigMaps

Static Pods were never meant to read API resources directly, since they aren't created through the API server — but a bug let them reference Secrets or ConfigMaps via fields like configMapRef or secretRef. That bug is now fixed: as of v1.37 these references are strictly prohibited, and the PreventStaticPodAPIReferences feature gate that previously let you opt out of the restriction has been removed.

See kubernetes/kubernetes#140226 for the original issue and discussion.

Ongoing major change: Future removal of cgroup v1 support

As modern Linux distributions and container runtimes use cgroup v2 as the default, support for the legacy cgroup v1 is officially being phased out. Since the v1.35 release, the failCgroupV1 setting has defaulted to true. Consequently, the kubelet will fail to initialize on any nodes that still rely on cgroup v1 unless an explicit configuration override is applied.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false # temporary override

Using this override should be considered a short-term fix. Advanced resource management capabilities, such as memory QoS and in-place scaling for memory-backed volumes, work only on cgroups v2. While the override remains available in Kubernetes v1.37, users are encouraged to migrate to cgroups v2, as support for cgroups v1 is planned to be removed in a future release.

To learn more about this deprecation, refer to KEP #5573.

Release notes

Check out the full details of the Kubernetes v1.37 release in our release notes.

Availability

Kubernetes v1.37 is available for download from the Kubernetes download page or direct from on GitHub.

To get started with Kubernetes, check out these tutorials or run local Kubernetes clusters using minikube. You can also easily install v1.37 using kubeadm.

Release team

Kubernetes is only possible with the support, commitment, and hard work of its community. Each release team is made up of dedicated community volunteers who work together to build the many pieces that make up the Kubernetes releases you rely on.

This requires the specialized skills of people from all corners of our community, from the code itself to its documentation and project management.

We would like to thank the entire release team for the hours spent hard at work to deliver the Kubernetes v1.37 release to our community.

The Release Team's membership ranges from first-time shadows to returning team leads with experience forged over several release cycles.

A very special thanks goes out to our release lead, Dipesh Rawat, for supporting us through a successful release cycle, advocating for us, making sure that we could all contribute in the best way possible, and challenging us to improve the release process.

Project velocity

The CNCF K8s DevStats project aggregates a number of interesting data points related to the velocity of Kubernetes and various sub-projects.

This includes everything from individual contributions to the number of companies that are contributing and is an illustration of the depth and breadth of effort that goes into evolving this ecosystem.

In the v1.37 release cycle, which ran for 15 weeks from May 18th, 2026, to August 26th, 2026, contributions to Kubernetes reached a maximum of 212 different companies and 1,754 individuals.

Source for this data:

• Companies contributing to Kubernetes
• Overall ecosystem contributions

By contribution we mean when someone makes a commit, code review, comment, creates an issue or PR, reviews a PR (including blogs and documentation), or comments on issues and PRs.

If you are interested in contributing, check out our getting started page.

Event updates

Explore the upcoming KubeCons worldwide:

• KubeCon + CloudNativeCon China: September 7–9, 2026, in Shanghai, China
• KubeCon + CloudNativeCon North America: November 9–12, 2026, in Salt Lake City, United States

Explore the upcoming Kubernetes Community Days (KCDs) taking place for the rest of 2026:

September 2026

• KCD x Ceph x OpenInfra Day Korea: September 1, 2026, in Seoul, South Korea
• KCD San Francisco Bay Area: September 1, 2026, in Mountain View, United States
• KCD Washington DC: September 15, 2026, in Washington, DC, United States
• KCD Gujarat: September 19, 2026, in Ahmedabad, India
• KCD São Paulo: September 26, 2026, in São Paulo, Brazil
• KCD Sofia: September 29, 2026, in Sofia, Bulgaria

October 2026

• KCD UK – Edinburgh: October 19–20, 2026, in Edinburgh, United Kingdom
• KCD Nigeria: October 24, 2026, in Lagos, Nigeria

November 2026

• KCD Porto: November 19–20, 2026, in Porto, Portugal
• KCD Hangzhou: November 28, 2026, in Hangzhou, China

December 2026

• KCD Suisse Romande: December 9–10, 2026, in Meyrin, Switzerland
• KCD Provence: December 10, 2026, in Aix-en-Provence, France
• KCD Florida – Miami: December 11, 2026, in Miami, United States

You can find the latest event details at the CNCF Events Page.

Upcoming release webinar

Join members of the Kubernetes v1.37 Release Team on Wednesday, September 23rd, 2026 at 4:00 PM (UTC) to learn about the release highlights of this release. For more information and registration, visit the event page on the CNCF Online Programs site.

Get involved

The simplest way to get involved with Kubernetes is by joining one of the many Special Interest Groups (SIGs) that align with your interests.

If you don't know where to start, join our monthly New Contributor Orientations where we teach the community how the project is structured, and we'll guide you on how to make your first contribution to the project.

• Read more on how to become a Kubernetes Contributor
• Read more about what’s happening with Kubernetes on our blog
• Join us on Slack
• Follow us on Bluesky for the latest updates
• Follow us on LinkedIn
• Follow us on X
• Join the community discussion on Discuss
• Post questions (or answer questions) on Stack Overflow
• Share your Kubernetes End User Story
• Learn more about the Kubernetes Release Team

YAML has been the standard way to write Kubernetes manifests for years. Every example, tutorial, and configuration file you come across is written in it. The problem isn't that YAML is a bad format. It's that YAML gives you a lot of choices, and not all of them are equally good for writing Kubernetes manifests. Some features make files harder to read, some are easy to misuse and others can lead to surprising behavior.

The interesting part is that Kubernetes doesn't actually need most of those features. It only relies on a small subset of YAML. This led to a simple question: if Kubernetes only needs a small part of YAML, why not standardize on that part and avoid the rest? Instead of introducing a new configuration language, SIG CLI introduced KYAML, a stricter, more consistent way to write YAML.

What is KYAML?

KYAML is a strict subset (or "dialect") of standard YAML, designed to be parseable by the existing ecosystem without any changes, as proposed in KEP 5295. It does not introduce a new format or a new parser. It just narrows the scope of choices you make when writing YAML, so everyone ends up making the same ones.

Think of it less like a new language and more like an agreed-upon style. Everything valid in KYAML is valid YAML.

How KYAML solves it

Standard YAML has a few well-known traps and JSON is not without its own.

Whitespace sensitivity. Indentation defines structure in YAML, which means a wrongly indented file can remain syntactically valid while representing a different object than intended. This gets especially painful with templating tools like Helm, where you are manipulating indentation from outside the YAML context.

Silent type coercion. String quoting is optional in YAML, which sounds convenient until it is not. Some values that look like strings get coerced into other types without warning. The classic example is the "Norway Bug".

country: NO

In standard YAML, NO is parsed as a boolean false, not the string "NO" and it has caught more than a few people off guard.

JSON is not the answer either. It lacks comment support, is strict about trailing commas, and requires every key to be quoted, none of which makes for a good config writing experience.

KYAML addresses all of these by making structure and types explicit:

• Does not depend on whitespace for structure
• Always quotes value strings so no silent type coercion
• Always uses {} for maps and structs
• Always uses [] for lists
• Allows comments and trailing commas, unlike JSON
• Includes a --- header to distinguish it from JSON at a glance, since both start with {

YAML calls this flow style, as opposed to the conventional block style most people use. KYAML sits halfway between JSON and YAML, more explicit than default YAML, friendlier than JSON.

Here is the same Pod manifest written in both formats for comparison.

Standard YAML

apiVersion: v1
kind: Pod
metadata:
 name: my-pod
 labels:
 app: demo
spec:
 containers:
 - name: nginx
 image: nginx:1.20

KYAML

---
{
 apiVersion: "v1",
 kind: "Pod",
 metadata: {
 name: "my-pod",
 labels: {
 app: "demo",
 },
 },
 spec: {
 containers: [{
 name: "nginx",
 image: "nginx:1.20",
 }],
 },
}

Notice the double-quoted string values, the braces around every mapping, the brackets around the list and the trailing commas. The additional syntax makes the document structure explicit instead of relying on indentation.

How to pretty print YAML as KYAML

There are different ways to get KYAML output.

Option 1: kubectl -o kyaml

Since Kubernetes 1.34, kubectl supports KYAML as a native output format.

# Kubernetes 1.35+ (beta; feature enabled by default, still requires -o kyaml CLI param)
kubectl get deployment my-app -o kyaml

# Kubernetes 1.34 (alpha, opt-in)
export KUBECTL_KYAML=true
kubectl get deployment my-app -o kyaml

To save the output to a file:

kubectl get deployment my-app -o kyaml > my-app.yaml

There are currently no plans to make KYAML the default output format. If you prefer using KYAML by default, you can configure your preferred default with kuberc. For more details, see the kuberc documentation.

# Kubernetes 1.36+
kubectl kuberc set --section defaults --command get --option output=kyaml

# Kubernetes 1.33–1.35 (alpha prefix still required)
kubectl alpha kuberc set --section defaults --command get --option output=kyaml

Option 2: Kubernetes' yamlfmt

sigs.k8s.io/yaml ships a yamlfmt tool that can convert files to KYAML.

Install via Go:

go install sigs.k8s.io/yaml/yamlfmt@latest

Running it against a file prints the KYAML version to stdout. It also accepts a directory, in which case it converts and prints every file in that directory. So you'll need to redirect the output to a file (or files) if you want the conversion to stick.

yamlfmt -o=kyaml my-deployment.yaml

It can also show you a diff instead of a full conversion:

yamlfmt -o=kyaml -d my-deployment.yaml

Option 3: Google's yamlfmt

For converting existing files, Google's yamlfmt added a dedicated kyaml formatter in v0.21.0.

Install via Go, or grab a binary from the releases page:

go install github.com/google/yamlfmt/cmd/yamlfmt@latest

It is also available as a pre-commit hook and as a Docker image for CI pipelines.

Add a .yamlfmt config to your project root:

formatter:
 type: kyaml

Preview the output without modifying your file:

yamlfmt -dry my-deployment.yaml

then apply:

yamlfmt my-deployment.yaml

To convert an entire directory:

yamlfmt ./k8s/

The kyaml formatter takes no additional configuration and does not share options with the default formatter so mixing them will cause an error.

For more on the available modes and flags, check the command usage docs.

Is KYAML worth adopting?

Every valid KYAML file is a valid YAML file. So whatever you write in KYAML, your existing tools, your kubectl, your CI pipelines, none of them need to change. You can even pass KYAML as input to any version of kubectl, not just 1.34+, because at the end of the day it is just YAML.

KYAML is not strictly necessary. You can keep writing block-style YAML and things will work. But it is a deliberate choice to make your configs less error-prone and more consistent especially across a team or a larger repo.

It is less of a migration and more of a better habit.

The Kubernetes SIG Network community is thrilled to share the release of Gateway API v1.6.0, which was released on June 30th of this year!

Gateway API has become the standard for modern, role-oriented, and expressive service networking in Kubernetes. In previous releases, Gateway API established a production-grade foundation for HTTP and TLS layer 7 traffic. With version 1.6.0, Gateway API takes a major step forward by expanding standard layer 4 protocol routing and introducing cleaner API boundaries for experimental innovation.

Here is a quick summary of what's new in Gateway API v1.6.0:

• TCPRoute and UDPRoute Graduate to Standard: Raw L4 TCP and UDP traffic routing reach GA stability in the v1 API version.
• Experimental API Group Separation: Experimental resources transition to a distinct API group (gateway.networking.x-k8s.io) with an X prefix to make experimental vs. standard boundaries crystal clear.

Let's dive into the details!

TCPRoute and UDPRoute graduate to Standard

Leads: Nick Young, Ricardo Katz and Zac Nixon

• GEP-2644 - TCPRoute
• GEP-2645 - UDPRoute

Until now, Gateway API only offered a stable routing model for HTTP and TLS traffic. Workloads that speak a raw protocol over TCP or UDP - databases, DNS, VoIP, gaming, IoT telemetry - had no portable way to plug into a Gateway. Users either fell back to a plain Kubernetes Service, or to an implementation-specific CRD that doesn't travel between Gateway controllers.

TCPRoute and UDPRoute close that gap: they route traffic to backends based on protocol and port alone, no L7 awareness required. With this release, both have graduated from the Experimental channel to Standard, and moved to the v1 API version. The v1alpha2 version of each was deprecated as of the v1.6 release, and will be removed in a future release.

How it works

A Gateway needs a listener that allows TCPRoute attachment:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: example-gateway
spec:
 gatewayClassName: example-gateway-class
 listeners:
 - name: foo
 protocol: TCP
 port: 12345
 allowedRoutes:
 kinds:
 - kind: TCPRoute

A TCPRoute then attaches to that listener and forwards traffic to a backend:

apiVersion: gateway.networking.k8s.io/v1
kind: TCPRoute
metadata:
 name: tcp-app
spec:
 parentRefs:
 - name: example-gateway
 sectionName: foo
 rules:
 - backendRefs:
 - name: my-foo-service
 port: 6000

Traffic arriving on the Gateway's port 12345 is proxied to the endpoints of my-foo-service on port 6000. Omitting sectionName and port from parentRefs attaches the route to every TCP listener on the Gateway instead of a single one.

UDPRoute follows the same pattern; swap the listener protocol and the route kind:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: example-gateway
spec:
 gatewayClassName: example-gateway-class
 listeners:
 - name: foo
 protocol: UDP
 port: 12345
 allowedRoutes:
 kinds:
 - kind: UDPRoute
---
apiVersion: gateway.networking.k8s.io/v1
kind: UDPRoute
metadata:
 name: udp-app
spec:
 parentRefs:
 - name: example-gateway
 sectionName: foo
 rules:
 - backendRefs:
 - name: my-foo-service
 port: 6000

XBackend arrives in Experimental

Leads: Keith Mattix II

• GEP-4894 - Backend Resource

Gateway API v1.6 introduces the new XBackend resource, which is a general-purpose decorator for Service (and other backend types) within Gateway API.

The Service resource is an amazing, stable, and flexible object, but that comes with some costs: The flexibility creates a lot of edge cases that Gateway API needs to handle, and the stability makes it impossible to add new concepts to Service.

The XBackend resource builds on the ideas in the upstream EndpointSelector KEP, to add a Gateway API-native object that still targets the backend app, while allowing the community to extend it to handle use cases that are difficult or dangerous to handle with Service.

The first version of XBackend includes support for ExternalHostname destinations, which are ruled out from Service support in Gateway API because of the possibility of confused deputy attacks.

For XBackend, this support is an Extended/Optional feature, allowing implementations and users to opt in once they understand the security tradeoffs.

This support is very useful for egress use cases (which are most commonly used for cluster-hosted agentic workloads), which the community is also working towards formalizing in GEPs about Gateways for Egress (work in progress, stay tuned!)

The XBackend API is experimental and its behavior can change, do not assume it is ready for production

An example of a Gateway with an ExternalName backend that can be used for egress to a cloud AI API is as follows:

# Gateway-level TLS remains authoritative for incoming connections
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
spec:
 listeners:
 - name: https
 protocol: HTTPS
 tls:
 certificateRefs:
 - name: gateway-cert
---
# Backend resource for external destination
apiVersion: gateway.networking.x-k8s.io/v1alpha1
kind: XBackend
metadata:
 name: ai-provider-api
 namespace: ai-apps
spec:
 type: ExternalHostname
 externalHostname:
 hostname: api.ai-provider.com

---
# HTTPRoute referencing XBackend
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
spec:
 rules:
 - backendRefs:
 - name: ai-provider-api
 kind: XBackend
 group: gateway.networking.x-k8s.io