AI infrastructure conversations often start with GPUs. Accelerators provide much of the compute behind model training and inference, so the focus is understandable.

But a production AI workload rarely starts and ends on a GPU. Data needs to be prepared and moved. Applications and orchestration services need to run. Models need to be loaded and served. Results may require additional processing.

Platform teams are therefore not simply managing GPU workloads. They are managing heterogeneous workloads that depend on CPU, GPU, memory, storage, and networking working together. For Kubernetes platform teams, the challenge is not just providing accelerators. It is matching the right resources to each stage of the workload.

Follow the workload, not the GPU

Consider a simplified AI inference pipeline:

Data → CPU preprocessing → GPU inference → CPU post-processing → application

The GPU may perform the most compute-intensive step, but overall performance depends on the complete path. If preprocessing cannot supply data quickly enough, the accelerator waits. If storage cannot deliver model artifacts efficiently, startup slows. If CPU, memory, or network capacity becomes constrained, adding more GPU capacity may do little to improve throughput. 

Instead of asking: How many GPUs does this workload need?, platform teams should ask:

What resources does each stage need, and where are the dependencies between them?

That shift helps teams optimize the workload as a system rather than optimizing one expensive component in isolation.

Match resources to the work

Different stages of an AI workload have different infrastructure requirements.

CPU resources can handle data preparation, tokenization, retrieval, orchestration, application logic, and post-processing. GPUs and other accelerators are suited to highly parallel operations such as model training and inference. Memory, storage, and networking determine how efficiently data and model artifacts move between these stages.

Even inference itself is not necessarily one uniform workload. For large language models, prompt processing and token generation can have different compute and memory requirements. This creates an opportunity for platform teams to match resources to the work rather than forcing an entire AI pipeline onto a single infrastructure profile.

Kubernetes provides a common orchestration layer for doing this. Dynamic Resource Allocation (DRA), for example, extends Kubernetes’ resource model by providing a more flexible, declarative way for workloads to request specialized devices. The important point is not DRA itself. It is the direction: specialized compute is increasingly part of the same cloud-native resource model as the rest of the application.

Observe the handoffs

Heterogeneous infrastructure also changes what platform teams need to observe. GPU utilization alone does not tell you whether an AI workload is running efficiently. Low GPU utilization could indicate insufficient demand. But it could also mean the accelerator is waiting for CPU preprocessing, data access, scheduling, or another upstream dependency.

Platform teams, therefore, need visibility across the complete workload:

CPU → data → accelerator → application

Correlating infrastructure and application telemetry makes it easier to identify where time is being spent and which resource is limiting performance. The objective isn’t to keep every resource at 100% utilization. It is to understand whether those resources are working together efficiently enough to meet the workload’s performance requirements.

Design for the whole system

As AI workloads move into production, infrastructure is likely to become more heterogeneous, not less. Kubernetes provides platform teams with a common control plane across these resources, while capabilities such as DRA are expanding the ways specialized hardware can participate in that model.

The key shift is conceptual: AI infrastructure is not a collection of GPUs with supporting services around them. It is a system of interconnected compute, memory, storage, and network resources.

For platform engineers, designing around that complete system, not one component, is what turns accelerator capacity into useful AI infrastructure.

Kubernetes has given platform teams a consistent way to deploy, scale, and operate containerized applications. Now, many of those same teams are being asked to support AI.

The transition is already underway. According to the CNCF 2025 Annual Cloud Native Survey, 66% of organizations hosting generative AI models use Kubernetes for some or all of their inference workloads. Yet only 7% of organizations deploy AI models daily. That gap highlights an important distinction: running AI on Kubernetes and having a Kubernetes platform ready to operate AI continuously are not the same thing.

The challenge is also showing up within platform teams. The 2025 State of AI in Platform Engineering research found that 35% of platform teams still don’t orchestrate AI workloads, pointing to a gap between AI adoption and the operational platforms needed to support it at scale.

AI doesn’t require platform teams to abandon cloud native practices. Kubernetes, GitOps, observability, automation, and self-service remain valuable foundations. But AI introduces new requirements around compute, scheduling, model delivery, and operations. 

So, what needs to change?

Extend the resource model beyond CPU and memory

AI is often discussed as a GPU workload, but production AI pipelines are heterogeneous.

Data preparation, preprocessing, retrieval, orchestration, and application logic may run on CPUs, while training or inference uses GPUs or other accelerators. A single workload may depend on several resource types.

This changes scheduling. Platform teams need to consider accelerator type and availability alongside CPU, memory, topology, and workload characteristics.

Kubernetes is evolving to support this. Dynamic Resource Allocation (DRA), for example, provides a more flexible, declarative way for workloads to request specialized hardware.

The goal isn’t simply to make GPUs available. It is to make a heterogeneous compute part of a consistent Kubernetes resource model.

Extend CI/CD to the model lifecycle

Cloud native teams have made application delivery repeatable through CI/CD and GitOps. AI adds another critical artifact: the model. 

Instead of managing only:

Code → build → test → deploy

teams may now need to manage:

Code + model + configuration → evaluate → deploy → observe → update

Models can be large, depend on specific runtimes or hardware, and require evaluation before deployment. Platform teams, therefore, need a clear way to answer the following questions: Which application, model, and configuration are running, and can we reproduce the deployment?

The principle is familiar: changes should remain versioned, repeatable, and auditable. The delivery pipeline simply needs to account for more than application code.

Observe the workload, not just the cluster

Traditional infrastructure metrics remain important, but CPU, memory, and request latency don’t tell the whole story for AI.

Depending on the workload, teams may also need visibility into accelerator utilization and memory usage, scheduling and queue times, inference latency, model loading time, throughput, and endpoint health.

​

The important point isn’t to create a separate monitoring stack for AI. It is to extend existing cloud-native observability so that infrastructure-, application-, and AI-specific telemetry can be correlated across the workload.

That makes it easier to answer a question that GPU utilization alone cannot: Where is the workload actually waiting?

Give AI developers a golden path

AI developers shouldn’t need to become Kubernetes infrastructure experts to deploy a model.

Platform teams can provide standardized self-service paths that encode common infrastructure and operational decisions:

Model → resources → deployment → endpoint → observability → policy

Developers specify what the workload needs; the platform provides a repeatable implementation.

This is the same platform engineering principle that helped simplify cloud native application delivery. The difference is that the golden path now needs to understand models and accelerators alongside containers, CPU, and memory.

Make AI another production workload

AI introduces new resource types and lifecycle requirements, but many of the underlying operational challenges are familiar.

The cloud native community already has strong patterns for orchestration, declarative infrastructure, automated delivery, observability, policy, and developer self-service. The opportunity is to extend those practices rather than create a parallel operational model for AI.

A Kubernetes platform becomes AI-ready when teams can deploy AI workloads consistently, observe them end-to-end, allocate the right resources, and provide developers with a consistent way to move from experimentation to production.

The question is no longer whether Kubernetes can run AI. It is whether operating AI on Kubernetes can become as routine as operating any other production workload.