The rise of on‑demand GPU instances has turned Kubernetes clusters into a common platform for training models, rendering video, and running inference at scale. Yet many organizations treat autoscaling as a plug‑and‑play feature, copying the same HorizontalPodAutoscaler (HPA) policies they use for pure‑CPU workloads. The result is a cascade of hidden problems that surface only after the first production spike. This article explains why basing GPU autoscaling decisions on CPU utilization alone is a strategic mistake, outlines the internal mechanics that make the approach brittle, and offers concrete guidance on building a more reliable scaling surface.

The intuitive but flawed assumption

At first glance, CPU usage appears to be a universal indicator of load. The Kubernetes HPA controller watches the resource.metrics.k8s.io/v1beta1 API for cpu percentages, computes a desired replica count, and the cluster obeys. When a pod also requests a GPU, the same controller continues to emit replica recommendations based on the same CPU signal. The underlying belief is simple: if the CPU is busy, the pod must be doing work, therefore more replicas are needed. Unfortunately, that belief collapses under three distinct realities.

1. GPU workloads often idle the CPU

Modern deep‑learning frameworks offload the heavy matrix operations to the GPU driver, leaving the host CPU to orchestrate data movement and kernel launches. In a typical inference request, the CPU spends less than 5 % of a core’s time preparing tensors, while the GPU consumes the bulk of the latency budget. As request volume climbs, the CPU metric may remain flat, but the GPU queue length grows dramatically. Autoscaling that watches only the CPU will see no trigger, allowing the GPU queue to saturate and response times to explode.

2. Resource fragmentation on shared nodes

Most cloud providers expose GPUs as whole devices attached to a node. A node can host multiple GPU‑enabled pods, but each pod usually requests nvidia.com/gpu: 1. If the HPA adds replicas based on CPU, the scheduler may pack additional pods onto a node that already hosts a full complement of GPUs, causing the scheduler to place the new pods on nodes without GPUs. Those pods then fail to start, generate back‑off errors, and pollute logs with “Insufficient GPU” messages. The cluster appears healthy from a CPU perspective while the GPU layer is effectively deadlocked.

3. Cost amplification through over‑provisioning

Cloud GPU instances carry a premium price tag, often three to ten times the cost of a comparable CPU‑only VM. When autoscaling reacts to a fleeting CPU spike—perhaps caused by a background maintenance job—the controller may spin up an entire new GPU node. If the GPU remains idle because the spike was unrelated to GPU work, the organization pays for a costly resource that delivers no value. Over time, these “ghost” nodes inflate the monthly bill without contributing to throughput.

What actually happens inside the control loop

Understanding the internal flow helps illustrate why the problem is not just a configuration oversight but a systemic mismatch. The HPA controller follows these steps on each reconciliation loop:

  1. Query the metrics API for the specified resource (e.g., cpu).
  2. Calculate the desired replica count using the formula desired = ceil[current_replicas * (current_metric / target_metric)].
  3. Patch the Scale subresource of the target Deployment or StatefulSet.
  4. The ReplicaSet controller creates or deletes pods to match the new replica count.
  5. The scheduler binds each new pod to a node that satisfies its resourceRequests and affinity rules.

Notice that nowhere in this pipeline does the controller ask “Does this pod need a GPU right now?” The decision is made exclusively on the metric you expose. If you expose only CPU, the controller never sees the GPU pressure that actually dictates performance. The result is a feedback loop that amplifies the three failure modes described earlier.

Alternative signals that actually reflect GPU demand

The Kubernetes ecosystem now offers several first‑class metrics that capture GPU utilization:

  • GPU Core Utilization – Exposed via the nvidia.com/gpu custom metrics API. It reports the percentage of each GPU’s SMs that are active.
  • GPU Memory Usage – Indicates how much VRAM is allocated, useful for workloads that perform large batch inference.
  • GPU Queue Length – Provided by the NVIDIA DCGM exporter, this metric shows how many kernels are waiting to execute, a direct proxy for contention.
  • Inference Latency Percentiles – By instrumenting the application with OpenTelemetry, you can surface request‑level latency as a custom metric and let the autoscaler react to SLA breaches.

Pairing these signals with a VerticalPodAutoscaler (VPA) or a custom controller that adjusts replicas based on GPU pressure yields a far more predictable scaling surface.

Designing a robust autoscaling strategy for GPU workloads

The following pattern has emerged from teams that have run multi‑tenant GPU clusters at scale for several years:

  1. Separate Autoscaling Policies – Deploy two HPAs per workload: one watching CPU for auxiliary tasks (e.g., pre‑processing) and another watching GPU metrics for the core compute path.
  2. Use a Custom Metrics Adapter – Install the prometheus-adapter or the NVIDIA dcgm-exporter to surface GPU metrics to the Kubernetes metrics API.
  3. Define a Minimum GPU Node Pool – Reserve at least one GPU node in each zone to avoid cold‑start delays when the scaler decides to add pods.
  4. Guard Against Over‑Provisioning – Apply a scaleDownDelay of at least 10 minutes and a scaleUpDelay of 2 minutes to smooth transient spikes.
  5. Implement Pod‑Level Anti‑Affinity – Use podAntiAffinity rules to spread GPU‑using pods across nodes, preventing a single node from becoming a hotspot.
  6. Cost‑Aware Scaling – Leverage the cloud provider’s spot‑instance pricing for GPU nodes and configure the scaler to prefer spot capacity when the queue length stays below a configurable threshold.

Case study: A media rendering pipeline

A streaming service migrated its video transcoding jobs from a pure‑CPU cluster to a mixed CPU/GPU Kubernetes fleet. Initially, they kept the existing CPU‑centric HPA configuration and observed occasional spikes in rendering latency. After three weeks of troubleshooting, the ops team discovered that the GPU queue length routinely hit 80 % while CPU stayed under 30 %. By introducing a second HPA that watched nvidia.com/gpu_utilization, the cluster began adding GPU nodes before the queue saturated. Latency dropped by 45 %, and the cost increase was limited to a 12 % rise in monthly spend because the scaler now added nodes only when the GPU metric crossed the 70 % threshold.

Potential pitfalls when switching to GPU‑aware scaling

While the benefits are clear, teams should be aware of two implementation traps:

  • Metric Staleness – GPU metrics are often collected at a lower frequency than CPU metrics. If the scrape interval is too long, the scaler may react late, re‑introducing latency. Tune the Prometheus scrape interval to at most 15 seconds for latency‑sensitive workloads.
  • Metric Mis‑alignment – Different frameworks report GPU utilization differently. For example, TensorFlow may show high utilization even when the model is waiting on data I/O. Correlate GPU metrics with application‑level latency to avoid scaling on false positives.

Conclusion

Autoscaling remains one of the most powerful features of Kubernetes, but its effectiveness hinges on the relevance of the signal you feed it. When the workload’s bottleneck lives on the GPU, a CPU‑only view is not just incomplete—it actively harms performance, reliability, and cost efficiency. By exposing GPU‑specific metrics, separating scaling policies, and respecting the unique scheduling constraints of accelerator hardware, you can turn a brittle system into a predictable, self‑healing platform that truly matches capacity to demand.

The hidden cost of ignoring GPU pressure is not merely a few extra dollars; it is lost user experience, wasted engineering time, and a cloud bill that spirals out of control. The next time you draft an autoscaling policy, ask yourself whether the metric you are watching actually represents the resource that is scarce. If the answer is “no,” redesign the control loop before the problem becomes visible in production.