Why the “smart” autoscaler isn’t always smart

Many cloud‑native teams have embraced AI‑powered autoscaling extensions that claim to predict GPU demand based on historic usage patterns. The promise is alluring: automatically spin up a nvidia.com/gpu node when a spike is detected, then tear it down as soon as the workload settles. In practice, the algorithm often over‑reacts to transient spikes, inflates the pod‑count, and leaves expensive GPU instances running idle for hours.

The hidden liability isn’t just the extra dollar bill; it’s the impact on latency‑sensitive inference jobs that share the same node pool. A sudden, unnecessary GPU allocation can starve critical pods, leading to missed SLA windows and degraded model accuracy (e.g., batch‑size mismatches). The following tutorial shows how to replace a black‑box AI autoscaler with a deterministic, rule‑based controller that respects both cost caps and latency budgets.

Understanding the baseline – the default Horizontal Pod Autoscaler (HPA)

Before adding any AI layer, you need a solid baseline. The standard HPA can scale CPU‑ or memory‑bound workloads, but it does not understand GPU metrics out of the box. We will first expose GPU utilization via the kube‑metrics‑server and then use a CustomMetric object that the HPA can consume. 

# Deploy the NVIDIA device plugin
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      containers:
      - name: nvidia-device-plugin-ctr
        image: nvidia/k8s-device-plugin:latest
        securityContext:
          privileged: true
        env:
        - name: FAIL_ON_INIT_ERROR
          value: "false"
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

With the device plugin running, install kube‑metrics‑server and enable the GPU collector: 

# kube‑metrics‑server deployment (partial)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server/metrics-server:v0.6.2
        args:
        - --kubelet-insecure-tls
        - --metric-resolution=15s
        - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
        - --requestheader-allowed-names=system:serviceaccount:kube-system:metrics-server
        - --gpu-collector-enabled=true   # <‑‑ enable GPU metrics

Verify that GPU utilization appears in the metrics API: 

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/gpu_utilization"
{
  "items": [
    {
      "metricName": "gpu_utilization",
      "timestamp": "2026-08-24T12:00:00Z",
      "value": "23"
    }
  ]
}

Implementing a deterministic rule‑based scaling controller

The controller runs as a sidecar in the kube‑system namespace, watches the custom metric, and decides whether to add or remove GPU nodes based on three configurable thresholds:

  • UpperUtil – if average GPU utilization > 80% for 5 minutes, consider scaling up.
  • LowerUtil – if average GPU utilization < 20% for 10 minutes, consider scaling down.
  • CostCap – never exceed a budget of $2,500 per day for GPU instances.

Below is a minimal Python controller that uses the kubernetes client library. It demonstrates the core logic; production code should add retries, logging, and leader election. 

#!/usr/bin/env python3
import os
import time
from datetime import datetime, timedelta

from kubernetes import client, config

# Load in‑cluster config
config.load_incluster_config()
v1 = client.CoreV1Api()
custom = client.CustomObjectsApi()

# Configurable thresholds
UPPER_UTIL = 80      # percent
LOWER_UTIL = 20      # percent
COST_CAP_DAILY = 2500  # USD

# Helper to fetch average GPU utilization across the cluster
def get_average_gpu_util():
    resp = custom.list_cluster_custom_object(
        group="custom.metrics.k8s.io",
        version="v1beta1",
        plural="pods",
        label_selector="app=gpu-workload"
    )
    values = [int(item["value"]) for item in resp.get("items", [])]
    return sum(values) / len(values) if values else 0

# Helper to compute daily GPU cost from node labels (simplified)
def get_daily_gpu_cost():
    nodes = v1.list_node(label_selector="cloud.google.com/gke-accelerator")
    total = 0
    for n in nodes.items:
        # Assume $3 per GPU‑hour, 24 h per day
        gpu_count = int(n.metadata.labels.get("cloud.google.com/gke-accelerator-count", "0"))
        total += gpu_count * 3 * 24
    return total

def scale_up():
    # Example: add a node‑pool via GKE API (placeholder)
    print("[INFO] Scaling up – requesting new GPU node")
    # In real world you would call GKE REST API or use Terraform

def scale_down():
    # Example: cordon and delete the least‑utilized node
    nodes = v1.list_node()
    gpu_nodes = [n for n in nodes.items if "nvidia.com/gpu" in n.status.allocatable]
    if not gpu_nodes:
        return
    target = min(gpu_nodes, key=lambda n: int(n.status.allocatable["nvidia.com/gpu"]))
    name = target.metadata.name
    print(f"[INFO] Scaling down – deleting node {name}")
    v1.delete_node(name)

def main():
    high_util_start = None
    low_util_start = None

    while True:
        avg_util = get_average_gpu_util()
        daily_cost = get_daily_gpu_cost()
        now = datetime.utcnow()

        print(f"[{now}] Avg GPU Util: {avg_util:.1f}% | Daily Cost: ${daily_cost}")

        # Scale‑up logic
        if avg_util