Problem Statement
Many organizations treat the built‑in Kubernetes scheduler as a “set‑and‑forget” component. It works well for batch jobs, web services, and typical CRUD workloads, but it was never designed with sub‑millisecond pod placement guarantees in mind. When a workload demands deterministic network latency—such as high‑frequency trading, real‑time video analytics, or edge AI inference—the default scheduler can silently add seconds of jitter by ignoring node‑level latency metrics and by favoring generic resource balance over proximity to specialized hardware.
Why the Stock Scheduler Falls Short
The default scheduler follows a three‑phase algorithm: filter, score, and bind. The filter phase discards nodes that lack required resources (CPU, memory, GPUs). The scoring phase applies a handful of generic predicates (e.g., pod affinity, taints, image locality). None of these steps consider network round‑trip time, NUMA latency, or hardware accelerator proximity. As a result, a pod that needs a NIC with <10 µs latency may land on a node whose NIC is several hops away, causing the application to miss its latency SLA without any visible error in the cluster.
Moreover, the scheduler runs as a single control‑plane process. Under heavy churn—hundreds of pods per second—its internal queue can become a bottleneck, adding additional scheduling latency that propagates to the pods themselves.
# Example of a pod that suffers from hidden latency
apiVersion: v1
kind: Pod
metadata:
name: latency‑critical‑worker
annotations:
scheduler.alpha.kubernetes.io/affinity: "true"
spec:
containers:
- name: worker
image: myregistry/ultra‑low‑latency:1.0
resources:
limits:
cpu: "2"
memory: "2Gi"
nvidia.com/gpu: "1"
nodeSelector:
disktype: ssd
The pod above requests a GPU and SSD storage, but nothing tells the scheduler that it must also be placed on a node whose NIC latency is below 10 µs. The default scheduler happily schedules it on any node that meets the generic criteria, potentially violating the latency contract.
Designing a Custom Scheduler for Latency Guarantees
Kubernetes allows you to run additional scheduler binaries alongside the default one. A custom scheduler can read a new pod annotation, latency‑target‑us, and filter nodes based on a NodeLatency custom resource that reflects measured NIC latency to the target data center. The following sections walk through a minimal implementation in Go, using the client-go library.
// main.go – entry point for the custom scheduler
package main
import (
"context"
"flag"
"fmt"
"time"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/fields"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/tools/clientcmd"
)
var (
kubeconfig = flag.String("kubeconfig", "", "Path to kubeconfig file")
schedulerName = flag.String("scheduler-name", "latency‑aware‑scheduler", "Name of the custom scheduler")
)
func main() {
flag.Parse()
config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
if err != nil {
panic(err)
}
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
panic(err)
}
// Watch only unscheduled pods that request our scheduler
podWatcher := cache.NewListWatchFromClient(
clientset.CoreV1().RESTClient(),
"pods",
v1.NamespaceAll,
fields.ParseSelectorOrDie(fmt.Sprintf("spec.schedulerName=%s,status.phase=Pending", *schedulerName)),
)
_, controller := cache.NewInformer(
podWatcher,
&v1.Pod{},
0,
cache.ResourceEventHandlerFuncs{
AddFunc: func(obj interface{}) {
pod := obj.(*v1.Pod)
go schedulePod(clientset, pod)
},
},
)
stopCh := make(chan struct{})
defer close(stopCh)
go controller.Run(stopCh)
// Block forever
select {}
}
func schedulePod(clientset *kubernetes.Clientset, pod *v1.Pod) {
target, ok := pod.Annotations["latency‑target‑us"]
if !ok {
fmt.Printf("Pod %s has no latency target, skipping\\n", pod.Name)
return
}
fmt.Printf("Scheduling pod %s with latency target %sµs\\n", pod.Name, target)
// List nodes and pick the first that satisfies the latency metric
nodes, err := clientset.CoreV1().Nodes().List(context.TODO(), v1.ListOptions{})
if err != nil {
fmt.Printf("Node list error: %v\\n", err)
return
}
for _, node := range nodes.Items {
// Assume a label nodeLatencyUs=XX is maintained by a monitoring daemon
if latencyStr, ok := node.Labels["nodeLatencyUs"]; ok {
// Simple integer comparison
var nodeLatency int
fmt.Sscanf(latencyStr, "%d", &nodeLatency)
var targetLatency int
fmt.Sscanf(target, "%d", &targetLatency)
if nodeLatency <= targetLatency {
// Bind pod to this node
bind := &v1.Binding{
ObjectMeta: v1.ObjectMeta{
Name: pod.Name,
Namespace: pod.Namespace,
},
Target: v1.ObjectReference{
Kind: "Node",
Name: node.Name,
},
}
err = clientset.CoreV1().Pods(pod.Namespace).Bind(context.TODO(), bind, v1.CreateOptions{})
if err != nil {
fmt.Printf("Bind error: %v\\n", err)
} else {
fmt.Printf("Pod %s bound to node %s\\n", pod.Name, node.Name)
}
return
}
}
}
fmt.Printf("No suitable node found for pod %s\\n", pod.Name)
}
The code above implements a minimalist scheduler that respects a latency‑target‑us annotation. It relies on a node label (nodeLatencyUs) that must be kept up‑to‑date by an external latency‑measurement daemon. If no node satisfies the constraint, the pod remains pending, making the latency requirement explicit rather than hidden.
Deploying the Custom Scheduler
Package the binary into a Docker image and run it as a static pod on every control‑plane node. The static pod manifest ensures the scheduler starts before the default one and can be given a higher priority for the same scheduling queue.
# scheduler‑deployment.yaml – static pod manifest
apiVersion: v1
kind: Pod
metadata:
name: latency‑aware‑scheduler
namespace: kube-system
spec:
containers:
- name: scheduler
image: myregistry/latency‑scheduler:latest
args:
- "--scheduler-name=latency‑aware‑scheduler"
- "--kubeconfig=/etc/kubernetes/kubelet.conf"
volumeMounts:
- name: kubeconfig
mountPath: /etc/kubernetes
hostNetwork: true
restartPolicy: Always
volumes:
- name: kubeconfig
hostPath:
path: /etc/kubernetes/kubelet.conf
After applying the manifest, verify that the scheduler appears in the API server:
# kubectl get pods -n kube-system | grep latency‑aware‑scheduler
latency‑aware‑scheduler 1/1 Running 0 2m