Your GPU Looks Idle While the Queue Builds

I'd been meaning to do this properly for a while. Not "run a model in a pod" — actually build the control plane: a Go operator that takes a declarative InferenceService and reconciles it into a running vLLM deployment, autoscales it, puts a gateway in front that knows what things cost.

On a laptop. kind cluster, a 4 GB consumer GPU, Ollama on the host as the cheap backend. The constraint was deliberate — 4 GB means you find out immediately when a decision is wrong.

Three things caught me out. Here they are.

The autoscaler was scaling the wrong way

I'll start with this one because it's the one that made me sit back and stare at the graph.

Decode is memory-bandwidth bound. The GPU spends its time shuttling weights and KV cache around, not saturating compute. Which means during decode the card looks unbusy. Utilisation sits low. Requests pile up behind it anyway.

Now put an HPA on CPU in front of that.

Queue grows. Latency climbs. Utilisation stays flat, or drops — the pod's waiting on memory, not burning cycles — and the autoscaler, doing exactly what you told it, scales down.

I want to be clear that this isn't a tuning problem. You can't fix it with a better threshold. The metric moves the wrong way relative to the thing you care about, so the control loop is inverted. Scaling down under load is the correct behaviour for the signal it was given.

Queue depth is the signal. vLLM publishes vllm:num_requests_waiting and KEDA will scale on it directly:

triggers:
  - type: prometheus
    metadata:
      query: vllm:num_requests_waiting   # not CPU. not DCGM utilisation.
      threshold: "4"
  - type: prometheus
    metadata:
      query: histogram_quantile(0.95, vllm:time_to_first_token_seconds_bucket)
      threshold: "2"

The second trigger is p95 time-to-first-token. I added it because throughput can look fine while the tail is falling apart, and I'd rather the SLO have a vote of its own than be inferred from how much work is getting done.

Now, a correction — the ground shifted while I was building this, and I'd rather say so than leave a tidier claim standing.

"Queue depth, not utilisation" is too blunt once you disaggregate. Prefill is compute-bound. Decode is memory-bound. At real scale they're separate pools with separate signals: decode on queue depth, prefill on compute utilisation. Different bottleneck, different number. llm-d's autoscaler watches per-pod KV cache utilisation and queue depth and hands target replica counts to KEDA; Dynamo's planner runs two loops, one on TTFT and one on inter-token latency.

Mine is one pool, because one 4 GB card is one pool. So the honest version of the rule is narrower than my heading: scale on whatever the bottleneck actually is for that phase. For decode that isn't utilisation, and pointing an HPA at it doesn't make your autoscaling merely suboptimal — it inverts it.

The container that kept dying with a clean log

Weight loading takes minutes. Liveness probes assume seconds. You can see where this goes.

No startupProbe, so liveness fires mid-load, kills the container, container restarts, starts loading again, gets killed again. CrashLoopBackOff.

The part that cost me the time: the application log is clean. Nothing went wrong in the application. The log just stops, mid-load, every time, with no error — which reads like the process vanished rather than like something killed it. I went looking for an OOM. There wasn't one.

startupProbe:
  httpGet: { path: /health, port: 8000 }
  failureThreshold: 60       # from spec.weightLoadSeconds
  periodSeconds: 10          # 10 minutes of grace before liveness takes over

I made the budget a CRD field rather than an annotation. It's model-dependent, and the consequence of getting it wrong is an infinite restart loop that looks like a hardware fault, so it deserves to be a first-class part of the spec rather than a string somebody remembers to set.

Who owns replicas

Shorter one. Both the operator and the autoscaler have opinions about spec.replicas.

Let both of them write it and they argue. Operator reconciles to desired state, KEDA scales up, operator puts it back, pods flap.

So: the operator writes replicas once, at creation, and never touches it again. After that it belongs to the autoscaler. If you're running GitOps you need ignoreDifferences on /spec/replicas too, or Argo joins in as a third party to the argument.

Every operator that lives alongside an autoscaler runs into this. It's worth deciding explicitly rather than finding out from a flapping graph at an unhelpful hour.

Classification at admission, not in the reconciler

The CRD carries a data classification (Public, Internal, Restricted) and CEL admission rules enforce placement against it. A Restricted model doesn't land on a node pool that hasn't declared the matching tier.

That it happens at admission is the whole point. Reject the bad combination before a pod is scheduled and you're done. Catch it in the reconciler and something is already running, and now you're writing remediation for a state that should never have been allowed to exist. I'd rather write the CEL rule.

GPUs get claimed through DRA (resource.k8s.io/v1, GA in 1.34) rather than the old device plugin, which turns the claim into a real scheduling object instead of an opaque count.

Cost goes last

The gateway routes across vLLM on the GPU and Ollama on the host, and Ollama is effectively free at the margin. So the obvious rule is cheapest-first.

The obvious rule is wrong. The order is:

1. Data classification. A Restricted request has fewer legal destinations. Not a preference — filters first.
2. Capability. Can this backend actually serve it. Context length, modality.
3. Latency. Will it hold the SLO.
4. Cost. Of whatever's left, take the cheapest.

Put cost first and you will eventually route regulated data somewhere cheap and be entirely correct about the arithmetic while doing it. Cost is a tiebreak between acceptable options. It is not an input to whether an option is acceptable.

The evening I lost to an OOM that wasn't

Worth writing down, because I'd do it faster next time.

A FlashInfer JIT sampler failure that presents as an out-of-memory error.

Everything in you says: lower gpu_memory_utilization. Shrink max_model_len. Get the KV cache down. I did all three, in that order, watching it fail the same way each time, getting steadily more convinced the card just wasn't big enough.

The memory was never the problem. The sampler was failing to compile and the error surfaced late, wearing someone else's clothes. Turning it off on this hardware fixed it in about a minute.

The lesson isn't about FlashInfer. It's that "OOM on a 4 GB card" is such a plausible story that it will absorb however much time you're willing to give it before you think to question the label.

Was the laptop constraint worth it

Yes, though I'd understand the scepticism.

Most published inference setups assume a datacentre and put the interesting decisions behind a managed service. Four gigabytes of VRAM removes that option. Every choice is explicit because there's no headroom to absorb a bad one — which is precisely why the autoscaling bug showed up on day one instead of in somebody's production incident review.

If you're doing this at scale, use llm-d or Dynamo. The disaggregation and KV-aware routing work is well past what one person should be reimplementing. What neither of them gives you is the understanding of why your autoscaler is pointed at the wrong number, and that turned out to be the expensive part.

Code's at github.com/lake-of-dreams/llm-inference-platform Go, Python, Kubernetes 1.34+, MIT. make verify runs end to end against real vLLM and real Ollama and fails fast if they aren't up. No mocks.