Open source · Go, Python · MIT

llm-inference-platform

A Kubernetes control plane for serving large language models. A Go operator reconciles anInferenceService custom resource into a running vLLM deployment, and the gateway in front of it routes on data classification before anything else.

Why autoscaling on GPU utilisation is wrong

The default instinct is to scale inference pods on GPU utilisation, the way you would scale a web service on CPU. It fails, and it fails in the worst direction. Token decode is bound by memory bandwidth, not by compute. Under load the queue grows and latency climbs while utilisation stays flat or drops, because the device is waiting on memory rather than burning cycles. An autoscaler watching utilisation does exactly what it was told and scales the deployment down at the moment it is most needed.

This platform scales on queue depth and p95 time-to-first-token through KEDA. Those are the signals that actually move when a model is saturating.

Design decisions

• The operator never writes replicas. It sets them once at creation and never again, so it cannot fight the autoscaler over the same field.
• Weight loading gets a startup probe. Without one, the liveness probe fires mid-load and the container restarts forever, with a clean application log and no error to find.
• Data classification routes first. A request marked Restricted has fewer legal destinations than a Public one. That is a constraint, not a preference, so it filters before capability, latency and cost are considered.
• CEL admission rules enforce the classification at the API server, so an invalid spec is rejected rather than reconciled.
• GPUs are claimed through DRA, Kubernetes dynamic resource allocation, rather than through the older device-plugin resource counting.

There is a longer write-up of the autoscaling problem inthe blog post on queue depth versus GPU utilisation. The operator discipline comes from Oracle Verrazzano.