Modelplane v0.6 is out, and most of it is about getting more from the hardware you have. Small models can now share a GPU node rather than each taking a whole one. A fleet can serve on AMD Instinct GPUs and on CPUs alongside NVIDIA GPUs. And a node pool can use GPU capacity you've reserved on GKE or EKS.
Disaggregated prefill/decode serving also works with SGLang now, because you write the router the same way you write your engines. Here's what's new.
Small models share a node
Modelplane's fleet scheduler picks a cluster and a node pool for each model replica. It used to measure a pool's room in nodes, so every pod that claimed a GPU took a whole node, however many of the node's GPUs it used. Qwen3-8B fits on one NVIDIA L4. A pool of two nodes with four L4s each has room for eight replicas of it, but the scheduler placed two.
The scheduler now counts free GPUs. Modelplane installs an agent on every
cluster that reports how many of each pool's nodes have each number of GPUs
free, and the InferenceCluster publishes those counts in its status. This pool
has one node with one of its four L4s free, and can grow by two more nodes:
status:
schedulablePools:
- name: gpu-l4x4-a
nodes: 3 # The most nodes the pool can have.
observedNodes: 1 # The nodes the pool has now, ready or not.
nodesByFreeDevices:
- freeDevices:
gpu.nvidia.com: 1
nodes: 1
devices:
- name: gpu
claim: DRA
driver: gpu.nvidia.com
deviceClassName: gpu.nvidia.com
count: 4The scheduler fits a replica's pods against those counts. Four one-GPU replicas fit one four-GPU node, and a replica with two four-GPU engines fits one eight-GPU node. The fleet scheduler still picks only the cluster and the pool, and each cluster's own scheduler picks the node.
There's nothing to configure. The agent reports on NVIDIA GPU pools today, so AMD and CPU pools still count in whole nodes. Modelplane doesn't request CPU or memory for an engine, so set requests on engines you expect to share a node.
AMD GPUs and CPUs
Each cluster now installs the device software its pools need, based on the
DRA
drivers its InferenceClasses use. A pool of AMD GPUs gets
AMD's GPU operator, and a CPU pool gets
dra-driver-cpu. A
CPU-only cluster doesn't get NVIDIA's GPU operator or DRA driver, which every
cluster used to.
Here's the class from the AMD recipe. It describes a pool of AMD Instinct MI325X nodes:
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
name: amd-mi325x-8x
spec:
description: "BYO 8x AMD Instinct MI325X 256GB"
devices:
- name: gpu
claim: DRA
driver: gpu.amd.com
deviceClassName: gpu.amd.com
count: 8
attributes:
productName: { string: AMD_Instinct_MI325_OAM }
capacity:
memory: { value: "262128Mi" }A ModelDeployment selects AMD GPUs the way it selects NVIDIA ones, with a CEL
expression over the devices a class publishes, so one fleet can serve from
both. Modelplane's telemetry collector reads AMD's metrics exporter too, and
renames its series to the same modelplane_* names it uses for NVIDIA's.
Modelplane installs AMD's software only on clusters you bring yourself, and
doesn't provision AMD GPUs on any cloud yet.
A CPU pool lets you try Modelplane end to end without finding a GPU. An
InferenceClass whose provisioning block omits the accelerator describes one,
and the CPU recipe
serves Qwen2.5-0.5B from it on Civo with vLLM's CPU image. A CPU pool needs
Kubernetes 1.36, because dra-driver-cpu relies on a DRA feature that's on by
default only from that release.
Prefill/decode with SGLang
Disaggregated serving splits a model across two engines. Prefill reads the prompt and builds the KV cache, and decode generates tokens from that cache. A router sends each request to both. On Modelplane that's llm-d's endpoint picker, which picks the pods, and a sidecar in each decode pod that hands the prompt to the prefill pod the picker chose.
The router has to match the engines. SGLang needs a different connector on the sidecar from vLLM. A router tuned for vLLM also lets short or cached prompts skip prefill, and SGLang's decode engine rejects those requests. Modelplane used to configure one router, for vLLM with the NIXL connector.
Now you write a PrefillDecode deployment's router the same way you write its
engines. You pick the endpoint picker's image, flags, and configuration, and put
the sidecar in the decode pod beside the engine. Modelplane runs the picker and
wires it into the cluster's gateway, and passes your configuration through
without interpreting it. Here's an SGLang deployment, trimmed from the
routing docs:
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
name: laguna-pd
namespace: ml-team
spec:
replicas: 1
template:
spec:
modelCacheRef:
name: laguna
serving:
mode: PrefillDecode
router: EndpointPicker
endpointPicker:
template:
spec:
containers:
- name: epp
image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0
# Omitted for brevity.
config:
apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig
plugins:
# SGLang's decode engine can't serve a request alone, so every
# request goes through prefill.
- type: always-disagg-pd-decider
# Omitted for brevity.
engines:
- name: prefill
phase: Prefill
# Omitted for brevity.
- name: decode
phase: Decode
members:
- role: Standalone
# nodeSelector omitted for brevity.
template:
metadata:
labels:
llm-d.ai/engine-type: sglang
spec:
containers:
- name: sidecar
image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0
args:
- --secure-proxy=false
- --kv-connector=sglang
- --vllm-port=8001 # The decode engine's port, whatever the engine.
ports:
- containerPort: 8000
# readinessProbe omitted for brevity.
- name: engine
image: lmsysorg/sglang:v0.5.12.post1-cu129
command: ["python3", "-m", "sglang.launch_server"]
args:
- --model-path=poolside/Laguna-S-2.1-FP8
- --served-model-name=$(MODELPLANE_SERVED_MODEL_NAME)
- --tp-size=8
- --port=8001
- --disaggregation-mode=decode
- --disaggregation-transfer-backend=nixl
# Omitted for brevity.
ports:
- name: http
containerPort: 8001Serving a new engine or KV transfer method no longer waits on a Modelplane
release, only on a router that supports it. A Unified deployment doesn't need
any of this. It keeps Modelplane's default picker unless it sets a router of its
own.
Reserved capacity
The largest GPU machines can be hard to get without a reservation. A node pool
on GKE can now use a Compute Engine reservation you've already bought. A node
pool on EKS can use an On-Demand Capacity Reservation, as well as a Capacity
Block. Register the reservation as a CapacityReservation, and reference it
from the pool:
apiVersion: modelplane.ai/v1alpha1
kind: CapacityReservation
metadata:
name: l4-reservation
spec:
provider: GKE
gke:
zone: us-central1-a
name: l4-reservation
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
name: gke-reserved
spec:
cluster:
source: GKE
gke:
region: us-central1
nodePools:
- name: l4-pool
className: gke-l4-1x-g2
nodeCount: 2
maxNodeCount: 2
zones: [us-central1-a]
reservationRef:
name: l4-reservationThe pool gets nodes only from its reservation, and never falls back to on-demand capacity.
Degraded replicas keep serving
An engine can run several copies within a replica. A replica used to take
traffic only while every copy of every engine was ready, so one crashed copy
withdrew the whole replica. A deployment with one replica went unreachable
until the crashed copy came back.
Now a replica serves while at least one copy of each engine is ready, and the
endpoint picker sends requests only to the copies that are. Ready still means
full capacity. A degraded deployment reads as serving but not ready, and
kubectl get shows the difference in a new SERVING column.
GPU metrics on Nebius
Modelplane maps NVIDIA's DCGM metrics onto modelplane_gpu_* names, but
nothing exported DCGM metrics on Nebius. Modelplane now installs NVIDIA's DCGM
exporter on Nebius clusters, and their GPUs report those series too.
Try it
The full release notes are on GitHub. The getting-started guide covers standing up a fleet. If you're a platform engineer new to inference, Annie Talvasto's field guide to inference engineering covers the concepts, production operations, and tooling you'll meet. Modelplane is Apache 2.0 and moving fast at github.com/modelplaneai/modelplane, and questions are welcome in Slack.




