Modelplane: orchestrating GPU clusters and AI inference on top of CrossplaneArticle · digest for November 3
Architectural isometric diagram of the Modelplane orchestrator uniting heterogeneous GPU compute nodes on top of Crossplane.
Article5 sections≈ 6 min

Modelplane: orchestrating GPU clusters and AI inference on top of Crossplane

If your team has ever had to deploy open language models — say, the Qwen 2.5 family, Llama 3, or fresh DeepSeek checkpoints — inside your own closed perimeter, you've definitely felt the full pain of today's compute market. Getting a monolithic pool of a few dozen NVIDIA H100 accelerators from a single cloud provider today is either impossible due to queues or prohibitively expensive.

In reality, infrastructure quickly turns into a patchwork quilt. Two A100 nodes are rented in AWS, an L40S cluster runs in a specialized neocloud like CoreWeave or Lambda Labs, and another couple of servers with consumer RTX 4090 cards sit on-premises for experiments. At the single-pod level in Kubernetes, the stack has long been clear: grab vLLM or SGLang, configure PagedAttention, hook up autoscaling via KEDA. But as soon as there are more than a dozen pods and clusters are scattered across three continents, the platform team drowns in manual traffic routing and version alignment.

An architectural answer to this chaos was proposed by Nic Cope, one of the key maintainers of the Crossplane project for the past eight years. His new open-source project Modelplane turns a fragmented GPU zoo into a single managed inference platform using battle-tested Kubernetes abstractions.

Control plane architecture: Crossplane v2 instead of homemade operators

Modelplane's main engineering boldness lies in what the author deliberately refused to do. The first impulse of any architect in this situation would be to sit down and write yet another heavy Kubernetes operator in Go from scratch, with a dozen custom controllers, hand-rolled reconciliation loops, and fragile state synchronization logic.

Cope took another path: Modelplane is built entirely on native core abstractions of Crossplane v2 — Composite Resource Definitions (XRDs), Compositions, and Composition Functions. Instead of reinventing the wheel, the project relies on ready-made core mechanisms:

  • reliable reconciliation loops;
  • resource deletion finalizers;
  • automatic runtime configuration drift protection;
  • the native Namespaced Composites model introduced in the second major version of Crossplane, which rid the platform of the cumbersome Claim/XR pairing.

The composition functions themselves are written in Python — the primary language of modern machine learning. Thanks to schema generation and strictly typed classes via the latest Crossplane CLI, model placement logic could be described compactly and without the risk of typos in endless YAML dictionaries.

Declarative management: CEL filters for VRAM

Modelplane cleanly separates two areas of responsibility:

  1. Platform engineers operate cluster-level entities InferenceCluster and InferenceClass. They wire up Crossplane cloud providers, provision virtual machines, and monitor physical node health.
  2. ML developers work exclusively at the project namespace level through the ModelDeployment resource. They don't need to know IP addresses or network topology: they simply declare which model they want to run and how much VRAM it needs.

To express hardware requirements, the Common Expression Language (CEL) is used. Let's look at what a typical model deployment manifest looks like:

apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-8b
  namespace: ml-team
spec:
  replicas: 2
  engines:
  - name: qwen-engine
    members:
    - role: Standalone
    nodeSelector:
      devices:
      - name: gpu
        count: 1
        selectors:
        - cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
    template:
      spec:
        containers:
        - name: engine
          image: vllm/vllm-openai:v0.23.0
          args:
          - "--model=Qwen/Qwen3-8B"
          - "--served-model-name=qwen"

Pay attention to the CEL selector: instead of hardcoding a specific card model like nvidia.com/gpu-model: A100-SXM4-80GB, the manifest declares a minimum VRAM threshold of 20 gigabytes. The Modelplane scheduler will automatically find a suitable free node — whether that's an L40S, an A100, or a sliced instance card in the cloud.

Infrastructure layer: Composite Resource Definitions

At the platform level, the control plane translates high-level requests from the development team into concrete infrastructure resources. Below is an example of a Composition manifest binding a platform inference class to a physical cluster:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: inference-cluster-aws
  labels:
    provider: aws
    tier: high-memory
spec:
  compositeTypeRef:
    apiVersion: modelplane.ai/v1alpha1
    kind: InferenceCluster
  mode: Pipeline
  pipeline:
  - step: provision-gpu-nodes
    functionRef:
      name: function-python-placement
    input:
      apiVersion: modelplane.ai/v1alpha1
      kind: PlacementConfig
      minVramPerNode: "80Gi"
      spotTolerance: true

The function-python-placement function polls quota availability metrics across connected accounts on the fly, spins up spot instances to save budget, and dynamically configures the edge routing gateway.

Dynamic scheduling and an OpenAI-compatible gateway

To the company's application services, this entire complex hybrid architecture looks like an ordinary remote service speaking the OpenAI API protocol (/v1/chat/completions).

A gateway built on Kubernetes Gateway API accepts an incoming request for the qwen model, checks current queue depths on vLLM backends, and routes the token stream to the least loaded pod. If one site suddenly loses power or a cloud provider reclaims a spot server, the Crossplane controller instantly shifts traffic to a backup cluster without dropping active client sessions.

Paired with engineering practices for building fault-tolerant agent systems, which we covered in detail in our guide to AI agent engineering, this approach avoids rewriting backend code when switching compute providers.

Dynamic queue scaling

When configuring queues in Modelplane, use KEDA scaling on the vllm_num_requests_waiting metric. Scaling on classic CPU or GPU memory utilization is useless: inference engines allocate 90% of VRAM for the KV cache right at startup, so the real marker of overload is specifically the queue size of pending prompts.

Engineering trade-offs and network overhead

It would be a mistake to view Modelplane as a magic silver bullet that solves distributed systems problems for free. You still pay for a universal abstraction with engineering trade-offs.

The main challenge is network connectivity. For inference pods across different clouds to securely exchange traffic and pull heavy model weights from a shared registry, you need an overlay network stretched across sites (WireGuard, Cilium ClusterMesh, or Cloudflare Magic WAN). Transferring the weights of a 70-billion-parameter model totaling 140 gigabytes over the public internet takes noticeable time, so caching artifacts on local NVMe drives of the nodes remains mandatory.

Still, for mature platform teams, Nic Cope's approach sets a new standard: instead of bloating a zoo of one-off bash scripts, you get a clean declarative architecture that scales by native Kubernetes rules.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 3