KServe from Zero to Production: Running and Scaling ML Models in KubernetesLongread · digest for October 22
Cinematic still life of an engineer's desk with a server GPU accelerator card and a monitor displaying KServe InferenceService status.
Longread4 sections≈ 6 min

KServe from Zero to Production: Running and Scaling ML Models in Kubernetes

When a Data Science team gets their first LLM or embedding model ready for production, the temptation is almost always to take the quick route: grab a Dockerfile, toss in Python, FastAPI, and the Transformers library, wire up a /predict route, and roll out a plain Kubernetes Deployment.

For the first week, everything looks great. Then the harsh reality of production infrastructure kicks in:

  • Traffic spikes: Kubernetes' built-in HorizontalPodAutoscaler (HPA) looks at node CPU or memory. Meanwhile, a GPU can be 100% full just from KV-cache allocation even when there are zero active requests! HPA starts spawning pods uncontrollably, gobbling up every free GPU in the cluster.
  • Traffic dies down at night: there are no requests, yet the pods sit there idling, holding onto NVIDIA A100 or H100 instances at several dollars an hour per accelerator.
  • You need to roll out new model weights: how do you shift 10% of traffic to a canary without blowing up memory or duplicating the entire setup?

To solve these problems at an industry-standard level, the Cloud Native Computing Foundation (CNCF) created KServe (formerly KFServing). Let's unpack KServe's architecture, how it pairs with the vLLM runtime, and how to configure an InferenceService manifest step by step.

KServe Architecture: What's Under the Hood?

KServe isn't just a single binary. It's a Kubernetes operator that cleanly stitches together three foundational layers:

  1. Networking layer (Istio Service Mesh): handles intelligent ingress routing, mTLS encryption, traffic splitting for canary rollouts, and granular latency metrics.
  2. Serverless layer (Knative Serving): manages replica lifecycles, intercepts request queues via the queue-proxy sidecar, and scales down to zero (Scale-to-Zero) when traffic disappears.
  3. Inference runtime layer: KServe ships with battle-tested, optimized runtimes—vLLM, NVIDIA Triton Inference Server, HuggingFace TGI, TorchServe, and ONNX Runtime.

Instead of hand-assembling a Deployment, Service, Ingress, HPA, and network policies brick by brick, platform engineers declare a single Custom Resource: InferenceService.

The platform splits the serving pipeline into three logical stages:

  • Transformer: an optional service for pre-processing (like text tokenization or image resizing) and post-processing results.
  • Predictor: the main compute engine where the model weights and GPUs live.
  • Explainer: a dedicated interpretability node (LIME, SHAP) that explains why the model made a given decision.

Separating these stages lets you offload lightweight CPU work (tokenization) onto cheap compute nodes while keeping expensive GPU memory strictly for tensor math. We also looked at infrastructure design for neural networks when exploring applying Kubernetes design patterns to LLM workloads.

The Role of vLLM: Continuous Batching and PagedAttention

In modern LLM pipelines, video memory (VRAM) management is almost always the bottleneck. Traditional libraries statically allocate memory for the maximum context length, leading to brutal fragmentation where 60–80% of VRAM sits wasted.

Architectural diagram of vLLM's PagedAttention and continuous batching mechanism: dynamic allocation of KV-cache memory pages without VRAM fragmentation.

The vLLM engine introduced PagedAttention: memory for the attention Key-Value cache is allocated dynamically in fixed-size pages, much like virtual memory pages in an operating system. Combined with Continuous Batching, vLLM packs 2–4x more concurrent requests onto a single GPU while keeping Time to First Token (TTFT) low.

KServe v0.14+ supports vLLM as a native ServingRuntime.

Step-by-Step Guide: Deploying an InferenceService

Let's deploy a production service for Qwen-2.5-7B-Instruct using KServe and vLLM.

Step 1: Cluster Prerequisites

You'll need a Kubernetes 1.28+ cluster with the NVIDIA GPU Operator, Istio, Knative Serving, and the KServe Operator installed. Assume the model weights are already stored in a corporate S3 bucket at s3://ml-models/qwen-2.5-7b-instruct/.

Step 2: The InferenceService Manifest

Create qwen-inference.yaml:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen-llm-service
  namespace: ml-inference
  annotations:
    serving.kserve.io/autoscalerClass: "kpa"
    autoscaling.knative.dev/target: "10"
    autoscaling.knative.dev/metric: "concurrency"
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 5
    model:
      modelFormat:
        name: vLLM
      storageUri: "s3://ml-models/qwen-2.5-7b-instruct/"
      resources:
        limits:
          cpu: "8"
          memory: "32Gi"
          nvidia.com/gpu: "1"
        requests:
          cpu: "4"
          memory: "16Gi"
          nvidia.com/gpu: "1"
      args:
        - "--model=/mnt/models"
        - "--tensor-parallel-size=1"
        - "--max-model-len=8192"
        - "--gpu-memory-utilization=0.9"

Let's break down the key settings:

  • storageUri: KServe automatically spins up a Storage Initializer init container that pulls the model weights from S3 into the pod's local volume before vLLM starts.
  • autoscaling.knative.dev/metric: "concurrency": scaling is driven not by CPU, but by the number of simultaneous in-flight requests per replica (targeting 10 concurrent sessions).
  • minReplicas: 1: for large models, keeping at least one warm replica is recommended so users don't eat a cold start.

Apply the manifest:

kubectl apply -f qwen-inference.yaml
Step 3: Verifying Status and Testing Inference

Check the deployment status via the CLI:

# Check pod readiness and the service URL
kubectl get inferenceservice qwen-llm-service -n ml-inference

In the output, READY should flip to True, and the assigned endpoint will appear in the URL column.

Now let's test the model through its OpenAI-compatible endpoint:

curl -s -X POST "http://istio-ingressgateway.istio-system.svc.cluster.local/v1/chat/completions" \
  -H "Host: qwen-llm-service.ml-inference.example.com" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-llm-service",
    "messages": [
      {"role": "system", "content": "You are an experienced research engineer."},
      {"role": "user", "content": "What is the main advantage of PagedAttention?"}
    ],
    "max_tokens": 128,
    "temperature": 0.7
  }'

Cold Starts and Scale-to-Zero Trade-offs

Scaling to zero (minReplicas: 0) sounds amazing on paper: no requests at night, pods shut down, and the company stops paying for GPU rentals.

In practice, a cold start for a 15–30 GB model has three phases:

  1. Pod scheduling and startup (5–15 seconds).
  2. Downloading weights from S3 over the internal cluster network (20–60 seconds).
  3. Loading tensors into GPU VRAM and compiling the CUDA graph (15–30 seconds).

That means the first user to show up could be waiting over a minute for a response! The trade-off is straightforward: for internal background batch jobs, go ahead and set minReplicas: 0. For interactive bots and user-facing services, keep minReplicas: 1 and let autoscaling handle the traffic spikes.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 3