When a Data Science team gets their first LLM or embedding model ready for production, the temptation is almost always to take the quick route: grab a Dockerfile, toss in Python, FastAPI, and the Transformers library, wire up a /predict route, and roll out a plain Kubernetes Deployment.
For the first week, everything looks great. Then the harsh reality of production infrastructure kicks in:
- Traffic spikes: Kubernetes' built-in
HorizontalPodAutoscaler(HPA) looks at node CPU or memory. Meanwhile, a GPU can be 100% full just from KV-cache allocation even when there are zero active requests! HPA starts spawning pods uncontrollably, gobbling up every free GPU in the cluster. - Traffic dies down at night: there are no requests, yet the pods sit there idling, holding onto NVIDIA A100 or H100 instances at several dollars an hour per accelerator.
- You need to roll out new model weights: how do you shift 10% of traffic to a canary without blowing up memory or duplicating the entire setup?
To solve these problems at an industry-standard level, the Cloud Native Computing Foundation (CNCF) created KServe (formerly KFServing). Let's unpack KServe's architecture, how it pairs with the vLLM runtime, and how to configure an InferenceService manifest step by step.
KServe Architecture: What's Under the Hood?
KServe isn't just a single binary. It's a Kubernetes operator that cleanly stitches together three foundational layers:
- Networking layer (Istio Service Mesh): handles intelligent ingress routing, mTLS encryption, traffic splitting for canary rollouts, and granular latency metrics.
- Serverless layer (Knative Serving): manages replica lifecycles, intercepts request queues via the
queue-proxysidecar, and scales down to zero (Scale-to-Zero) when traffic disappears. - Inference runtime layer: KServe ships with battle-tested, optimized runtimes—vLLM, NVIDIA Triton Inference Server, HuggingFace TGI, TorchServe, and ONNX Runtime.
Instead of hand-assembling a Deployment, Service, Ingress, HPA, and network policies brick by brick, platform engineers declare a single Custom Resource: InferenceService.
The platform splits the serving pipeline into three logical stages:
- Transformer: an optional service for pre-processing (like text tokenization or image resizing) and post-processing results.
- Predictor: the main compute engine where the model weights and GPUs live.
- Explainer: a dedicated interpretability node (LIME, SHAP) that explains why the model made a given decision.
Separating these stages lets you offload lightweight CPU work (tokenization) onto cheap compute nodes while keeping expensive GPU memory strictly for tensor math. We also looked at infrastructure design for neural networks when exploring applying Kubernetes design patterns to LLM workloads.
The Role of vLLM: Continuous Batching and PagedAttention
In modern LLM pipelines, video memory (VRAM) management is almost always the bottleneck. Traditional libraries statically allocate memory for the maximum context length, leading to brutal fragmentation where 60–80% of VRAM sits wasted.
Architectural diagram of vLLM's PagedAttention and continuous batching mechanism: dynamic allocation of KV-cache memory pages without VRAM fragmentation.
The vLLM engine introduced PagedAttention: memory for the attention Key-Value cache is allocated dynamically in fixed-size pages, much like virtual memory pages in an operating system. Combined with Continuous Batching, vLLM packs 2–4x more concurrent requests onto a single GPU while keeping Time to First Token (TTFT) low.
KServe v0.14+ supports vLLM as a native ServingRuntime.
Step-by-Step Guide: Deploying an InferenceService
Let's deploy a production service for Qwen-2.5-7B-Instruct using KServe and vLLM.
Step 1: Cluster Prerequisites
You'll need a Kubernetes 1.28+ cluster with the NVIDIA GPU Operator, Istio, Knative Serving, and the KServe Operator installed. Assume the model weights are already stored in a corporate S3 bucket at s3://ml-models/qwen-2.5-7b-instruct/.
Step 2: The InferenceService Manifest
Create qwen-inference.yaml:
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: qwen-llm-service
namespace: ml-inference
annotations:
serving.kserve.io/autoscalerClass: "kpa"
autoscaling.knative.dev/target: "10"
autoscaling.knative.dev/metric: "concurrency"
spec:
predictor:
minReplicas: 1
maxReplicas: 5
model:
modelFormat:
name: vLLM
storageUri: "s3://ml-models/qwen-2.5-7b-instruct/"
resources:
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: "1"
requests:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
args:
- "--model=/mnt/models"
- "--tensor-parallel-size=1"
- "--max-model-len=8192"
- "--gpu-memory-utilization=0.9"
Let's break down the key settings:
storageUri: KServe automatically spins up aStorage Initializerinit container that pulls the model weights from S3 into the pod's local volume before vLLM starts.autoscaling.knative.dev/metric: "concurrency": scaling is driven not by CPU, but by the number of simultaneous in-flight requests per replica (targeting 10 concurrent sessions).minReplicas: 1: for large models, keeping at least one warm replica is recommended so users don't eat a cold start.
Apply the manifest:
kubectl apply -f qwen-inference.yaml
Step 3: Verifying Status and Testing Inference
Check the deployment status via the CLI:
# Check pod readiness and the service URL
kubectl get inferenceservice qwen-llm-service -n ml-inference
In the output, READY should flip to True, and the assigned endpoint will appear in the URL column.
Now let's test the model through its OpenAI-compatible endpoint:
curl -s -X POST "http://istio-ingressgateway.istio-system.svc.cluster.local/v1/chat/completions" \
-H "Host: qwen-llm-service.ml-inference.example.com" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen-llm-service",
"messages": [
{"role": "system", "content": "You are an experienced research engineer."},
{"role": "user", "content": "What is the main advantage of PagedAttention?"}
],
"max_tokens": 128,
"temperature": 0.7
}'
Cold Starts and Scale-to-Zero Trade-offs
Scaling to zero (minReplicas: 0) sounds amazing on paper: no requests at night, pods shut down, and the company stops paying for GPU rentals.
In practice, a cold start for a 15–30 GB model has three phases:
- Pod scheduling and startup (5–15 seconds).
- Downloading weights from S3 over the internal cluster network (20–60 seconds).
- Loading tensors into GPU VRAM and compiling the CUDA graph (15–30 seconds).
That means the first user to show up could be waiting over a minute for a response! The trade-off is straightforward: for internal background batch jobs, go ahead and set minReplicas: 0. For interactive bots and user-facing services, keep minReplicas: 1 and let autoscaling handle the traffic spikes.


