If you've ever self-hosted language models on your own servers, you know how brutal the fight for every single token per second can be. The vLLM engine has long been the industry gold standard: PagedAttention, continuous batching, and distributed inference squeeze nearly everything they can out of your GPUs. But once you start chasing frontier-level optimization for a specific weight architecture, engineers inevitably hit a ceiling.
Historically, compute kernel optimization has been an elite craft. There are maybe a couple hundred specialists worldwide who can write razor-sharp C++, CUDA, and OpenAI's low-level Triton language. Tuning a single operator by hand takes weeks of painstaking experiments with GPU registers and shared memory.
Engineers at Google DeepMind decided to test a bold hypothesis: what if they handed the job of optimizing inference for the open Gemma 4 model to their flagship reasoning model, Gemini 4 Argon? The result blew past even optimistic expectations: the model acted like an autonomous systems compiler and submitted eight pull requests to the official vLLM repo, nearly tripling throughput. Let's pop the hood and look at the exact bottlenecks the algorithm uncovered.
Speculative Decoding with MTP: Predicting Tokens in Batches
The core curse of autoregressive models is sequential generation. To output a fifty-word sentence, the GPU has to make fifty sequential passes through the weights. Even when compute units are sitting idle, memory bandwidth latency chokes performance.
The Gemma 4 architecture was designed from the start around Multi-Token Prediction (MTP)—the ability to forecast multiple tokens in a single forward pass. In vLLM, however, that potential sat untapped because there wasn't an efficient scheduler for it.
Gemini 4 Argon engineered a dedicated speculative drafter module. The mechanics are clean: a lightweight drafter generates a chain of 4 candidate tokens in parallel on the fly. Then the main Gemma 4 E2B model verifies the entire batch in one single forward pass. Whenever the guessed tokens match, they go straight into the output. On structured outputs—code, JSON, or logs—the acceptance rate goes through the roof, delivering a massive speed boost without losing semantic accuracy.
Five Custom Triton Kernels: Operator Fusion and Stopping Memory Leaks
Speculative decoding alone wasn't enough, though; deeper bottlenecks were hiding where the model talks to GPU memory. Argon profiled the compute graph and generated five custom kernels in Triton.
Three engineering moves in this refactor stood out to me the most:
- Fusing RMSNorm and QKV linear projection (Kernel Fusion). In the standard pipeline, a tensor gets normalized, written out to global GPU memory, and then read right back by the next operator for matrix multiplication. Argon merged those steps into one fused kernel: data stays inside fast local register memory, cutting out an entire round-trip to VRAM.
- Hardware RoPE operator with cached trigonometry. Rotary Position Embedding constantly requires computing sines and cosines. The generated kernel pairs cached coordinate grids with streaming multiplication.
- Sparse gather in the GEMV head. This was the heaviest win: in stock vLLM, an intermediate buffer of temporary tensors chewed through up to 1,540 MB of VRAM. Argon rewrote the algorithm for direct streaming reads, dropping auxiliary VRAM overhead to nearly zero.
Freeing up that memory meant batch sizes could scale up significantly without crashing into Out of Memory errors.
NVIDIA A100 Benchmarks: 108,000 Tokens per Second Without Quality Loss
The numbers speak for themselves. Testing ran on a standard server-grade NVIDIA A100 with 80 GB of memory:
- Baseline Gemma 4 E2B throughput on vLLM sat at 37,900 tokens per second.
- After applying Gemini 4 Argon's patch set, throughput jumped to 108,000 tokens per second.
- Total speedup factor: 2.85x.
Better yet, logit verification showed a 100% match against the reference distribution. This is pure compute optimization—no weight quantization, no context truncation, and no degradation in answer intelligence.
For hosting providers, nearly tripling speed translates directly into shrinking server fleets. Where you previously needed a cluster of three servers costing tens of thousands of dollars a month, a single rack now handles the load.
What This Changes for Open Models
This case isn't just a pretty benchmark win. We're watching the birth of a new software era: recursive optimization of AI by AI itself.
OpenAI originally built Triton so Python developers could write fast GPU kernels without diving into raw CUDA assembly. As it turns out, Triton is an ideal playground for reasoning models. A neural net easily holds the GPU memory layout in context, respects tile sizes, and outputs code that goes toe-to-toe with hand-tuned work from top silicon architects.
Of course, generated kernels still need strict validation: code tuned for Ampere (A100) can behave differently on Hopper (H100) or Blackwell. Plus, speculative decoding shines brightest on deterministic tasks—crank generation temperature up to 1.5 for poetry and the token acceptance rate drops, making the speedup more modest. Still, the foundation is laid: the era of manual low-level grunt work is giving way to autonomous AI compilers.

