If you follow machine learning and large language models (LLMs) closely, you've probably grown skeptical of standard leaderboards and synthetic benchmarks like MMLU, GSM8K, or HumanEval. Every time a new model drops, marketing teams flood the feed with victory charts showing a fraction-of-a-percent edge over yesterday's leader. Then you plug that hyped model into a real production backend—parsing nested JSON or running autonomous agents across a sprawling codebase—and the magic turns into a pumpkin: it hallucinates, breaks tool-call schemas, and loops endlessly.
The root problem was that objective data on how models actually perform in the wild stayed locked inside OpenAI, Google, Microsoft, and Anthropic data centers. All we ever saw was polished corporate marketing.
Now that wall has a massive crack in it. OpenRouter, the independent model gateway handling billions of daily requests from hundreds of thousands of engineers worldwide, teamed up with Andreessen Horowitz (a16z) to publish a landmark empirical study: "State of AI: An Empirical 100 Trillion Token Study with OpenRouter" (preprint on arXiv).
Behind this report sits an unprecedented anonymized telemetry dataset—more than 100 trillion real-world inference tokens. Let's peel back the glossy wrapper on the AI industry and look at the hard engineering numbers: how the balance between open and closed weights has shifted, why the hyperscaler moat myth collapsed, and how reasoning models turned compute architecture upside down.
Beyond Synthetic Benchmarks: An Objective View of Real Traffic
What makes the a16z and OpenRouter study so valuable is the cleanliness and real-world nature of its sample. Synthetic benchmarks have long suffered from data contamination—models get tuned to ace academic puzzles like students cramming leaked exam sheets. Production is nothing like a classroom: you don't get neat prompts, you get messy user logs, partial context, and tight latency budgets.
OpenRouter operates as a neutral inference marketplace: it doesn't train its own weights, instead offering one unified API across hundreds of models—from proprietary giants (OpenAI GPT-4o, Anthropic Claude 3.5 Sonnet, Google Gemini) to open-weight architectures (DeepSeek, Meta Llama, Qwen, Mistral, Moonshot, MiniMax, Yi, GLM). Its traffic doesn't come from casual users asking for apple pie recipes; it comes from developers, startups, and product teams wiring LLMs into live backends, CLI agents, IDEs, and automated pipelines.
The researchers tracked request patterns over 18 months. At the start of the window, weekly traffic sat at a modest 1.8 trillion tokens; by autumn 2026, it had surged to an astonishing 36.8 trillion tokens per week! Every transaction was analyzed across dozens of telemetry dimensions: prompt-to-completion ratios, Time-To-First-Token, tool-call frequency, validation errors, and serving cost.
In our earlier piece on developer infrastructure transformation and OpenRouter LLM gateway consolidation, we flagged the rising importance of smart routing. Now we have hard empirical proof of that tectonic shift.
The Great Geographic Shift: Rise of China's Open Models
The study's loudest finding shatters the assumption that Silicon Valley holds an unassailable lead in AI. Across 100 trillion tokens, the data captures a dramatic geographic flip.
In spring 2025, closed US APIs accounted for more than two-thirds of all generation volume on the platform. Open models looked marginal, and Chinese releases were often dismissed as fast followers. Twelve months later, inference volume on open Chinese models—led by the DeepSeek family and Alibaba's Qwen lineup—didn't just catch up to US services; it nearly doubled their traffic! By autumn 2026, Chinese models were steadily generating over 19.5 trillion tokens per week compared to roughly 14.0 trillion for US giants.
The authors also put to rest the "DeepSeek decline" narrative pushed by consulting firms in late 2025. The arrival of DeepSeek-V3 and the reasoning model DeepSeek-R1 triggered a stampede of developer adoption. When code generation and math accuracy match top commercial models at $0.14–$0.28 per million tokens versus $2.50–$15.00 for closed competitors, engineering pragmatism beats brand loyalty every time.
In our breakdown of how Qwen 3.8 Max drove a shift toward open language models, we dug into its attention mechanisms and Mixture-of-Experts (MoE) design. The a16z report confirms that businesses are voting with their wallets for openness and cost efficiency.
From One-Shot Answers to Hidden Thinking: The Test-Time Compute Revolution
The second fundamental shift documented in the study is happening inside the physics of inference itself. The industry has decisively entered the era of reasoning models and Test-Time Compute.
Architectural cutaway and Test-Time Compute tree diagram for reasoning models with hidden thinking nodes.
In classic autoregressive models (GPT-4, Llama 3), inference followed a predictable ratio: you sent context in, and the model streamed an answer out in a single pass. The prompt-to-completion ratio hovered steadily between 3:1 and 5:1—roughly 20–30 output tokens for every 100 input tokens.
Reasoning models turned that architecture inside out:
- The model no longer tries to guess the final answer on the very first token;
- It enters a hidden internal reasoning phase (thinking tokens / chain-of-thought), branching through intermediate hypotheses, checking proof steps, and pruning dead ends;
- Hidden thinking tokens now outnumber the visible final answer by 5x to 15x!
Inference is no longer simple text interpolation—it's algorithmic search over a decision tree (tree search / Monte Carlo Tree Search). For infrastructure engineers, that means GPU load per user request shoots up dramatically. Where memory bandwidth used to be the main bottleneck, generating hundreds of hidden reasoning steps slams tensor cores right back to full utilization.
Agentic Loops and Autonomy: What Actually Burns the Compute
Who is consuming all these trillions of tokens? Request metadata debunks another popular myth—that people mostly use LLMs to draft emails or summarize school essays.
Production usage clusters around three dominant workloads:
- Code generation, review, and refactoring (Coding Assistance). This is the undisputed locomotive of token consumption. Modern IDEs and agentic terminals (Cursor, Windsurf, Claude Code, Aider) feed massive context windows (32k to 128k tokens) packed with full source files, ASTs, unit test logs, and compiler output.
- Agentic feedback loops (Agentic Loops & Tool Calling). Developers rarely make one-off LLM calls anymore. The standard pattern is an autonomous loop: the model emits API arguments, the runtime executes a command in a sandbox, feeds the output back to the model, and lets it pick the next step. A single user task can trigger 10 to 50 iterations, burning tens of thousands of tokens in seconds.
- Roleplay and interactive narrative. Surprisingly, the runner-up in open-model traffic isn't enterprise support bots—it's interactive entertainment platforms. Open models free from heavy-handed corporate filters have become the default engine for virtual companions and game characters.
The Cinderella Effect and Jevons Paradox: Production Inertia vs. Cheap Tokens
One of the most fascinating sections of the report examines developer cohort retention. The researchers spotted a pattern they dubbed the "Cinderella Effect."
Here's how it plays out: when a breakthrough model launches, a cohort of developers builds and tunes their prompts, JSON schemas, and system instructions specifically around its quirks. Months pass, competitors ship faster, cheaper, and smarter models, yet that cohort keeps sending millions of requests to "their" old model.
Why? Prompt brittleness. Swapping out the underlying model in a live backend is an engineer's nightmare: response formatting shifts, edge cases drift, and tool-call parsers break. Teams put off migrating until a new model arrives that beats the current one not by a few percent, but by orders of magnitude. DeepSeek-V3 was one of those rare step-changes that broke the inertia across hundreds of thousands of projects in weeks.
Dropping inference prices by an order of magnitude doesn't shrink total AI spend. Instead, cheaper compute triggered a 100x explosion in consumption: teams stopped rationing context, adopted multi-agent swarms, and wired up continuous background audits.
Economists call this Jevons Paradox: making a resource more efficient to use increases total demand for it rather than saving it. When a token costs fractions of a cent, nobody bothers trimming system prompts—engineers happily feed the model entire git histories and full API docs.
Next-Gen Backend Architecture: Routing, Prefix Caching, and Timeouts
Drawing on this empirical telemetry, the report's authors lay out concrete architectural guidelines for engineers building reliable LLM-backed systems.
If you're designing a modern AI backend, three infrastructure patterns need an immediate rethink:
- Dynamic multi-provider routing. Tying your backend to a single host's direct API is a recipe for downtime. Open-weight inference runs across distributed data centers, so a resilient gateway should automatically balance traffic across independent providers based on latency and GPU availability.
- Context prefix caching (Prompt Caching). Since agent loops repeatedly send the same system prompt and codebase context, hosting stacks with KV-cache reuse (like vLLM with PagedAttention or SGLang) cut input token costs and latency by 50–80%.
- Rethinking HTTP timeouts. Classic 5–10 second microservice timeouts will kill connections to reasoning models. The hidden thinking phase can run 30 to 60 seconds before the first byte arrives (Time-To-First-Token). Your architecture needs async queues, Webhooks, Server-Sent Events, or gRPC streaming instead of blocking synchronous calls.
Structure your prompts so all static content (system instructions, tool definitions, fixed DB schemas) sits at the very beginning, and dynamic user data goes at the end. That lets inference engines reuse the compiled KV cache across concurrent sessions.
The Paradigm Shift: Why Open Weights Are Now Core Engineering Infrastructure
OpenRouter's 100-trillion-token study closes the first chapter of generative AI. The era of wide-eyed hype over closed monopolies has given way to pragmatic engineering.
The corporate myth of an "unbreachable moat" around proprietary labs is gone. Open weights, self-hosting inside your own secure perimeter, and fierce price competition among independent inference hosts have turned high-quality LLM compute into a commodity.
For developers and architects, that means real freedom: we're no longer hostage to one vendor's pricing whims, license changes, or sudden rate limits. The future of software engineering belongs to open standards, transparent architecture, and owning your compute stack.

