If you've followed the evolution of large language models closely, you remember what a revolution it felt like jumping from a modest 8k tokens to 128k and 200k windows in GPT-4o and Claude 3.5 Sonnet. It seemed the practical ceiling had been reached: Google Gemini 1.5 Pro pushed the bar to 2 million tokens, and beyond that milestone the industry hit a long pause. The reason is straightforward — quadratic growth in attention compute and catastrophic memory degradation when scaling classic transformers.
That silence didn't last long. Research startup Mirendil, founded by veterans of leading global labs — Anthropic, Google DeepMind, OpenAI, and xAI — came out swinging with its first flagship model, Fern 0.1. Its headline feature sounds almost like sci-fi: a 10-million-token context window.
To put that scale in physical terms: 10 million tokens equals roughly 23 megabytes of uncompressed plain text. That fits an entire 500,000 lines of source code (a full corporate monorepo with all its microservices) or continuous logs from months of autonomous agent runs. Let's look at how engineers broke through this barrier, what rigorous benchmarks show, and why a giant context window doesn't repeal the laws of math.
Mirendil-Base Architecture: 10 Million Tokens Without RoPE Hacks
For years, the industry's standard trick for stretching context has been positional embedding interpolation — methods like RoPE scaling (YaRN, LongRoPE). You train a model on short sequences (say, 8k or 32k) and then artificially "stretch" the rotation angles of position vectors. There's no free lunch: once you push several times past the base length, attention blurs, detail recall drops, and the model starts scrambling chronological order.
Mirendil took a fundamentally different path. Fern 0.1 is a post-trained checkpoint of the base Mirendil-Base architecture (ranging from 100 to 500 billion parameters). The key engineering call was ditching after-the-fact hacks: the architecture was designed and trained from day one on ultra-long sequences. Attention was calibrated so hidden-state variance wouldn't collapse across millions of positions.
Just as notable is how the model was built. Mirendil representatives stated that pre-training took just 3.5 months with a total compute budget under $10 million. The entire project was led by a single principal researcher coordinating a network of specialized AI agents: over 99% of the training code and synthetic data filtering pipelines were written by autonomous systems.
The RULER Benchmark: Perfect Fact Retrieval Meets Reasoning Drop-Off
Claiming 10 million tokens is easy; proving the model actually remembers what's in the middle of the text — instead of suffering from classic "Lost in the Middle" syndrome — is much harder. To validate the architecture, Fern 0.1 was run through the standardized RULER suite (a synthetic ultra-long-context benchmark from NVIDIA researchers).
The results split into two stark pictures:
- Needle In A Haystack: In Single-Needle and All-Needles Retrieval tests, Fern 0.1 hit 100% accuracy across the full depth up to 10 million tokens. The model behaves like a flawless semantic search engine: a config variable or passport number buried among millions of lines is pulled out without a miss regardless of position.
- Multi-Hop Question Answering: Once the task got harder, the illusion of omnipotence faded. When answering a question requires not just copying a fact, but finding three indirect clues across different parts of a 20-megabyte document, cross-referencing them, and drawing a logical conclusion, Fern 0.1's accuracy drops to 45–51%. The model sees individual trees with crystal clarity, yet loses the connections across the vast forest.
On coding and bug-fixing benchmarks, the model also posted respectable numbers: 73.4% on SWE-Bench Verified and 44.2% on the tougher DeepSWE suite. That puts it solidly in frontier territory, though on unconventional architectural leaps it still trails specialized reasoning models. We previously examined how the open-model landscape is shifting under massive compute loads in our 100 trillion tokens of inference study.
The RAG-less Paradigm: Where 10 Million Tokens Change the Rules
Being able to load 10 million tokens in a single prompt calls the traditional enterprise RAG (Retrieval-Augmented Generation) stack into question. A classic RAG pipeline has always been fragile:
- Documents have to be sliced into fixed-size chunks;
- Text chunks are run through an embedding model;
- Vectors are stored in a database (Pinecone, Qdrant, Milvus);
- At query time, k-nearest-neighbor search pulls the top-5 fragments, often missing crucial context on the adjacent page.
With a 10M-token window, vector chunking simply becomes unnecessary for most local workloads. Welcome to the RAG-less era:
- End-to-end monorepo audits. You feed the model the entire project archive at once. An agent can trace how changing the signature of one core function ripples across thirty microservices. We saw a similar knack for long autonomous sessions when analyzing supermodel Qwen 3.8 Max.
- Analyzing hundreds of financial filings. Bank analysts can load quarterly filings, investor call transcripts, and audits spanning 10 years, asking for end-to-end debt dynamics without risking dropped footnotes.
- Continuous agent session memory. AI agents can keep weeks of run histories, terminal dumps, and test outputs in working memory without compacting or forgetting intermediate steps.
Hardware Reality and Inference Economics
That engineering triumph comes with a steep hardware bill. Ultra-long context exposes harsh economics:
- Time To First Token (TTFT). The prefill phase for 10 million tokens requires multiplying colossal matrices. Even on a cluster of modern accelerators, processing such a prompt before generating the first word can take anywhere from tens of seconds to a minute and a half. Real-time interactive chat is out of the question.
- Per-query cost. Pushing 10 million tokens through a hundred-billion-plus parameter model in a commercial cloud can cost tens of dollars for a single call. That rules out consumer chatbots and limits adoption to high-margin enterprise use cases.
- Early access status. At launch, model weights have not been open-sourced. Mirendil offers API access via an Early Access form at
mirendil.com, primarily for research institutes and enterprise partners.
Fern 0.1 proves that 10 million tokens is no longer a theoretical concept — it's a working reality. Still, developers will have to master the crucial distinction: telling blind fact retrieval in a giant context apart from genuine comprehension of what was read.

