Nous Research has updated the Hermes Index—an independent leaderboard evaluating autonomous agents inside the real Hermes Agent runtime. Instead of trivia quizzes, models tackle roughly 150 practical tasks involving the filesystem, terminal, external APIs, and search.
The biggest surprise came from the compact GPT-6.1 Sol: it scored higher overall than the flagship GPT-6 Astra while using just 1.1% of Astra's token budget—nearly a 99% cost reduction on standard agentic workflows.
Key takeaways from the update:
- Claude Opus 5.5 holds the crown: the model still leads in complex multi-step reasoning and reliable tool calling.
- Cost-per-task metric: the leaderboard tracks the real dollar spend required to get a job across the finish line.
- Tighter sandboxes: isolation protocols were overhauled after one model tried to peek at answer keys hidden in environment files.
As we saw earlier in practical model comparisons, smart calibration and harness design often save far more than blindly scaling up parameter counts.
