Every time a conference demo pitches a new "autonomous AI agent," we get the same sci-fi promise: production breaks at 3 a.m., on-call engineers sleep peacefully, and the neural net reads logs, patches configs, and closes the incident all by itself. Any engineer who has carried a pager at hundreds of thousands of RPS just smiles wryly—real-world infrastructure has way too many hidden dependencies and red herrings to hand the keys to a language model.
Or are we closer to that line than it seems?
At Signals Berlin, Alex Palcuie—a former Google Staff SRE with eight years of experience who now serves on Anthropic's reliability team—gave a talk with a blunt title: "Can Claude fix itself? From no to maybe." Back in the spring at QCon London, his verdict was a flat "no." After six months of real on-call shifts using the terminal agent Claude Code, his stance shifted to a cautious "maybe."
After digging into Alex Palcuie's talk, I broke down three real incidents at Anthropic to see where modern AI agents already shine—and where they remain downright dangerous in production.
Three Production Incidents: From a "Move 37" Moment to a Red Herring
The most valuable part of Anthropic's experience is the candid log of how the agent behaves when telemetry suddenly goes sideways.
The first incident hit on New Year's Eve. A flagship Claude model started throwing a spike of HTTP 500 errors. The on-call engineer launched Claude Code to localize the failure. Scanning the traces, the model spotted a bizarre pattern: crashes happened in the image-preprocessing module exclusively on requests with exactly 22 attached images.
A typical on-call responder would have filed a ticket for the backend team and restarted the services. The agent went further: it correlated client metadata and discovered the requests were coming from 200 accounts with similar email patterns, all created three days earlier as part of a batch of 4,000 accounts. It was a coordinated DoS attack. The agent recommended paging the security team immediately, and they blocked the attackers. Inside Anthropic, they called this a "Move 37" moment—a nod to AlphaGo's unexpected 37th move in Go.
The second incident exposed a broken monitoring pipeline. An inference router suddenly drained a healthy cluster and shifted traffic onto neighboring nodes, overloading them. In the incident channel, engineers blamed the data-center network. Claude, however, looked closely at the numbers: several servers were reporting token-generation speeds that were physically impossible for the installed accelerators. The hardware wasn't broken—the meter was. It turned out the metrics collector had undergone resharding, and two shards polled the exact same servers in parallel for five minutes, doubling the reported load.
The third incident, though, was a cold shower for enthusiasts. Error rates on a cluster jumped to 41%. Claude inspected the dashboards and noticed that local cache hit rates had plummeted to near zero right before the spike. It spun up a neat, convincing theory: the cache flushed, requests fell back to slow disk, and nodes timed out. Logical on paper—and completely wrong, a classic red herring. In reality, cache misses were happily served by the shared distributed cache, and the true culprit was plain CPU saturation. The agent fell into a classic trap: if you hunt for anomalies across hundreds of graphs without a hypothesis, you will always find a correlation.
The Agent in the OODA Loop: Where AI Excels and Where It Fails
To systematically map AI boundaries in operations, Anthropic overlaid the model's behavior onto the military OODA loop: Observe — Orient — Decide — Act.
In the Observe phase, the agent is phenomenally fast. A human struggles to juggle dozens of Grafana tabs while grepping terabytes of logs. The agent tears through them in seconds and surfaces non-obvious patterns.
The Orient phase is where the main risk lives. Language models are prone to confabulation: they invent causal links where there is only coincidence. Blindly trusting an agent's root-cause theory can derail an incident response for hours.
In Decide, the model is genuinely helpful—suggesting candidate actions cross-referenced against runbooks and postmortems.
But at Act, there is an absolute red line. No SRE org should let a model run mutating commands against a live cluster unsupervised.
The Action Classifier and the Telemetry Fragmentation Ceiling
Anthropic's core takeaway is that agent autonomy must be strictly asymmetric, enforced by a mandatory Action Classifier.
Any safe read-only operation—running GET requests, tailing logs, measuring latency, capturing memory dumps—can execute autonomously. The moment the model tries to run kubectl delete, bounce a load balancer, or push a hotfix, execution pauses for explicit human approval. That lines up closely with practical lessons from running Claude Code, where tight permission boundaries make or break reliability.
The second barrier is telemetry cost and fragmentation. As Sylvain Kalache from Rootly points out, streaming all logs to cloud storage drives massive bandwidth bills, and incident context is scattered across Prometheus for metrics, Loki for logs, and Jaeger for traces. Without deep integration via protocols like MCP, the agent is flying half-blind.
Claude Code can already serve as a sharp on-call copilot today. Just make sure a human engineer keeps both hands on the production wheel.

