How the Claude Code Team Verifies 90% AI-Written Code: `/verify`, Three Test Layers, and Locking Agents Out of Their Own ChecksArticle · digest for November 22
Engineering still life of verification tools: a matte terminal running /verify, a closed padlock on REVIEW.md, and three test control traffic lights.
Article4 sections≈ 11 min

How the Claude Code Team Verifies 90% AI-Written Code: `/verify`, Three Test Layers, and Locking Agents Out of Their Own Checks

At first glance, getting reliable AI code looks simple: write a long system prompt, break the task into steps, and read the diff yourself. That works for a couple of small features. On a live production repo, manual review quickly turns into a bottleneck because the agent writes code faster than you can open pull requests.

Anthropic hit this wall firsthand: Claude now writes and reviews about 90% of Claude Code's own codebase. Without automated verification, an engineer can barely juggle one or two parallel sessions before running out of attention. With a solid verification loop, that same engineer can run dozens or even hundreds of agents at once.

Why Prompts Hit a Ceiling and Why Flaky Tests Break Agents

In Anthropic's internal benchmarks on structured file generation, adding a separate rubric-based grader boosted task success by 10 percentage points. The biggest jump came on the hardest cases where prompt tweaks had stopped helping.

Test code has another major advantage: once an agent writes a test or benchmark, it runs as plain deterministic software. You can execute that script a thousand times and get an answer while spending almost zero tokens.

Still, two traps await. First are flaky tests that randomly pass or fail on identical code due to timeouts or race conditions. They annoy human engineers, but they actively break agents: after a few random failures, a model learns to just rerun the suite instead of fixing the underlying bug.

The Claude Code repo enforces strict rules here: agents are forbidden from skipping flaky tests, adding sleep calls, or retrying until green. If you can't find the root cause, you delete the test with a written explanation. The second trap is slow suites that stretch the feedback loop and bloat batch sizes. Heavy suites move to pre-commit checks, leaving only fast verification inside the active coding loop.

Three Test Layers That Give Agents Eyes and Hit Blind Spots

Mocked unit tests create false confidence: everything turns green, then the app crashes on the first real click. That's why the team prioritizes End-to-End (E2E) checks that walk through a real user flow from start to finish.

To help the agent see real failures, they give it "hands and eyes" in the runtime environment. Web apps get Chrome DevTools Protocol and the Claude for Chrome extension, mobile apps get iOS and Android MCP servers, desktop apps get screen control, and backends run against a local API harness.

Comparison of a unit test checkout.test.ts failing with a blind timeout vs. an E2E test checkout.spec.ts with an error screenshot and step trace (Source: claude.dev blog)

The difference is obvious at a glance. Unit test checkout.test.ts fails with a blank Error: Timeout after 30s—no step, no trace, forcing the agent to guess blindly. E2E test checkout.spec.ts logs every step: signup and cart passed, then checkout stalled. Right next to it sits a screenshot showing the payment form failed to load and the submit button stayed disabled. With that evidence, the model fixes the bug immediately.

The second layer is property-based testing. When an AI writes its own example tests, it misses the same edge cases it missed in the code. Here, you define a general rule (an invariant), and a library feeds the function thousands of random inputs: fast-check in JS/TS, Hypothesis in Python, or proptest in Rust. Teams usually check four properties: round-trip encoding/decoding, invariants, idempotency (running an operation like a migration or payment retry twice yields the same result), and equivalence between two implementations.

import fc from "fast-check";
import { encodeSession, decodeSession } from "./session";

// Verify encoding never loses data on arbitrary inputs
test("session round-trip preserves payload", () => {
  fc.assert(
    fc.property(
      fc.record({
        id: fc.uuid(),
        messages: fc.array(fc.string()),
      }),
      (session) => {
        const restored = decodeSession(encodeSession(session));
        expect(restored).toEqual(session);
      }
    )
  );
});

Fifteen lines of code hammer the parser with empty arrays, broken Unicode, and huge strings the agent would never think to write by hand. When the library finds a failing input, it automatically shrinks it to the minimal case.

The third layer is differential testing. During a migration or major rewrite, the old system stays running as a reference oracle: identical requests hit both versions, and a script flags any mismatch in outputs.

The Workflow: /verify, Three Attempts, and a 39-Incident Checklist

In practice, all of this lives inside a repeatable repo ritual. Modern Claude Code includes built-in /run, /verify, and /run-skill-generator commands, while project rules go into .claude/skills/verify/SKILL.md (see the skills documentation). Unlike CLAUDE.md, which loads on every prompt, skills follow the Agent Skills standard and load only on demand without cluttering context.

---
name: verify
description: Build, launch, and smoke-test the running app E2E.
---

1. Start local service (`npm run dev`), wait for 200 on `/health`.
2. Run E2E flow against live app via CDP, Playwright, or API harness.
3. Save screenshots and step traces on failure; skip unit-only fallback.
4. After 3 failed fix rounds, stop and escalate to a human with traces.

The three-round fix limit works as a circuit breaker. If checks aren't green after three tries, the agent must stop and escalate to a human with fresh traces. Otherwise, a model can spin in circles for hours burning tokens.

Next, the diff runs through a 39-item reliability checklist. Each item has a trigger, a check, and a link to a past incident. For instance, a rule covering old --resume session loading stays quiet when you edit buttons, but the moment a diff touches session storage or message types, Claude checks backward compatibility with older records. On a typical PR, only 3 to 10 of the 39 items fire, keeping noise low.

One agent handles the review across cumulative rounds, remembering earlier notes. If a bug still slips into production, the team doesn't add a rule impulsively: they wait until it repeats in 2+ PRs, have three adversarial agents try to disprove it, replay the incident across three bug-finders, and then get human sign-off. Bloated .md files are pruned periodically.

Banning Self-Edits: REVIEW.md, Hooks, /goal, and the Commit Gate

The whole system collapses if an agent can tweak the exam rules to fit its code. That's why the team enforces a strict ban: a PR is never allowed to edit its own verification rules. Managed Claude Code Review runs in four stages (Find → Verify → Rank → Post), tags findings as Important, Nit, or Pre-existing, and always reads REVIEW.md from the base branch—never from the pull request itself. Every review is pinned to a specific commit.

So checks don't depend on the model's memory, deterministic hooks sit on top of skills in .claude/settings.json (run /hooks to inspect active rules, or check the hooks guide). They fire reliably on PreToolUse, PostToolUse, Stop, and Notification events.

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          {
            "type": "command",
            "command": "jq -r '.tool_input.file_path' | xargs npx prettier -w"
          }
        ]
      }
    ],
    "Stop": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "npm run lint && npm test"
          }
        ]
      }
    ]
  }
}

After every file edit, the hook formats the code; when a turn ends, it runs the linter and test suite. For longer tasks, /goal pairs the worker with a separate verifier model that loops until the target condition passes honestly.

/goal All E2E checkout tests and linters pass without editing test assertions

The final checkpoint before Git is the commit gate, a guard that checks whether /verify or /code-review ran and stamps every commit with its verification method (including an honest "none"). Click through the simulator below to see how its three modes and the two-block session unlock behave.

Commit gate simulator: verification modes and loop protection

It's best to roll out the gate in observe mode first: gather stats on false alarms, clean up the noise, and only then flip to enforce. Allowing a commit through after two blocks in one session ensures a broken test never stalls an urgent hotfix, while still stamping the commit honestly as "none".

According to the claude.dev breakdown, plan availability is straightforward. Skills in .claude/skills/, built-in /verify and /code-review, settings.json hooks, CLAUDE.md, and /goal are open to all Claude Code users. Managed GitHub Claude Code Review (Quick, Standard, and Deep tiers, plus base-branch REVIEW.md reading) is limited to Team and Enterprise plans, does not run builds or execute code (it inspects only the diff and surrounding files), and is unavailable to organizations using Zero Data Retention (ZDR), HIPAA/BAA, or CMEK.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 5