Google AQuA: A Free Open-Source AI Auditor for Catching Silent Hallucinations in Agent DialogsNote · digest for November 9
An ironic surrealist still life with an AQuA auditor magnifying glass revealing a hidden hallucination behind a polite facade.
Note3 sections≈ 5 min

Google AQuA: A Free Open-Source AI Auditor for Catching Silent Hallucinations in Agent Dialogs

Almost everyone gets fooled at first when checking the quality of customer-facing chatbots—developers and store owners alike. You launch an AI consultant on your site and check the gauges: the server is steady, replies fly out instantly, and not a single complaint lands in support. It looks like automation nailed it on the first try. I caught myself feeling that same false calm until I opened the raw chat logs.

That's where an uncomfortable truth surfaces: shoppers almost never argue with a bot—they just close the tab silently. How do you catch these quiet mistakes in AI assistant dialogs when dashboards are glowing green and customers walk away without a word? To bring that blind spot into the open without reading chats by hand, Google engineers released the free auditor AQuA (Ambient Quality Agent) in the Google ADK Recipes repository.

The Illusion of Green Dashboards: Why Customers Leave Quietly Over Polite Hallucinations

When ordinary software breaks, it throws an error and monitoring sounds the alarm. Language models trigger a different chain: when a bot doesn't know the exact answer or gets tangled in rules, it doesn't crash—it politely invents plausible-sounding fiction. In engineering this is called a Silent Failure: technically the conversation went fine, yet the customer got misled.

In an online store, that trap hits revenue directly. Say a bot is helping during a sale and a shopper asks whether a promo code applies to an espresso machine. Instead of checking the exclusions list, the assistant promises a discount on a non-promotional item. The shopper adds the machine to the cart, enters the promo code at checkout, sees the original price, and quietly leaves for a competitor. Dashboards stay green while the store loses an order. We wrote about how strict guardrails stop that kind of improvisation in our breakdown of a deterministic pipeline for agents.

Inside Google AQuA: How an Auditor Agent Reviews Logs of Other Conversations

Since uptime charts can't spot semantic defects, the creators placed AQuA in an Outer Loop. Putting a censor inside live chat would double the customer's wait time for every reply. Instead, the auditor runs asynchronously in the background, analyzing archives of completed chats. We touched on this kind of oversight in our review of the background observer agent Google AQuA, and Julian Goldie SEO's breakdown walks through its five-step loop:

  1. Sampling: on each run, the agent randomly pulls up to 1,000 recent dialogs from the database.
  2. Audit: a judge model scores each chat against a checklist of 9 common failures plus your own plain-language rules.
  3. Clustering: similar slip-ups are grouped together, turning a pile of logs into a handful of clear issues.
  4. Verification: a second independent AI re-reads the chats and filters out false alarms from the first judge.
  5. Prompt attribution: each failure gets tagged (new, recurring, or resolved), and AQuA points to the exact line in the system prompt that caused the hallucination.

In the store example, the auditor clusters promises of nonexistent discounts, double-checks them with the second judge, and highlights the culprit in the instructions—a vague line like "Check promo code terms when possible."

Step-by-Step Quality Loop: From Catching an Error to Fixing the System Prompt

To a human, polite words like "try to" or "when possible" sound like guidance, but to a neural network they signal that the rule is optional. In Google's tests on a concierge bot, only 5 out of 32 dialogs passed without issues: the auditor flagged 9 error clusters (3 were filtered out by the second judge, leaving 6 real problems), and in 15 conversations the agent simply forgot to offer seat selection. Replacing soft phrasing in two prompt lines with strict "always," "never," and "must"—without changing the model—cut missed seat offers to 2, dropped missed dietary requests from 7 to 0, and lifted clean dialogs to 13. In the espresso machine case, swapping "check when possible" for "Never promise a discount without checking the promotions table" immediately closes the loophole for fabrication.

An automated auditor has its trade-offs: out of the box, AQuA only works with agents built on Google Agent Development Kit in Google Cloud, double-checking spends tokens across two models, and random sampling can miss rare bugs. Still, you can try it free on a laptop in demo mode (on the test set it catches 16 errors with a 69% check pass rate), and for any site widget you can easily export 100 chats once a week and run them through two AI windows using those same five steps.

Don't worry if your first audit reveals the bot is fibbing to customers, and don't try to rewrite the entire prompt in one evening. Friction where natural speech meets algorithms is everyday routine even at Google. Open your assistant's prompt, find a couple of soft words like "preferably," and replace them with firm prohibitions—one fixed line will save tomorrow's orders.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 2