
One in Nine Sites Blocks AI Bots by Accident: Audit of 811 Companies and a 15-Minute Diagnostic Guide
Out of 811 websites that AI assistants already recommend to customers, 96 block AI bots from crawling them. In 84 of those cases, robots.txt is completely clean. Instead, hosting providers, Cloudflare, or security plugins silently slam the door with 403 and 429 errors, sending warm traffic straight to competitor aggregators. Here is how to separate search bots from training crawlers, avoid common robots.txt traps, and check your site in 15 minutes using curl and server logs.










