In November 2026, the agency GUSAROV audited .by websites that AI assistants already recommend to customers. They uncovered a strange pattern: roughly one in nine accessible sites blocks AI bots on its own. Even more surprising, in nearly nine out of ten cases, the owner never meant to ban anyone. Everything loads fast in a browser and robots.txt is clean, yet the hosting provider or a security module silently returns an access error to bots.
Why a Blocked Site Hands Warm Leads to Directories
Search-enabled AI tools like ChatGPT, Claude, Perplexity, and Google AI Overviews don't just answer from memory. When someone looks for a contractor or a course, the system sends a search bot to live pages, reads current terms, and builds an answer with links.
OpenAI explicitly warns that a site closed to OAI-SearchBot won't appear in ChatGPT search answers (except as basic navigational links). Anthropic gives the same warning for Claude-SearchBot and Claude-User.
The GUSAROV study also revealed a paradox. Blocked sites are mentioned just as often as open ones, with a median of 7 mentions for both groups. AI still learns about a company from maps, ratings, and directories. The real problem is where it pulls the details from: it grabs prices and terms from third-party platforms where data goes stale, and links to an aggregator right next to your competitors.
That's why you need to track two metrics separately: brand visibility (how often you are named) and site visibility (how often your link is cited). In the EDUGUSAROV school case (edugusarov.by) on October 27, 2026, the brand appeared in 56.1% of answers, while its site link appeared in only 43.3% (65% in ChatGPT Search, 30% in Claude, and ranked first in 76% of mentions). The site had always been open to bots, yet on September 21 it was mentioned in only 30% of answers across 37 course-related prompts. Three different brand spellings, an old address in directories, reviews sitting on someone else's listing, and missing FAQ pages held it back. After five weeks of cleanup, mentions climbed to 56%, putting it first among 131 brands. The authors note honestly that they changed their methodology on October 22 (tracking 5 search-enabled AIs instead of 7 platforms), and numbers still fluctuate under the same setup (54% → 55% → 56% → 50% by November 9), so you should always look at a series of measurements.
"Bot access is the foundation. While it's closed, everything else in AI search optimization works at half strength. The check takes 15 minutes, and it's worth doing before anything else." — Andrey Gusarov, co-founder of GUSAROV
Where Invisible Blocks Hide: Hosting, WAF, and Plugins
Let's quickly clarify three terms. robots.txt is a root file containing access rules (Allow/Disallow) for bots. User-Agent is the ID string a browser or bot sends to introduce itself to a server. A WAF (Web Application Firewall) is a security shield at your hosting provider, in Cloudflare, or inside a CMS plugin that blocks attacks and scrapers with 403 Forbidden or 429 Too Many Requests instead of 200 OK.
On November 18, 2026, GUSAROV tested 948 .by domains that AI assistants had mentioned at least three times between August and November. Each accessible site was checked twice, using both browser and bot User-Agents.
| What was tested on accessible sites | Sites | Share |
|---|---|---|
| Homepage loaded during the check | 811 | 100% |
| Blocks at least one AI bot | 96 | 11.8% |
| Blocked only by site or hosting security (403/429 with clean robots.txt) | 84 | 10.4% |
| Blocked only by robots.txt | 5 | 0.6% |
| Blocked by both security and robots.txt | 7 | 0.9% |
| Fully closed in robots.txt to all bots, including Google | 5 | 0.6% |
Very few companies intentionally block AI search. Out of 765 sites with an accessible robots.txt, only 12 deliberately blocked at least one search bot. Meanwhile, out of 57 sites that named AI bots specifically, 34 did so to allow them.
The invisible wall comes from three hidden sources. First, shared hosting rules: 23 sites returned the exact same generic 403 Forbidden page to Claude, and another 21 returned a different boilerplate block page to GPTBot. Almost all of them sit on a handful of local hosting providers. Second, Cloudflare: on July 1, 2025, as part of Content Independence Day, Cloudflare switched new domains to block AI crawlers by default. Third, CMS security plugins (such as WordPress plugins) automatically reject unfamiliar bots.
How many of 811 sites block bots via security (WAF and hosting)
Anthropic's bot gets blocked noticeably more often than Perplexity, while OpenAI's crawler sits in between. One fair caveat: the test requests were sent from outside Belarus with spoofed User-Agent headers and only checked the homepage. Real bots arrive from their own IP ranges, so only your server logs tell the full story.
Search Bots vs. Training Bots: Who to Let In and Who to Block
You don't have to block every bot out of fear that your articles will train future models. AI platforms separate search crawlers (which drive links), user-triggered agents (which fetch a page on a person's request), and model-training bots.
| Purpose | OpenAI | Anthropic | Perplexity | Recommendation |
|---|---|---|---|---|
| Search and answers with links | OAI-SearchBot | Claude-SearchBot | PerplexityBot | Allow |
| Fetching on user request | ChatGPT-User | Claude-User | Perplexity-User | Allow |
| Base model training | GPTBot | ClaudeBot | — | Business decision |
These settings are independent. OpenAI lets you allow OAI-SearchBot while blocking GPTBot (they also use OAI-AdsBot to check ads). PerplexityBot doesn't train foundation models and applies robots.txt updates within 24 hours. Anthropic respects the Crawl-delay directive (for example, Crawl-delay: 1) to reduce server load without a full ban. Here is a working robots.txt block:
# Allow AI search and user-triggered bots
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
# Duplicate restricted sections from User-agent: *
Disallow: /cart/
Disallow: /account/
# Block model-training bots
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
This setup welcomes bots that bring traffic and citations, while politely closing the door on data collectors for future model training.
A rule block for a specific bot completely overrides the general User-agent: * rules for that bot. If you grant Allow: / to AI search bots and forget to copy over Disallow lines for your cart and account pages, those bots can crawl your private service pages.
15-Minute Diagnostics and Three Silent Interference Issues
You can test your site yourself without hosting access using curl in Terminal (macOS/Linux) or cmd (Windows 10/11). Just replace example.by with your domain:
# 1. Response code: 200 means allowed, 403/429 means blocked by security
curl -s -o /dev/null -w "%{http_code}\n" https://example.by/
curl -s -o /dev/null -w "%{http_code}\n" \
-A "GPTBot/1.2" https://example.by/
curl -s -o /dev/null -w "%{http_code}\n" \
-A "OAI-SearchBot/1.0" https://example.by/
curl -s -o /dev/null -w "%{http_code}\n" \
-A "ClaudeBot/1.0" https://example.by/
curl -s -o /dev/null -w "%{http_code}\n" \
-A "PerplexityBot/1.0" https://example.by/
# 2. Contents of robots.txt
curl -s https://example.by/robots.txt
If a normal request returns 200 but a bot gets 403 or 429, a hidden security rule is firing. Next, have a developer check real visits in nginx logs:
# nginx status codes for AI bots (default log format)
grep -E "OAI-SearchBot|ChatGPT-User|GPTBot|Claude|PerplexityBot" \
/var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
# Request count per bot
grep -oE "OAI-SearchBot|ChatGPT-User|GPTBot|Claude|PerplexityBot" \
/var/log/nginx/access.log | sort | uniq -c
Lots of 403/429 codes mean you need to adjust your security rules. If no bots appear at all, an external WAF is dropping them before they reach your server. Whitelist bots using official IP JSON lists rather than easily spoofed User-Agent strings: openai.com/searchbot.json, openai.com/chatgpt-user.json, perplexity.ai/perplexitybot.json, perplexity-user.json, and claude.com/crawling/bots.json. Also, don't block Anthropic by IP at the firewall level, or its bot won't even be able to read your refusal in robots.txt.
While you're at it, check three quiet issues from the audit. Geoblocking: 137 of 948 sites didn't load from abroad (58 rejected even a browser, 48 timed out), so add official bot IPs to your exceptions. JS rendering: according to Vercel and MERJ tests (December 2024), major AI bots don't execute JavaScript and only see raw HTML; against a median of 9,417 characters, 45 sites (5.5%) served fewer than 500 characters. Forgotten Disallow: /: five sites still blocked everyone, including Google, left over from staging. And don't waste time on llms.txt: Google doesn't use it, and no major AI platform has confirmed reading it.
15-minute diagnostic checklist
- Run
curlon your homepage for a browser and four AI bots — confirm all return200. - Check
robots.txt: ensure there is no globalDisallow: /and no block on AI search bots. - If you see
403or429, check hosting rules and AI-bot blocking in Cloudflare or CMS plugins. - Separate search and training bots in
robots.txt, copying private-sectionDisallowrules fromUser-agent: *. - Send official bot IP JSON lists to your developer for WAF whitelisting, and verify page text appears in raw HTML without JS.
- Check server logs a week later to confirm search bots are visiting and getting
200.





