AI search answers
How do I let AI crawlers read my site?
Short answer: Allow the AI agents explicitly in robots.txt — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot, Applebot-Extended, CCBot and the rest — and repeat your full disallow list inside every named group. A named user-agent group replaces the wildcard group rather than adding to it, so an incomplete named group grants or blocks far more than intended.
The rule that causes most accidental blocks
In robots.txt, a crawler obeys only the most specific group that matches it. If you write a group for GPTBot with a single Disallow line, everything else in your wildcard group stops applying to GPTBot. The reverse also happens: a named group intended as a courtesy ends up exposing paths the wildcard group protected.
The safe pattern is to repeat the entire disallow list in every named group, which is exactly what this site does.
Training crawlers vs retrieval crawlers
GPTBot, ClaudeBot, CCBot and Google-Extended feed training and generative-feature corpora. OAI-SearchBot, Claude-SearchBot, PerplexityBot and ChatGPT-User fetch pages to answer a live question. If you want to be recommended, both matter — blocking the retrieval agents is the more immediate loss.
Access is not only robots.txt
- Server-render your content — many AI fetchers do not execute JavaScript
- Do not gate key pages behind interstitials, consent walls or logins
- Check for bot filtering at the CDN or WAF layer, which robots.txt cannot override
- Return real 200 responses, not soft 404s, for pages you want cited
How to verify
Fetch your own pages with each agent’s user-agent string and confirm you get the full HTML back. Check your server logs for the AI agents by name. If they never appear, something upstream of robots.txt is refusing them.
People also ask
- Should I block AI crawlers to protect my content?
- That is a real business decision, but be clear about the trade: blocking them removes you from the answers they generate. You cannot be recommended by a model that is not allowed to read you.
- Does allowing AI crawlers hurt my Google rankings?
- No. Google-Extended governs generative features only; it is separate from Googlebot and standard search indexing.
Keep reading
Get named by AI instead of your competitor
GEO by MyFast.ai publishes the sourced content, seeds the off-site conversation and scans six engines monthly so you can see exactly where you stand. $1,499/month, no contract.