GEO & AI search glossary
AI crawler
What is an AI crawler, and should I allow it?
A bot operated by an AI company that fetches web pages for training or for live retrieval — GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot and others. Blocking them in robots.txt removes you from the answers they generate.
There are two distinct jobs. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) build the corpus a model learns from. Retrieval crawlers and user-triggered fetchers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User) fetch pages at question time to answer a live query.
If your business wants to be recommended, both matter, and a common accident removes you from all of them: naming a bot in robots.txt creates a group that replaces the wildcard rules rather than adding to them, so a partial block can quietly become a full block. This site publishes an explicit allow group for every major AI crawler and repeats the full disallow list inside each one.
Related terms
- llms.txtA plain-text file at the root of a site that summarizes what the site is and points AI systems to its most important pages. This site publishes one at /llms.txt and a longer version at /llms-full.txt.
- Search Engine Optimization (SEO)The long-standing practice of improving a site so it ranks higher in a list of search results. It remains the technical floor GEO is built on: a page a model cannot crawl or read cannot be cited.
- Generative engineAn AI system that answers a question by writing new text rather than returning a list of links — ChatGPT, Claude, Gemini, Grok, DeepSeek and Meta AI are the six we optimize for.
Want AI engines naming your business?
GEO by MyFast.ai does the work for you — GEO-optimized articles published to your own domain, forum swipe files, and monthly visibility scanning across ChatGPT, Claude, Gemini, Grok, DeepSeek and Meta AI. $1,499/month, no contract.