robots.txt for AI bots
Term from the world of AI visibility – explained simply by the GeoStars team.
Rules in robots.txt that define which AI crawlers are allowed to read your website – sensibly separated by training, search index and live fetch.
Want to know how your brand stands on this topic? GeoStars measures your AI visibility across ChatGPT, Google AI Overviews, Gemini, Perplexity, Microsoft Copilot and Google AI Mode – First measurement in minutes · 7 days free, charged only afterwards.
Which entries exist
Every provider uses its own names. OpenAI distinguishes GPTBot for training, OAI-SearchBot for search in ChatGPT, and ChatGPT-User for fetches on user request. Perplexity uses PerplexityBot, Anthropic uses ClaudeBot. Google-Extended is not a crawler of its own, but a control signal: it governs whether content is used for Gemini models. According to Google, it does not control the AI Overviews in Google Search.
The most common mistake
Blocking all AI bots across the board to prevent training often also blocks the search crawlers – then an AI can no longer use the page as a source. A deliberate decision per purpose is better. GeoStars checks robots.txt against 48 AI bots and shows weekly which ones have been newly blocked or allowed again.
More in the Glossary.
Set your course
today.
Start free – in a few minutes you will know where your brand stands across the AI universe.