Definition
An AI crawler is a bot run by an AI company to fetch web pages, either to train models (such as GPTBot or ClaudeBot) or to answer users in real time (such as OAI-SearchBot or Claude-SearchBot).
Also called AI bot, GPTBot, ClaudeBot
MegaMCP glossary32 terms
Explained
Each crawler has its own user agent, so robots.txt can allow AI search crawlers while blocking training crawlers. Google-Extended is a robots.txt token controlling use of content for Google's Gemini models; it doesn't affect Google Search.
Firewalls and bot protection often block AI crawlers even when robots.txt allows them, so check both.