Reading a robots.txt for AI crawler access: the user agents to check first
Open /robots.txt and check that nothing blocks the crawlers behind AI search citations. Training crawlers are a separate, licensing choice.
Type your domain followed by /robots.txt into a browser (for example https://yourdomain.com/robots.txt). The file loads as plain text.
Look for rules that block the crawlers behind AI search. For ChatGPT these are OAI-SearchBot, which indexes pages for ChatGPT search, and ChatGPT-User, which fetches a page when a user asks. For Claude they are Claude-SearchBot and Claude-User. For Perplexity it is PerplexityBot. Google AI Overviews and AI Mode use Googlebot, and Microsoft Copilot uses Bingbot.
No explicit Allow is needed. A crawler with no matching rule is allowed by default. What causes harm is a Disallow: / under one of those names, or under User-agent: *.
GPTBot, ClaudeBot, Google-Extended and Applebot-Extended are different. They control whether content may be used to train models. Blocking them is a licensing decision and does not remove a site from ChatGPT, Claude, Google or Apple search answers.
robots.txt is not the only gate. Firewall and CDN bot settings can block crawlers before they ever read the file, so check those settings too.