AI crawler
GPTBot
Also: GPT bot, OpenAI GPTBot
OpenAI documents three separate user agents, each controlled on its own line in robots.txt. GPTBot is for training. OAI-SearchBot finds and indexes pages for ChatGPT search answers and citations. ChatGPT-User fetches a page when a person asks ChatGPT to open it.
No explicit Allow is needed for any of them: a crawler with no matching rule is allowed by default. What matters for citations is that nothing blocks OAI-SearchBot, including firewall or CDN bot settings, and that the page content is present in the HTML the server returns.
Sources
- OpenAI lists GPTBot, OAI-SearchBot and ChatGPT-User as separate crawlers with separate robots.txt controls (OpenAI, "Overview of OpenAI Crawlers", platform.openai.com/docs/bots).