AI crawler

GPTBot

Also: GPT bot, OpenAI GPTBot, ChatGPT crawler

GPTBot is one of three primary AI crawlers a clinic, firm, or contractor website must welcome. ClaudeBot is Anthropic's equivalent. PerplexityBot is Perplexity's. Google-Extended is Google's training data signal for Gemini. A correctly configured robots.txt names each of these with an explicit Allow directive.

Many Wix and Squarespace defaults from 2023 silently added user agent disallows for GPTBot. A clinic site that has not been audited since 2023 may be blocking the crawler without the owner knowing. The fix is to open /robots.txt and replace the legacy block with explicit Allow directives.

Related