AI crawler
ClaudeBot
Also: anthropic-ai, Claude crawler, Anthropic web crawler
ClaudeBot is one of three primary AI crawlers a clinic, firm, or contractor website must welcome alongside GPTBot (OpenAI) and PerplexityBot (Perplexity). The robots.txt user agent string is ClaudeBot and the older variant is anthropic ai. A correctly configured robots.txt names both with explicit Allow directives.
Anthropic has been more transparent than most about crawler behavior. ClaudeBot fetches pages on demand when Claude is asked a question that requires fresh information. The fetched content is not retained as training data unless the site has opted in via signal. The fetch is used to ground the current response with current information.