AI crawler
ClaudeBot
Also: Anthropic training crawler, Claude crawler
Anthropic documents each agent with its own robots.txt control. Blocking ClaudeBot opts a site out of training and does not remove it from Claude search. Blocking Claude-SearchBot can keep pages out of Claude search results.
The older names anthropic-ai and Claude-Web are deprecated. No explicit Allow is needed for any agent, because a crawler with no matching rule is allowed by default.
Sources
- Anthropic lists ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user requested fetches) as separate agents (Anthropic Help Center, "Does Anthropic crawl data from the web").