Which crawlers should you allow for AI search?
Allow the search and citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Googlebot, Bingbot and Applebot, because blocking them removes you from those AI answers. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot are training controls, and Google-Extended does not affect AI Overviews or AI Mode. Check your CDN too, because since 15 Sep 2026 Cloudflare's Block setting also blocks Googlebot, Bingbot and Applebot; the KailxLabs $4,995 answer ready build checks all of this.
Which crawlers power AI answers?
These crawlers decide whether an AI app can find and cite your pages. Blocking any of them removes you from that app's answers.
| Crawler | Operator | What it does | Advice |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds and indexes pages for ChatGPT search answers and citations | Allow |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user or agent asks for it | Allow |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers | Allow |
| Perplexity-User | Perplexity | Fetches a page live for a user question | Allow |
| Claude-SearchBot | Anthropic | Indexes pages for Claude search answers | Allow |
| Claude-User | Anthropic | Fetches a page when a Claude user asks for it | Allow |
| Googlebot | Google Search, including AI Overviews and AI Mode | Allow | |
| Bingbot | Microsoft | Bing results and Copilot answers | Allow |
| Applebot | Apple | Apple search and Siri AI answers, which may link to sources | Allow |
Operator crawler documentation · Apple Applebot page, published 4 Sep 2026 · checked 26 Sep 2026
Which crawlers control training?
These govern whether your content trains AI models. Allowing or blocking them is a licensing decision, not a visibility one.
| Crawler | Operator | What it controls | Advice |
|---|---|---|---|
| GPTBot | OpenAI | Model training. Blocking it does not remove you from ChatGPT search. | Your choice |
| ClaudeBot | Anthropic | Model training. Claude search uses Claude-SearchBot. | Your choice |
| Google-Extended | Use of your content for training Google models. It does not affect AI Overviews or AI Mode. | Your choice | |
| Applebot-Extended | Apple | Training only. Apple says its rules are not considered in search ranking. | Your choice |
| CCBot | Common Crawl | An open web archive that many AI models are trained on. | Your choice |
Google Search Central help on the Search generative AI control, 31 Aug 2026 · Apple Applebot page, 4 Sep 2026
Which other settings block AI?
robots.txt is only one layer. Three newer settings can remove you from AI answers without touching it.
Block now blocks search
Block and Block on pages with ads now also stop Googlebot, Bingbot and Applebot. A separate Disallow AI Training setting handles training. New domains are offered presets at signup, which you can change.
Cloudflare blog · 15 Sep 2026
Search generative AI control
A Search Console setting, now worldwide, that removes a site from AI Overviews, AI Mode and Discover's AI features. It does not govern training. Make sure it is off unless you chose it.
Google Search Console help · 31 Aug 2026
nosnippet
A nosnippet rule opts a page out of Siri and Apple search AI answers. Pages marked as not free to access are not used as AI context.
Apple Applebot documentation · 4 Sep 2026
How do you check access?
Test what each crawler actually gets, not what your settings page says.
- Read robots.txt for each user agent above, including rules under the catch all group.
- Check the CDN and firewall. Bot rules, challenges and rate limits can block a crawler that robots.txt allows.
- Check your server logs for each crawler, and verify them against the operator's published IP ranges, because user agents can be faked.
- Check page level tags such as nosnippet and noindex on the pages you want cited.
- Check Search Console for the Search generative AI control, and export the free AI impression reports while you are there.
Access checked, then earned
Answer ready build
$4,995 one time. Crawler access checked for every AI search crawler, at robots.txt, CDN and page level, plus the pages and facts behind it.
Answer ready websiteAI visibility baseline
$1,500 one time. Whether each AI app names you once it can read you.
About the baselineAI shopping readiness
$4,995 one time. Crawler access plus feeds for each AI shopping program.
AI shopping readinessAI crawler questions, answered.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is a training crawler. ChatGPT search relies on OAI-SearchBot, and live page fetches come from ChatGPT-User. Blocking GPTBot is a licensing choice, not a visibility one.
Does Google-Extended control AI Overviews?
No. Google-Extended governs training. To leave AI Overviews, AI Mode and Discover's AI features, Google now offers a separate Search generative AI control in Search Console, worldwide since 31 Aug 2026.
I use Cloudflare. Am I blocking AI search?
Check your bot settings. Since 15 Sep 2026, Cloudflare's Block and Block on pages with ads settings also apply to Googlebot, Bingbot and Applebot, so they cut normal search too. Cloudflare also added a separate Disallow AI Training setting.
Is allowing crawlers enough to get cited?
No. Access makes you eligible; it does not earn a mention. The $4,995 answer ready build checks access for every AI search crawler and fixes the pages and facts behind it.
- 26 Sep 2026: page published with the Cloudflare, Google and Apple changes of August and September 2026.
Talk to the founder.
20 minutes with Kailesk Khumar. You leave knowing where AI names you today, and what it would take to become the answer.
- A live look. We ask five of your buyer questions on ChatGPT, Google and Perplexity while you watch.
- The reasons. The three biggest reasons AI names a rival instead of you.
- The plan. A clear next step and its price, whether or not you hire us.
Pick a time that suits you. The calendar invite arrives straight away. Prefer a link? Book on Cal.com.