AI crawlers explained: OAI-SearchBot, GPTBot, ClaudeBot and what to allow

AI companies run separate crawlers for three jobs: collecting pages to train models, building the search index their answers cite, and fetching a page when a user asks for it. At OpenAI, GPTBot collects training data, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches on demand. Blocking GPTBot doesn't remove you from ChatGPT search. Blocking OAI-SearchBot does: OpenAI says opted-out sites "will not be shown in ChatGPT search answers."

Anthropic, Perplexity, Meta, Amazon and Mistral split their bots the same way. Google and Microsoft don't. Their AI answers come from the normal Googlebot and Bingbot crawl, so the controls sit elsewhere.

I read every AI company's crawler documentation on October 24, 2026. Below: the full crawler table, three robots.txt files to copy, a Cloudflare setting that now blocks Google, and how to spot a fake crawler.

The three jobs AI crawlers do

Training crawlers collect pages that may train future models: GPTBot, ClaudeBot, Meta-ExternalAgent, MistralAI-Training and CCBot. By the owners' accounts, blocking a pure training crawler leaves their AI search alone.

Search crawlers index pages ahead of time so a product can find and cite them: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Amzn-SearchBot and MistralAI-Index. Blocking one is how you leave that product's AI search.

User-triggered fetchers grab one page because a person asked: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User, MistralAI-User and Google-Agent. Several owners say these may not follow robots.txt, because a human started the request.

Two things break the pattern. Google-Extended and Applebot-Extended are control tokens with no crawler behind them. And Applebot, Amazonbot and Meta-ExternalAgent mix jobs, so blocking them can cost more than training.

Also, robots.txt is a request. RFC 9309 says its rules "are not a form of access authorization." Put anything private behind a login.

AI crawler list: every documented bot by company

Every AI-related crawler documented as of October 24, 2026. The token goes after User-agent: in robots.txt. The robots.txt column is what the owner says, and "Little" means the owner says the fetcher may ignore the rule anyway. Company names link to their docs.

GPTBot vs OAI-SearchBot vs ChatGPT-User

OpenAI's crawler page says its settings are independent: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot."

OAI-SearchBot: the crawler behind ChatGPT search

OAI-SearchBot crawls pages so ChatGPT search can find and cite them. OpenAI says opted-out sites "will not be shown in ChatGPT search answers, though can still appear as navigational links," and that robots.txt changes take "~24 hours" to apply. Its ChatGPT search help article adds a second condition: "confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses."

Here is OpenAI's example user agent. The version may change, so match on the token. The IP ranges are at openai.com/searchbot.json.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

GPTBot: training only

GPTBot collects pages that may train OpenAI's models: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." It doesn't affect ChatGPT search.

Should you block it? If you sell something and want AI assistants to know what you do, I'd allow it. If your product is the text itself, blocking is a reasonable call. I haven't seen published evidence that training data changes whether ChatGPT recommends a brand, so don't decide on the hope of a boost.

ChatGPT-User and OpenAI's other agents

ChatGPT-User fetches a page when a ChatGPT user, a custom GPT or a GPT Action asks for it. OpenAI says "robots.txt rules may not apply" and that it "is not used to determine whether content may appear in Search." In a 2025 test, though, Cloudflare saw it fetch robots.txt and stop when disallowed.

OAI-AdsBot only checks ChatGPT ad landing pages. OpenAI's cloud browser agent has no token at all. Its allowlisting article says it signs requests with HTTP Message Signatures, also called Web Bot Auth, instead.

What ChatGPT tends to cite once it can read your pages is covered in how to get cited and recommended by ChatGPT.

What is ClaudeBot? Anthropic's three crawlers

ClaudeBot is Anthropic's crawler for collecting training data for Claude. It has nothing to do with Clawdbot or OpenClaw, assistant apps with similar names. Anthropic's crawler page, updated April 7, 2026, lists three bots:

• ClaudeBot. Disallowing it means "future materials should be excluded from our AI model training datasets."
• Claude-SearchBot, which indexes for Claude's search. Disallowing it "may reduce your site's visibility and accuracy in user search results."
• Claude-User, which fetches pages when a user asks. Disallowing it "may reduce your site's visibility for user-directed web search."

Unlike OpenAI, Perplexity, Google, Meta and Amazon, Anthropic makes no exception for its user fetcher: "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt." It publishes one IP list for all three bots and warns that blocking its IPs "impedes our ability to read your robots.txt file." For Anthropic, robots.txt is the control to use.

Older lists name anthropic-ai and Claude-Web. An Anthropic spokesperson told 404 Media in July 2024 that both "are no longer in use." Name the current three.

PerplexityBot, Perplexity-User and the Cloudflare dispute

Perplexity has no training crawler. It says it "does not build foundation models, so your content will not be used for AI model pre-training." Its crawler docs list PerplexityBot, which builds the search index, and Perplexity-User, which fetches pages for users and "generally ignores robots.txt rules."

Blocking PerplexityBot is partial. Perplexity won't index the page text but "may still index the domain, headline, and a brief factual summary." Its own pages also disagree: the help center says a feature that summarized robots-blocked URLs "has been disabled to prevent misuse," while the docs still say Perplexity-User generally ignores robots.txt.

The August 2025 dispute

On August 4, 2025, Cloudflare accused Perplexity of using "stealth, undeclared crawlers." It says it built new, unindexed domains that disallowed all bots and blocked both Perplexity agents, yet Perplexity still answered detailed questions about them. It reported "20-25m daily requests" from the declared Perplexity-User, plus an undeclared agent posing as Chrome on macOS at "3-6m daily requests" from IPs outside Perplexity's ranges. Cloudflare removed Perplexity from its verified bots.

Perplexity replied the same day, arguing that user-driven fetching is different from crawling and that its "user-driven agents do not store the information or train with it." It said Cloudflare "confused Perplexity with 3-6M daily requests of unrelated traffic from BrowserBase," a cloud browser service it says it uses for fewer than 45,000 requests a day.

Neither side published data anyone else can check, so I can't tell you who is right. The practical point holds either way: naming Perplexity-User in robots.txt may not stop user fetches.

Googlebot, Google-Extended and AI Overviews

Google has no separate crawler for AI Overviews or AI Mode. Its generative AI guide says they retrieve pages "from our Search index," and its AI features page calls Googlebot's robots.txt rules "the control for site owners." Block Googlebot and you leave Google Search entirely. Google says there are no extra requirements beyond being indexed and eligible for a snippet. How that overlaps with classic SEO is covered in GEO vs SEO vs AEO.

What Google-Extended controls

Google-Extended is a token with no crawler behind it. Per Google's crawler list, disallowing it keeps your content out of training future Gemini models and out of grounding in Gemini Apps and Vertex AI. The second part is easy to miss: the Gemini app stops using your pages as live sources.

It doesn't touch Search: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." One loose end: Google's Search generative AI control page says Google-Extended limits "training of the models used to generate responses in Search generative AI features," which the Google-Extended entry doesn't mention. Google's two pages don't match.

The Search Console setting for AI features

Since August 31, 2026, Search Console has a setting under Settings > Search generative AI that includes or excludes a site from AI Overviews, AI Mode and generative features in Discover. Include is the default. Google says the setting "isn't used as a ranking or inclusion signal affecting other parts of Search" and "doesn't affect AI training." Changes take "a few days."

The older route is snippet controls, per Google's robots meta tag spec. nosnippet stops a page being used as a direct input for AI Overviews and AI Mode, max-snippet limits how much is used, and noindex removes the page from Search. They cost more, because a page must be eligible to show "with a snippet" to be a supporting link. And they only work if Googlebot can crawl the page.

Microsoft: Copilot runs on Bingbot

Microsoft documents no Copilot crawler. Its webmaster guidelines say "Bing and Copilot search experiences rely on the same core crawling, indexing, and ranking foundation as traditional search," and warn against blocking Bingbot.

The AI controls are meta tags. noarchive "prevents content from being used in Copilot responses and grounding results," and Microsoft's 2023 announcement says such pages stay out of training and still appear in search. nocache "limits Copilot to using only the URL, title, and snippet." There's no robots.txt training opt-out. Cloudflare reported in September 2026 that Microsoft is building one, "targeted for early 2027." I couldn't find that on a Microsoft page.

Bing Webmaster Tools has had an AI Performance report since February 10, 2026. It counts citations in Copilot and Bing's AI summaries, but not clicks.

Apple, Meta, Amazon and the rest

Apple uses Applebot for Spotlight, Siri and Safari search, and says its data "may also be used to help train Apple foundation models." Applebot-Extended "does not crawl webpages." Disallowing it opts you out of Apple model training without leaving search, but not out of AI answers in Siri and Search. Apple says to use nosnippet for that.

Meta says Meta-ExternalAgent crawls for "training foundation AI models or improving products by indexing content directly," so blocking it may cost more than training. Allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses." Meta-ExternalFetcher fetches for users and "may bypass robots.txt rules." FacebookBot no longer appears in Meta's docs.

Amazon separates Amazonbot, which "may be used to train Amazon AI models," from Amzn-SearchBot, which serves search such as Alexa and "does not crawl content for generative AI model training." Amzn-User fetches for users and "may not follow all robots.txt directives."

CCBot belongs to Common Crawl, a free public web archive that many model builders download. A Mozilla Foundation report found that at least 64% of 47 text LLMs released between 2019 and October 2023 used filtered Common Crawl data. Blocking CCBot stops future crawling. Common Crawl doesn't say it removes past crawls.

ByteDance's Bytespider page describes a crawler for Toutiao Search and doesn't mention AI training, though third-party lists call it a training crawler. One academic study saw it fetch robots.txt and then not respect it during its test. DuckDuckGo says DuckAssistBot data "is not used in any way to train AI models," and a disallow takes effect after 72 hours.

xAI, which makes Grok, documents no crawler. Its full developer docs said nothing about robots.txt or crawler names when I checked. GrokBot and similar tokens come from third-party lists, so a rule for them may do nothing.

robots.txt for AI: three files you can copy

My default for a company that wants AI assistants to recommend it is file 2, allow everything. File 1 suits publishers and anyone whose product is the text. Three rules apply to all of them:

• A crawler obeys only the group that names it most specifically, per Google's robots.txt spec. One with its own group ignores *, so copy shared Disallow lines into it.
• Tokens are case-insensitive. Paths are case-sensitive.
• Unnamed crawlers can borrow rules. Applebot follows the Googlebot group, and Amzn-SearchBot follows other search bots.

File 1: show up in AI answers, opt out of training

# Training crawlers and training-control tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /

# Everyone else, including OAI-SearchBot, Claude-SearchBot, PerplexityBot,
# Meta-WebIndexer, Amzn-SearchBot, MistralAI-Index, DuckAssistBot,
# Googlebot, Bingbot and Applebot
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

• Google-Extended also removes you from grounding in the Gemini app. If Gemini citations matter more to you than training, delete that line.
• Meta-ExternalAgent and Amazonbot are mixed-use, so you may lose some product indexing.
• Microsoft has no robots.txt training opt-out, and noarchive also removes you from Copilot. You can't reach this goal for Microsoft today.
• Bytespider is left out because ByteDance documents it as a search crawler. Add it if you want ByteDance out entirely.

File 2: allow everything

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

An empty file or none at all works the same: RFC 9309 lets a crawler "access any resources" when robots.txt returns a 4xx error. If AI crawlers still can't reach you, look at your CDN or firewall.

File 3: block all AI crawlers, keep classic search

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: CCBot
User-agent: Bytespider
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: MistralAI-Training
Disallow: /

User-agent: *
Allow: /

What this file can't do:

• Keep you out of AI Overviews and AI Mode. Use the Search Console setting.
• Keep you out of Copilot. Use noarchive or nocache.
• Keep you out of Apple's AI answers. Use nosnippet.
• Stop user fetchers whose owners say they may ignore it: ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User and Google-Agent. That takes firewall rules on user agent plus published IP ranges, or signature checks. Anthropic says Claude-User obeys.
• Name agents that declare no token, such as anything from xAI.

It also takes you out of ChatGPT search, Claude's search, most of Perplexity's index and Meta AI's citations. For a business that wants to be recommended, I think that's usually the wrong trade.

Cloudflare users: "Block" now blocks Google

If Cloudflare sits in front of your site, its settings decide what crawlers reach before robots.txt matters. On September 15, 2026, Cloudflare made its "Block" and "Block on pages with ads" training settings apply to mixed-use crawlers "including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training." Blocking AI training that way can now cut you out of Google, Bing and Apple search.

To refuse training and keep search, use the new Disallow AI Training setting. Calling Googlebot "mixed-use" is Cloudflare's label, since Google puts training control in Google-Extended. Cloudflare's July 2026 press release and its September post also disagree on what existing free sites got by default, so check your dashboard.

1. Set Search to Allow under Security > Settings > AI bot policies.
2. For Training, choose Disallow AI Training, not Block or "Block on pages with ads."
3. For Agent, "Block on pages with ads" blocks chat fetchers like ChatGPT-User on ad pages. Cloudflare doesn't publish the exact list. Its AI Assistant category is my best guess.
4. In AI Crawl Control, confirm OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed.
5. Read your live robots.txt. Bot Preference Sync puts Cloudflare's rules before yours, and its example managed block disallows GPTBot, ClaudeBot, Google-Extended, CCBot and others.
6. Review WAF rules and Bot Fight Mode. OpenAI and Perplexity ask you to allow their IP ranges, and Cloudflare says Bot Fight Mode "cannot be bypassed or skipped" with WAF custom rules.
7. Check your server logs. Search crawlers should get 200, not 403 or a challenge page.

How to check and verify AI crawlers in your logs

After robots.txt and your CDN, your server logs are the real test. Search them for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot, and look for 200 responses rather than 403s or challenge pages. Faking a crawler's user agent from your laptop only tests user-agent rules.

Anyone can claim to be GPTBot, so confirm each visitor. Vendors document up to three checks:

1. Run a forward-confirmed reverse DNS lookup. Find the IP's hostname, check the domain, then confirm the hostname resolves back to the same IP.
2. Match the IP against the vendor's published list.
3. Check request signatures. OpenAI's cloud browser agent signs requests, and Google is experimenting with signing for Google-Agent.

Google's documented example:

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Re-download the lists on a schedule. Bing says to refresh daily. A request that fails its vendor's checks is probably an impersonator, and Common Crawl's FAQ warns about crawlers that "falsely identify themselves as CCBot."

Inkie's website checks include whether AI search crawlers can reach your site. To check what ChatGPT, Gemini, Perplexity, AI Overviews and AI Mode say about you today, the free AI visibility report runs 5 buyer questions across all five, no card needed. For ongoing tracking, I compared the options in best AI visibility tools.

Does llms.txt help with AI crawlers?

llms.txt is a Markdown file at /llms.txt that summarizes a site for language models. Jeremy Howard of Answer.AI proposed it in September 2024. It doesn't control crawling.

Google's AI optimization guide says it "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." None of the other crawler docs I read for this guide say their crawlers read it. Several publish one for their own docs, which isn't the same thing. In a 90-day, single-site test by Otterly, which sells AI monitoring, the file got 84 requests among more than 62,100 AI-bot visits.

It's optional. Nobody has shown it changes whether ChatGPT or Google cites you, so fix crawler access first. The format and the evidence are in llms.txt explained.

Common mistakes with AI crawlers and robots.txt

• Disallowing OAI-SearchBot to stop training. That removes you from ChatGPT search. Disallow GPTBot alone.
• Blocking Google-Extended to leave AI Overviews. It has no effect there.
• Giving a bot its own group and losing the shared rules. A new OAI-SearchBot group ignores the Disallow: /admin/ you wrote under *.
• Using retired or guessed tokens such as anthropic-ai, Claude-Web, FacebookBot or GrokBot. None is a current documented token.
• Letting a firewall overrule robots.txt. Bot Fight Mode, WAF rules and rate limits can block crawlers your file allows.
• Pairing a robots.txt disallow with noindex or nosnippet. A crawler can't read a tag on a page it can't fetch. And on Google, nosnippet also drops you from AI Overview links.
• Trusting Content-Signal lines like ai-train=no. No crawler doc I read says it reads them.

Frequently asked questions

What is the ClaudeBot?

ClaudeBot is Anthropic's crawler for collecting Claude training data, and Anthropic says it honors robots.txt. Blocking it doesn't block Claude-SearchBot or Claude-User, which serve Claude's answers. It is unrelated to Clawdbot and OpenClaw, assistant apps with similar names.

What are the top 10 web crawlers?

It depends on the site, so check your server logs. For search and AI visibility, the ten I'd watch are Googlebot, Bingbot, Applebot, OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and CCBot. Googlebot and Bingbot matter most, because they also feed Google's AI features and Microsoft Copilot.

Should I block GPTBot?

Only if you don't want OpenAI training on your content. Blocking GPTBot has no effect on ChatGPT search, which uses OAI-SearchBot. For most companies that want AI assistants to know them, I'd allow it.

Does blocking GPTBot remove me from ChatGPT?

No. OpenAI says its crawler settings are independent, so you can disallow GPTBot and still appear in ChatGPT search. Blocking OAI-SearchBot is what removes you, though your site can still show up as a navigational link.

Does Google-Extended affect AI Overviews?

No. Google says Google-Extended doesn't affect inclusion in Google Search, and AI Overviews and AI Mode are part of Search. Blocking it opts you out of Gemini training and grounding in the Gemini app. To leave AI Overviews and AI Mode, use the Search generative AI setting in Search Console.

How do I check if AI crawlers can reach my site?

Read your live robots.txt and find the groups that apply to OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. Check your CDN and firewall settings. Then search your server logs for those crawlers and confirm they get 200 responses, not 403s or challenge pages.

Your next step

Open your live robots.txt and find the group that applies to OAI-SearchBot. If it's blocked there, or your CDN blocks it, fix that first, because ChatGPT search can't show pages its crawler can't read. Then decide on training separately.

Access makes you eligible. It doesn't make AI answers mention you. To see whether ChatGPT, Gemini, Perplexity and Google's AI features name, recommend or cite you today, run the free AI visibility report.