Key points
- Most AI companies run separate crawlers for building a search index, fetching pages when a user asks, and collecting training data. You can allow one and block another in robots.txt.
- Blocking a company’s search crawler, such as OAI-SearchBot or Claude-SearchBot, can remove your site from that assistant’s search answers. Blocking a training crawler, such as GPTBot or ClaudeBot, does not.
- Bots that fetch a page because a user asked may not follow robots.txt, and in 2025 Cloudflare accused Perplexity of using undeclared crawlers. Robots.txt is a request, not a lock.
AI assistants can only cite what they can read. Before worrying about content, check which AI crawlers can reach your site and which you have blocked, often without meaning to. This article is part of our AI search visibility guide.
The main AI crawlers
| Company | Crawler | What it does | Follows robots.txt? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Finds pages to show in ChatGPT search | Yes |
| OpenAI | GPTBot | Collects content that may be used to train OpenAI models | Yes |
| OpenAI | ChatGPT-User | Visits a page when a ChatGPT user’s request needs it | May not apply |
| Anthropic | Claude-SearchBot | Indexes content to improve Claude’s search results | Yes |
| Anthropic | ClaudeBot | Collects content that could contribute to training | Yes |
| Anthropic | Claude-User | Fetches a page when a Claude user asks a question | Yes |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity results; not used for training | Yes |
| Perplexity | Perplexity-User | Fetches pages during user requests | Generally no |
| Google-Extended | A robots.txt token controlling use of content for Gemini training and grounding | Yes (token only) |
Sources: OpenAI (OpenAI), Anthropic (Anthropic), Perplexity (Perplexity) and Google (Google).
Search, user and training crawlers are different
The most important point is that these jobs are separate. OpenAI says that if a site blocks OAI-SearchBot it will not appear in ChatGPT search answers, while blocking GPTBot signals that content should not be used for training (OpenAI). Anthropic similarly warns that blocking Claude-SearchBot or Claude-User may reduce a site’s visibility in Claude’s search results (Anthropic). Google says Google-Extended does not affect inclusion or ranking in Google Search (Google).
So a publisher can make a deliberate choice: allow AI search crawlers so its journalism can be found and cited, and decide separately whether to allow training crawlers.
Example robots.txt rules
To allow AI search but opt out of training by OpenAI and Anthropic:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Changes are not instant. OpenAI and Perplexity say robots.txt changes can take about 24 hours to take effect (OpenAI). Anthropic advises against blocking its IP addresses as a way to opt out, because it may not work reliably (Anthropic).
Check your firewall and hosting too
Robots.txt is not the only gate. Security plugins, content delivery networks and hosting firewalls often block unfamiliar bots by default, and some now block AI crawlers unless told otherwise. If AI crawlers never appear in your logs, check these settings before anything else.
The limits of robots.txt
Robots.txt is a voluntary standard. Crawlers that act on a user’s behalf may not follow it: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it (Perplexity). In August 2025, Cloudflare said Perplexity had used undeclared crawlers that disguised themselves as ordinary browsers to reach sites that had blocked it, a claim Perplexity rejected (TechRepublic). Publishers who need firm control have to combine robots.txt with firewall rules.
A worked example
Newsinterpretation’s robots.txt currently allows all crawlers. NI Geovanta, the plugin we use, classifies AI bot visits into three groups. In the 30 days to 22 October 2026 it logged about 25,800 visits from training crawlers, about 7,500 from AI search indexers and 132 visits in which an assistant opened a page to answer someone’s question. That mix, heavy on training and light on answering, is typical of a site that AI systems read but rarely cite. Its count is also a minimum: bot requests served from a server page cache never reach WordPress.
Frequently asked questions
What is GPTBot?
GPTBot is OpenAI’s crawler for collecting content that may be used to train its models. It is separate from OAI-SearchBot, which finds pages for ChatGPT search.
If I block GPTBot, will my site disappear from ChatGPT?
Not from ChatGPT search, according to OpenAI. Search inclusion is controlled by OAI-SearchBot. Blocking GPTBot only signals that content should not be used for training.
Does blocking Google-Extended affect Google Search?
No. Google says Google-Extended does not affect a site’s inclusion in Google Search or its ranking.
Do all AI crawlers obey robots.txt?
No. Declared search and training crawlers from major companies say they do, but bots that fetch pages on a user’s behalf may not, and Cloudflare has accused Perplexity of using undeclared crawlers.
Last reviewed: 22 October 2026. Sources are linked inline.
Disclosure: NI Geovanta is developed by Newsinterpretation, the publisher of this article. It is used here as a worked example; other tools can perform similar checks.
