Is your website blocking ChatGPT without you knowing? A guide to AI crawlers
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot: what each AI crawler does, how to set up robots.txt and what Cloudflare's new rules change for your site.
In short
ChatGPT, Claude and Perplexity can only cite you if their search crawlers can read your site. You can block model training and stay citable, but you need to tell the right crawlers apart in robots.txt and check your CDN and firewall too, starting with Cloudflare's new settings of 15 September 2026.
- Every AI provider uses different crawlers for search, for training and for user requests.
- Blocking GPTBot or ClaudeBot stops training; blocking OAI-SearchBot or Claude-SearchBot removes you from answers.
- Since 15 September 2026 Cloudflare blocks Training and Agent by default on ad pages of new domains.
- The check takes ten minutes: robots.txt, CDN settings, content in the HTML and server logs.
Written by the Geosnap AI agent, reviewed and approved by Rinald Sefa.
Translated from the Italian original. Read the original
If ChatGPT, Claude or Perplexity never cite your website, the cause may be more mundane than you think: a rule in your robots.txt file, a CDN setting or a firewall that keeps their crawlers out. Nobody notices, because people still see the site as usual. This guide covers which AI crawlers exist, what each one does, how to decide which ones to let in, and what changed with Cloudflare's new settings.
Why robots.txt matters for getting cited by AI
AI assistants can only cite pages they can read. Before answering a question, ChatGPT, Claude and Perplexity search the web with their own crawlers: if your site blocks them, someone else shows up in the answer.
robots.txt is the file at the root of your site that tells crawlers what they may visit. The AI crawlers of the major providers say they respect it, so a single Disallow line is enough to make you disappear from their searches. The problem is that many of these lines were added in recent years to block model training, without distinguishing between crawlers that collect training data and crawlers looking for sources to cite.
Which AI crawlers exist and what they do
Each provider runs several crawlers with different purposes. These are the officially documented ones:
| Provider | User agent | Purpose | Respects robots.txt |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Finding sites to show in ChatGPT search | Yes |
| OpenAI | GPTBot | Collecting content to train models | Yes |
| OpenAI | ChatGPT-User | Visiting a page when a user asks | Not always |
| Anthropic | Claude-SearchBot | Indexing content for Claude's searches | Yes |
| Anthropic | ClaudeBot | Collecting content for training | Yes |
| Anthropic | Claude-User | Reading a page at a user's request | Yes |
| Perplexity | PerplexityBot | Surfacing and linking sites in Perplexity results | Yes |
| Perplexity | Perplexity-User | Visiting a page at a user's request | Generally no |
Google-Extended | Use of content to train Gemini and for grounding | Yes |
OpenAI explains that sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, at most as navigational links. Perplexity notes that Perplexity-User generally ignores robots.txt, because a person requested the visit. Google, for its part, states that Google-Extended does not affect inclusion in Google Search or ranking.
Search and training can be separated
Yes: you can keep your content out of model training and still be citable. OpenAI says so explicitly: each setting is independent of the others, so you can allow OAI-SearchBot and block GPTBot. The same goes for Anthropic's crawlers: according to Claude's documentation, blocking ClaudeBot excludes future content from training, while blocking Claude-SearchBot may reduce your visibility in results.
A robots.txt that stays visible in answers without contributing to training can look like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /It's a choice, not a rule: many brands prefer to leave training open too, because models learn about the brand. What matters is that it's a deliberate decision, not a side effect.
Cloudflare: what changed on 15 September 2026
If your site runs through Cloudflare, check its bot settings: CDN rules apply even when robots.txt is open. In July 2026 Cloudflare split AI crawlers into three categories: Search (indexing for search results), Agent (automated actions on behalf of users) and Training (data collection for models).
From 15 September 2026, for new domains, Training and Agent are blocked by default on pages that display ads, while Search stays allowed. On the same day Cloudflare updated how it handles mixed-use crawlers, such as those from Google, Apple and Microsoft that serve both search and training: blocks now apply to them too. In practice, an overly strict setting can remove a site from classic search results as well.
How to check your site in 10 minutes
- Open your robots.txt by adding
/robots.txtto your site's address. Look forDisallowlines under the user agents in the table, and for aUser-agent: *rule withDisallow: /, often left behind after a redesign. - Check your CDN and firewall. On Cloudflare, look at the AI bot settings in the domain's Security section; on other services, look for rules blocking user agents or IP ranges. Anthropic warns that blocking IP addresses can stop crawlers from reading robots.txt, with unpredictable effects.
- Make sure your content is in the HTML. A December 2024 Vercel study found that the major AI crawlers don't run JavaScript: if services, prices and reviews only appear after the page loads, they don't exist for them.
- Look at your server logs to see which crawlers actually visit the site and which pages return errors.
- Give changes time. OpenAI says it takes about 24 hours for a robots.txt update to reach its systems.
In Geosnap's technical analysis these checks are automatic: robots.txt is read for every AI crawler, and errors that block bots show up among the actions to take. To understand which sites AI relies on when it can't read yours, see where LLMs get their information.
Sources
- Overview of OpenAI Crawlers (OpenAI, 2026)
- Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic, 2026)
- Perplexity Crawlers (Perplexity, 2026)
- Google's common crawlers (Google Search Central, 2026)
- Your site, your rules: new AI traffic options for all customers (Cloudflare, 2026-07-01)
- Have it both ways: stay discoverable in search while disallowing AI training (Cloudflare, 2026-09-15)
- The rise of the AI crawler (Vercel, 2024-12-17)
Frequently asked questions
If I block GPTBot, will my site disappear from ChatGPT?
No. GPTBot collects content to train models. What counts for ChatGPT search answers is OAI-SearchBot: if you let it in, your site stays citable even with GPTBot blocked.
Does blocking Google-Extended hurt my Google rankings?
According to Google, no: Google-Extended does not affect inclusion in Google Search or ranking. It covers the use of content to train Gemini and for grounding.
How long does a robots.txt change take to apply?
OpenAI says about 24 hours for its systems. Other providers don't give precise timings: check again after a few days, including in your server logs.
Is robots.txt enough to block every AI crawler?
No. Crawlers acting on a user's request, such as Perplexity-User, may not follow it. Stricter control needs CDN or firewall rules, used carefully so you don't block search crawlers too.