AI blocks in robots.txt: how many sites actually enforce them? 2026 data
HasData tested 10,894 domains: 39.5% of GPTBot bans exist only on paper, 178 sites block without saying so and 118 lost their rules in Cloudflare's switch.
In short
Only 13% of the 10,894 domains analysed by HasData disallow an AI crawler in robots.txt, but the file tells you little: 39.5% of sites that ban GPTBot still serve it the page, and 178 block it without declaring it. Behind Cloudflare one request in four ends in a verification page, and 118 sites lost their managed rules in the move to Bot Preference Sync. To be cited you need to check the server and the CDN, not just robots.txt.
- In September 2026 only 13% of the 10,894 domains analysed by HasData disallow an AI crawler in robots.txt; publishers block five times more than the rest of the web.
- 39.5% of sites that disallow GPTBot still serve it the page, while 178 sites block it without declaring it: the server decides, not the file.
- Behind Cloudflare, 24.7% of GPTBot requests end in a verification page, and 118 sites lost their managed rules in the move to Bot Preference Sync.
- Blocking crawlers does not remove you from Google AI Mode: 51.9% of cited domains disallow at least one AI crawler. Being reachable is still the condition for ChatGPT search.
Written by the Geosnap AI agent, reviewed and approved by Rinald Sefa.
Translated from the Italian original. Read the original
The robots.txt file is the first place anyone looks to see whether a site is open or closed to AI crawlers. But it only says what the site declares: what actually happens is decided by the server, the CDN and the firewall. In 2026 HasData measured both on almost 11,000 domains, before and after Cloudflare's September 15 switch to new default settings. The results matter to any brand that wants to be cited by ChatGPT, Gemini or Google AI Mode: many sites block without meaning to, and others think they block but don't.
How many sites block AI crawlers in robots.txt?
Few, and fewer every month: in September 2026, 13.0% of the domains analysed disallow at least one AI crawler in robots.txt, down from 15.2% in July. The figures come from HasData's AI Crawler Block Index, updated on October 7, 2026, which covers 10,894 domains: the Tranco top 10,000 plus 1,148 news homepages. The picture is very different for general sites and for publishers.
| Rule in robots.txt | Top web (July → September) | Publishers (July → September) |
|---|---|---|
| Disallows at least one AI crawler | 10.3% → 8.3% | 56.4% → 53.0% |
| Disallows GPTBot (OpenAI) | 7.9% → 5.7% | 50.5% → 46.9% |
| Publishes an llms.txt | 7.9% → 9.3% | 3.2% → 4.5% |
Publishers block about five times more than the rest of the web, and declared blocking is falling in both groups. llms.txt is growing but remains a niche: if you are wondering whether it is worth publishing one, we covered the data in Does llms.txt actually help?.
Two in five AI bans exist only on paper
Declaring a block and enforcing it are two different things. HasData took 2,096 domains (the top 1,000 of the open web plus every publisher) and sent each one two requests from the same IP address: one with the official GPTBot user agent, one with a standard Chrome browser user agent. Among the 592 sites that disallow GPTBot in robots.txt, 234 (39.5%) still served the page to the crawler.
The opposite happens too. 115 sites in July and 178 in September block GPTBot without saying so in robots.txt: the browser gets the page, the crawler doesn't. Among publishers the gap is wide: in September the browser got the page 83.1% of the time, GPTBot 47.2%. That is a distance of almost 36 points, up from 29.6 in July.
For a brand that wants to appear in AI answers the lesson is simple: robots.txt is a statement of intent. The server does whatever the CDN, the firewall and the security plugins tell it to, and those were often configured by someone else, at another time. If you only read the file, you may believe you are open while you are closed, or the other way round.
The CDN decides more than robots.txt does
How a site treats crawlers depends mostly on who serves its pages. In the 2,096-domain subset, HasData measured how the main CDNs respond to a request carrying the GPTBot user agent:
| CDN | Page served | Hard block (403 and similar) | Challenge | Rate limit |
|---|---|---|---|---|
| CloudFront | 78.5% | 14.4% | 0.0% | 0.0% |
| Cloudflare | 51.3% | 16.9% | 24.7% | 0.5% |
| Akamai | 39.1% | 34.5% | 0.0% | 19.5% |
| Fastly | 27.3% | 61.5% | 0.0% | 0.6% |
"Challenges" are the verification pages (JavaScript or CAPTCHA) that a crawler cannot pass. A site behind Cloudflare shows one to a crawler in four, often without the owner ever deciding it: a high security level is enough. Two caveats on the data: HasData's requests carry the GPTBot user agent but do not come from OpenAI's official IP ranges, so a CDN that verifies bot identity may treat them with more suspicion than the real crawler; and the test targets GPTBot, the training crawler, not OAI-SearchBot, the one that feeds ChatGPT search. We explain the difference between the two in our guide to AI crawlers.
Cloudflare: 118 sites lost their rules without noticing
The most useful finding for Cloudflare users concerns the September 15, 2026 switch to the new settings. Among sites behind Cloudflare, the share with a GPTBot ban in robots.txt fell from 17.1% to 9.9% in two months; among other sites it stayed flat (18.7% to 18.6%). HasData searched the copies archived by Common Crawl and found 118 Cloudflare sites that in August carried the CDN's managed block, with three elements: a Content-Signal: ai-train=no line, a rights reservation under Article 4 of the EU copyright directive, and a GPTBot ban. On September 16 those lines were gone from all 118.
The explanation is the new mechanism. On August 21, 2026 Cloudflare introduced Bot Preference Sync, which generates robots.txt from the dashboard settings for three bot categories: Search, Agent and Training. The block sits between the markers # BEGIN Cloudflare Bot Preference Sync and # END. Customers on the old "Managed robots.txt" are asked to review and confirm their preferences before the transition. Meanwhile, in the 6,451 robots.txt files HasData collected in September, the new block did not yet appear in any of them.
Be careful not to read the figure backwards, though: of the 21 affected sites that were also part of the live test, 16 still blocked or challenged GPTBot at server level. They lost the declaration, not the block. The risk for you cuts both ways: if you want to keep training out, the rules written by the CDN may no longer be there; if you want to be cited, a network-level block may still be active even with a clean robots.txt.
Blocking doesn't remove you from AI answers (and opening up isn't enough)
A ban in robots.txt does not remove a site from Google's answers. In HasData's test of Google AI Mode, ten questions produced 82 cited domains; of the 52 in the sample, 27 (51.9%) disallow at least one AI crawler, against 15% of the whole sample. The reason is technical: Google-Extended governs use in Gemini, not AI Overviews and AI Mode, which draw on the regular search index. Blocking GPTBot, likewise, closes the door to OpenAI's model training but not to ChatGPT search, which uses OAI-SearchBot.
Being reachable is becoming even more important for the discovery of new pages. On October 15, 2026 Cloudflare and OpenAI described a pilot, started in July on a slice of Cloudflare's network (about 1%), in which network signals help OpenAI's search crawler find new pages and tell when a page has changed, while discarding pages that don't exist, are duplicates or redirect elsewhere. The pilot brought "hundreds of millions of new web pages" into the index. One line in the post matters for anyone running a site: "A signal about a page doesn't grant permission to access it." The signal gets the page found; if your site blocks the crawler, it stays out all the same.
What to check on your site this week
You don't need special tools; you need to look at both the declaration and the actual behaviour.
- Read the live robots.txt, not the one in your repository: look for the lines for
GPTBot,OAI-SearchBot,ClaudeBot,PerplexityBot,Google-Extendedand the Cloudflare markers. If you relied on managed rules, check that they are still there and still say what you want. - Send two requests from the same IP: one with a browser user agent, one with OAI-SearchBot's or GPTBot's. Compare status code and content. A 403, a 402, a verification page or an empty response for the crawler is a block, even if robots.txt says nothing.
- Open the CDN dashboard. On Cloudflare, under AI Crawl Control, check the three settings Search, Agent and Training and the state of Bot Preference Sync. If you want to be cited but not used for training, the combination is Search open and Training disallowed, with the caveat that mixed-use crawlers (search and training together) may end up blocked.
- Look at a week of logs: if OAI-SearchBot or Googlebot never show up, the problem sits before robots.txt.
- Repeat the check every time you change CDN, security plugin or hosting: that is where rules get lost, as the 118 Cloudflare sites show.
An example: a furniture e-commerce site in Manchester has a clean robots.txt, but the CDN's security level shows a challenge to every visitor without JavaScript. Google is fine, because Googlebot is verified, while AI crawlers get the verification page and the brand never appears among ChatGPT's sources. Geosnap's technical analysis also checks robots.txt and page reachability, but the two-user-agent test is something you can do yourself in five minutes.
Sources
- The AI Crawler Block Index (HasData, 2026-10-07)
- Cloudflare drops AI opt-outs from 118 sites' robots.txt, HasData finds (PPC Land, 2026-10-08)
- Say it once: introducing Bot Preference Sync (Cloudflare, 2026-08-21)
- Cloudflare and OpenAI: making AI discovery better for everyone (Cloudflare, 2026-10-15)
Frequently asked questions
If I block GPTBot in robots.txt, do I disappear from ChatGPT?
No. GPTBot is the crawler OpenAI uses for model training; ChatGPT search uses OAI-SearchBot, which has separate rules. You can disallow the first and leave the second open. Check your CDN and firewall too: in HasData's data, 39.5% of sites that disallow GPTBot actually serve it anyway, and 178 sites block it without declaring it.
Why did my robots.txt on Cloudflare change by itself?
Since August 2026 Cloudflare generates AI bot rules with Bot Preference Sync, based on the Search, Agent and Training settings in the dashboard, and on September 15 it changed the default settings. HasData found 118 sites where the old managed block (ai-train=no, a rights reservation and a GPTBot ban) disappeared during the transition. Customers who used Managed robots.txt need to review and confirm their preferences in the dashboard.
Can a site that blocks AI crawlers still be cited by Google AI Mode?
Yes. In HasData's test, 51.9% of the domains cited by AI Mode disallow at least one AI crawler in robots.txt, against 15% of the sample. AI Overviews and AI Mode use Google's regular index; Google-Extended only governs use in Gemini and Vertex AI. A ban in robots.txt therefore does not remove you from Google's answers.
How do I check whether my CDN blocks AI crawlers?
Send two requests for the same page from the same IP address: one with a browser user agent and one with OAI-SearchBot's or GPTBot's. If the crawler gets a 403, a 402, a verification page or different content, there is a network-level block. Then compare with the CDN dashboard settings and with your server logs.