AI agents are visiting your site: block them or not? What Cloudflare says
What separates ChatGPT-User and other AI agents from crawlers, why Cloudflare says they cost websites money, why robots.txt is not enough and what to check today.
In short
AI agents such as ChatGPT-User, Claude-User and Perplexity-User open your pages on behalf of a person who is choosing right now, and OpenAI warns that for them robots.txt "may not apply". On 5 November 2026 Cloudflare published a long post and new per-agent controls, and its CEO said that for some customers they are "a bane": automated traffic overtook human traffic in May and the site pays for requests that bring no visits or sales. For a brand, blocking them all means disappearing exactly when a customer's assistant is comparing prices and availability: keep price lists and product pages readable, protect logins and forms, and check in your logs what agents actually receive.
- A crawler works for an engine, an agent works for a person in that moment: OpenAI separates OAI-SearchBot (search), GPTBot (training) and ChatGPT-User (user requests), and says robots.txt may not apply to the last one.
- According to Cloudflare, automated traffic overtook human traffic in May 2026, accounts for almost 60% of requests, and in nearly every problem case "the Site bears the primary cost".
- In the first half of 2026 TollBit saw about 15% of identified AI fetchers reach disallowed pages on European sites: if you really want to decide, act on the server and CDN, not just in the file.
- Since 15 September Cloudflare separates search, training and agents into three controls, and since 5 November lets you allow an agent across the site, block it only on pages with ads, or exclude it.
- For a brand the rule is: price, product and opening-hours pages readable by agents; gates only on login, forms and cart; check in your logs that ChatGPT-User is not getting 403s or challenge pages.
Written by the Geosnap AI agent, reviewed and approved by Rinald Sefa.
Translated from the Italian original. Read the original
Search and training crawlers are no longer the only bots reading your site. A growing share of requests comes from agents working for one specific person: an assistant comparing three quotes, reading a menu, checking whether a product is in stock. On 5 November 2026 Cloudflare published a long post and new controls dedicated to these agents, and the next day its CEO said that for some customers they are "a bane". Here is what separates them from crawlers, why they cost sites money, why robots.txt is not enough to govern them, and what to do if you want your brand to keep showing up in AI answers.
What is the difference between an AI crawler and an agent working for a user?
A crawler visits your site on behalf of an engine; an agent visits it on behalf of a person, right now. In its 5 November post, Cloudflare defines "Personal AI Agents" as automations that take instructions from a human and pursue a goal by controlling a cloud computer or browser, hosted by an operator such as OpenAI or Meta. Agents running on the user's own device are out of scope.
The distinction is already written into OpenAI's crawler documentation, which separates three user agents:
- OAI-SearchBot indexes pages for ChatGPT search: block it and the site will not appear in answers.
- GPTBot collects content to train models: blocking it does not remove you from search.
- ChatGPT-User opens a page when a user asks for it. OpenAI writes that it "is not used for crawling the web in an automatic fashion", that it does not decide what appears in search, and that because the actions are initiated by a user, "robots.txt rules may not apply".
The same pattern holds for the other engines: next to the search crawlers sit Claude-User and Perplexity-User, which only move when someone asks a question. If you tidied up your robots.txt this year following the AI crawler guide, these agents are the part those rules cover least.
Why does Cloudflare say agents cost websites money?
Because they generate requests the site pays for without getting a visit or a sale in return. On the third-quarter earnings call, reported by The Register on 6 November, Matthew Prince used the restaurant example: an agent choosing where to book opens a thousand websites and pulls down a thousand menus, and "999 of those restaurants have now had a cost that is just been applied to them". Some customers, he added, see agents as "a bane to their business and want to keep them away".
The numbers Cloudflare lines up are these:
- according to Fortune, automated traffic overtook human traffic in May 2026, earlier than expected, and "almost 60% of requests" to websites now come from bots;
- Prince expects automated traffic to reach a thousand times human traffic within five years;
- on the 6 November call he said roughly 5% of the web has turned on Cloudflare's content protection products since 1 July.
The 5 November post lists fourteen open problems, from identity verification to unauthorized card use, and concludes that "in nearly every challenge above, the Site bears the primary cost". One of the cases described is exactly the quote-comparison scenario: the site pays to serve the page "and gets no visit, ad impression, or purchase". Keep in mind, though, that Cloudflare sells the products that solve the problem it describes: the data is useful, but read it with that in mind.
Is robots.txt enough to govern agents?
No. Cloudflare says so plainly: "Robots.txt says where a crawler may go, but isn't fully suited to express what Agents may do." Today an agent infers a site's rules "by trying something and interpreting the Site response", usually by finding out whether it gets blocked. That is why Cloudflare is calling for a machine-readable way to publish which actions are allowed, and has launched a coalition, Trusted Agent Delegation, to bring the problem to the IETF.
The data confirms that the file alone does not stop agents. TollBit's report on the first half of 2026, covered by Search Engine Journal, found that on the European sites analysed about 15% of identified AI page-fetchers reached disallowed URLs; ChatGPT-User, Bytespider and Youbot did so on nearly half of the European sites that had listed them. The same report says only 9% of European sites disallow Claude-User, against 26% in North America, and 13% disallow Perplexity-User, against 26%.
This does not mean agents are breaking the rules: OpenAI openly states that robots.txt may not apply to ChatGPT-User. It means that if you really want to decide who gets in, you have to do it at the server or CDN level, just as with declared blocks that are never enforced. The new shared AI training signal is a different matter: it covers models, not agents.
What does a brand lose by blocking agents?
It loses the moment when a customer's assistant is choosing. When someone asks ChatGPT to "find me a hotel in Bologna with parking under 150 euros", the agent opens the price and room pages and answers with whatever it can read. If your page returns a bot challenge, an error or an empty shell, the answer gets built from your competitors' sites. Cloudflare writes it in its own list: "Agents are never expected to be humans", so a CAPTCHA stops them without the user ever knowing.
The choice is not all or nothing. Since 15 September 2026 Cloudflare has replaced its single "Block AI Bots" switch with three independent controls, search, training and agents, with different defaults for new sites: sites carrying advertising get search on, training disallowed and agents blocked on pages with ads; everyone else gets all three channels open. The same release says fewer than 1% of sites block search crawlers while 17% restrict training: most sites have already understood that search and training deserve different answers. The same logic applies to agents.
| Page type | What the agent does | What makes sense |
|---|---|---|
| Prices, product pages, opening hours, FAQ | Reads and compares to answer the user | Let it through: this is your chance to be chosen |
| Ad-supported editorial pages | Consumes the page without showing the ads | Consider a block there only, as Cloudflare does by default |
| Login, forms, cart, member areas | Acts on the user's behalf, with abuse risks | Targeted controls, not a site-wide block |
Think of a travel agency in Turin: if its departure list is readable, the customer's assistant will suggest it; if it sits behind a challenge, the assistant suggests another agency. The cost of serving that page is real, but the cost of not appearing is higher.
What to check on your site today
- Look at your logs. Search for the user agents ChatGPT-User, Claude-User and Perplexity-User and check which response code they get. A 403 or a challenge page on prices and product pages is a problem to fix right away.
- Check your CDN and firewall. On 5 November Cloudflare announced that Security Analytics breaks out personal assistants as their own traffic category and that its access policies let you, per agent, allow it across the site, block it only on pages with ads, or block it entirely. If you use another provider, look for the equivalent: the point is not to leave the decision to a generic anti-bot rule.
- Separate search, training and agents. Make three different decisions, not one. In its technical analysis Geosnap also checks robots.txt and whether pages are reachable by AI engines, so you can see whether a block exists only on paper or also on the server.
- Put the information in the HTML. An agent reading the page in a few seconds has to find price, availability, hours and terms in the text, not in an image or three clicks away.
- Protect actions, not reads. Cloudflare has opened early access to a feature that shows, on logins and sign-ups, whether the request comes from an agent and which one: use controls like this where there is a risk, not on the pages you want read.
What is coming in the next few months
A way to recognise agents and the person behind them. Cloudflare proposes that agents sign their requests with Web Bot Auth and declare a level of user identity, from none to a unique identifier, so that a site can stop a single abusive account without blocking the agent for everyone. The Trusted Agent Delegation coalition will take these proposals to the IETF, where the AIPREF group is already working on crawler preferences. The 15 September release also says that by early 2027 Cloudflare wants to give sites control over how much of their content ends up in AI summaries.
In the meantime the practical rule is simple: treat agents as customers with little patience. Make what you want to sell readable, put a gate only where you need one, and check in your logs that the gate has not ended up in front of the shop window.
Sources
- Agents aren't humans (but they kinda are) (Cloudflare, 2026-11-05)
- AI agents are a 'bane' for some website operators, says Cloudflare (The Register, 2026-11-06)
- Cloudflare Helps End the Search-or-AI-Training Tradeoff (Cloudflare, 2026-09-15)
- Bots now outnumber human traffic on the web. Cloudflare's CEO wants AI companies to pay for it (Fortune, 2026-09-18)
- Overview of OpenAI Crawlers (OpenAI Developers, 2026-11-08)
- OpenAI Says Robots.txt May Not Apply To ChatGPT's Fetch Bot (Search Engine Journal, 2026-08-14)
Frequently asked questions
If I block ChatGPT-User, do I disappear from ChatGPT search?
No. OpenAI writes that ChatGPT-User is not used to decide what appears in search: that role belongs to OAI-SearchBot. Blocking ChatGPT-User does, however, stop the assistant from opening your pages when a user asks it to, for instance to compare a price or read opening hours. In that moment the answer gets built from the sites it can read.
Why does an AI agent ignore robots.txt?
Because robots.txt was designed for crawlers that scan the web on their own, while an agent acts at a person's request, much like a browser would. OpenAI states that for ChatGPT-User "robots.txt rules may not apply", and Cloudflare says the file is not suited to expressing what agents may do. To really block them you need rules on the server or CDN.
How do I know whether AI agents are visiting my site?
Search your server or CDN logs for the user agents ChatGPT-User, Claude-User and Perplexity-User and look at the response codes. On Cloudflare, since 5 November 2026 Security Analytics breaks out personal assistants as a separate traffic category, with each agent's requests, paths and errors.
Do AI agents really weigh on the hosting costs of a small site?
It depends on how many requests you get and how much each page costs to serve. Cloudflare measures almost 60% of requests coming from bots across its network, but for a site with a few static pages the cost is negligible compared with the value of appearing in an answer. The problem becomes serious on sites with many dynamic pages or advertising, where a targeted block makes sense, not a total one.
Does Cloudflare block AI agents by default?
Only partly. Since 15 September 2026, for new sites that carry advertising, the default keeps search on, disallows training and blocks agents on pages with ads; for all other sites the three channels stay open. If your site is older or managed by an agency, check the settings rather than assuming.