Should I let AI crawlers read my website?

AI crawlers are not one thing, and one company runs several of them

“AI crawler” covers at least three different jobs, and the major companies run a separate bot for each. Training crawlers collect pages to build future models. Search crawlers build the index that sits behind an assistant's answers. User-triggered fetchers visit a single page because a person asked the assistant to look at it.

Each bot announces itself with a user agent, the name it gives when it asks your server for a page. Those names are what robots.txt rules match on. OpenAI documents four: GPTBot for training, OAI-SearchBot for ChatGPT's search features, ChatGPT-User for user actions, and OAI-AdsBot for pages submitted as adverts. Perplexity documents two, PerplexityBot and Perplexity-User. OpenAI states plainly that each setting is independent of the others.

Say a dental practice in Exeter blocked GPTBot in 2023, on the reasonable view that its treatment pages should not train someone else's model. That block did what it was meant to do. It did nothing at all to ChatGPT's search answers, because those run through OAI-SearchBot, a bot that did not exist when the rule was written and is not covered by it. The practice is out of the training data and fully available to the live search index at the same time, which is a coherent position, though probably not one anybody chose.

What blocking each control does

The controls named most often do unrelated things, and only one of them takes a business out of an assistant's answers.

Google-Extended is the one most often misread, partly because it is not a crawler. Google's documentation describes it as a standalone product token, a name that exists only inside robots.txt as a control. No bot sends it as a user agent, and crawling continues with the usual Google agents.

A builders' merchant near Plymouth might block Google-Extended expecting the AI summary above its listing to stop appearing. Nothing changes on the results page. Google's guidance on generative AI features, published 15 May 2026 and updated 10 July 2026, states that those features run on the existing Search index and ranking systems, and that a page must be indexed and eligible for an ordinary snippet before it can appear in an AI answer at all. What the merchant has changed is whether its stock and delivery information helps train Gemini and ground answers in the Gemini apps. That may be exactly what it wanted, but it is not what it thought it was buying.

The decision may already have been made for you

Most owners have never opened their robots.txt file, but something has usually written to it, or to the layer above it. Crawler policy is now a default set by hosting providers, content delivery networks and security services, and those defaults move.

Cloudflare, which sits in front of more than 20% of web domains by its own account, launched one-click AI bot blocking in July 2025. On 1 July 2026 it replaced that single switch with three separate categories: Search, Agent and Training. It also offers a managed robots.txt that prepends a Content-Signal line stating that crawling for search is acceptable and crawling for training is not. An owner reading their own file will find wording nobody at the business wrote.

The part worth diarising is 15 September 2026. From that date, for newly onboarded domains and pages that display adverts, Cloudflare will block Training and Agent crawlers by default while leaving Search allowed. Multi-purpose crawlers will be judged on all of their behaviours at once, and Cloudflare names Googlebot, Applebot and Bingbot as examples that will be blocked for anyone who has chosen to block Training, including through the older switch.

Consider a hotel near Salcombe whose developer ticked “Block AI Bots” in 2025 and has since moved on. Nobody at the hotel has looked at that dashboard since. The setting has not changed, but the classification beneath it has, and the same tick now reaches crawlers it did not reach before. A network-level block is also stronger than a robots.txt rule: Cloudflare's own post distinguishes between signals in robots.txt, which express a preference, and blocking, which stops the request. The hotel's robots.txt could be perfectly permissive and still be irrelevant.

How to check what your site is telling AI crawlers

Start with the file itself. Open yourdomain.com/robots.txt in a browser, since it is a public text file on every site, and look for the specific tokens: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended. A token that appears nowhere is allowed. A token under a Disallow rule is blocked, and the block applies only to the paths listed after it.

Then check the layer above, because that overrides the file. If the site sits behind a content delivery network or a security service, its bot settings decide whether the request reaches your server in the first place. Look for anything labelled AI bots, bot fight, or managed rules, and read what each category now covers rather than trusting a label chosen a year ago.

Server logs settle the question that dashboards cannot. Search the logs for the user agent strings and see which bots have arrived, how often, and what status codes they received. Repeated 429 or challenge responses mean the crawler is being turned away by rate limits rather than by policy. Because a user agent header can be typed by anyone, verify the requests against the published address lists: OpenAI publishes ranges at openai.com/searchbot.json and openai.com/gptbot.json, and Perplexity at perplexity.com/perplexitybot.json.

A law firm in Truro doing this properly would finish with three things: a robots.txt it can explain line by line, a security dashboard whose settings it chose deliberately, and log evidence of which assistants have read which pages. Allow time for changes to take effect, since both OpenAI and Perplexity say it can take around 24 hours for a robots.txt update to register. Google, meanwhile, began testing a Search Console toggle on 3 June 2026 that lets owners decide whether their site appears in and helps ground its generative Search features, alongside new impression data for those features. It is being rolled out to a subset of UK website owners first, so it is worth looking for.

What a block does not protect

Blocking is a publishing choice about who may quote you. It is a weak tool for anything else.

The file is public, and it is a request rather than a lock. Anyone can read yourdomain.com/robots.txt, which means a list of disallowed paths is also a list of the pages you would rather not draw attention to. Compliant crawlers honour it. Nothing enforces it.

Blocking also works forwards only. A rule added today does not remove content from models that were trained last year. Partial visibility often survives too: Perplexity states that a page blocked in robots.txt may still be indexed by domain, headline and a brief factual summary, and OpenAI states that a site opted out of OAI-SearchBot can still appear as a navigational link. User-triggered fetchers are a separate case again. Perplexity says Perplexity-User generally ignores robots.txt because a person requested the fetch, and OpenAI says the same rules may not apply to ChatGPT-User.

If material genuinely must not be read by anyone, the controls are authentication or removal, not a crawler token. Removal has a trap in it: a noindex tag can only work if the crawler is allowed to fetch the page and read the tag, so blocking the crawler and adding noindex defeat each other.

Being reachable is not the same as being described correctly

An open crawler policy makes a business eligible to be named. It does not make the naming accurate.

When First Mention tested how AI assistants described its own domain, three of five models resolved firstmention.co.uk to firstmention.co, an unrelated home-services SEO firm in the United States, and one of them attributed that company's content to First Mention. No crawler was blocked. Every page was reachable. The models simply could not tell two similarly named businesses apart, and confidently described the wrong one.

That is entity resolution: whether an assistant can reliably identify your business as distinct from others and describe it correctly. The work behind it is unglamorous and specific. It means naming the business identically everywhere it appears, generating organisation markup from the text visible on the page, verifying the external profiles that models cross-reference, and stating location and country without ambiguity. A hotel called The Ship, a dental practice sharing a surname with three others in the county, and any business whose domain resembles a larger one elsewhere all carry this risk. Nothing in a conventional ranking report will show it, because the pages can rank perfectly well while the description attached to them is somebody else's.

Common questions

More answers are on the First Mention FAQ hub.

← Back to all guides