Which AI Crawlers Should You Allow? GPTBot, ClaudeBot and the Rest
If you want your business to turn up in ChatGPT, Claude and Perplexity answers, allow their search and user-triggered bots: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. The training crawlers (GPTBot, ClaudeBot, CCBot, plus the Google-Extended and Applebot-Extended tokens) are a separate decision, and OpenAI, Google and Apple each say blocking their training bot does not affect whether you appear in their search features. Then check that your CDN or firewall agrees with your robots.txt, because a block at that layer wins whatever the file says.
Everything below comes from each company’s own crawler documentation, read on 29 October 2026. These pages change often, so the links are worth bookmarking.
AI crawlers do three different jobs
Most lists of “AI bots” lump everything together. The useful split is by what the bot does with your page, because that decides what blocking it costs you. The same split exists among AI companies as it does among search engines: some bots gather material, some build an index, and some fetch a page for one person.
- Training crawlers collect pages that may be used to train future models. Blocking one keeps your content out of that training set.
- Search crawlers build the index an assistant searches when it answers a question. Blocking one takes you out of those answers.
- User-triggered fetchers load a single page because a person asked the assistant to look at it. OpenAI and Perplexity say robots.txt may not apply to theirs, since a human started the request. Anthropic says its bots, Claude-User included, honour robots.txt.
| Bot (robots.txt name) | Company | Job | What blocking it does |
|---|---|---|---|
| GPTBot | OpenAI | Training | Signals your content should not be used to train OpenAI’s foundation models |
| OAI-SearchBot | OpenAI | Search | Your pages are not shown in ChatGPT search answers, though they can still appear as navigational links |
| ChatGPT-User | OpenAI | User-triggered | OpenAI says robots.txt rules may not apply, because a user started the visit |
| ClaudeBot | Anthropic | Training | Signals your future content should be excluded from Anthropic’s training data |
| Claude-SearchBot | Anthropic | Search | Your content is not indexed for Claude’s search, which Anthropic says may reduce your visibility |
| Claude-User | Anthropic | User-triggered | Claude cannot fetch your pages when someone asks about them |
| PerplexityBot | Perplexity | Search | You are not surfaced in Perplexity’s results (Perplexity says this bot is not used for model training) |
| Perplexity-User | Perplexity | User-triggered | Perplexity says this fetcher generally ignores robots.txt |
| Google-Extended | Token for Gemini training and grounding | No effect on Google Search, including AI Overviews | |
| Applebot-Extended | Apple | Token for Apple model training | Applebot still crawls you for Spotlight, Siri and Safari |
| CCBot | Common Crawl | Builds a free public copy of the web | CCBot stops fetching your pages for future crawls |
Sources: OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl.
OpenAI runs one more, OAI-AdsBot, which only visits landing pages submitted as ChatGPT ads. If you do not advertise there, you will not see it.
Other bots you will see in your logs
The table above covers the bots most people ask about. A few others turn up often enough to recognise.
| Bot | Company | What it does |
|---|---|---|
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | |
| Bingbot | Microsoft | Crawls for Bing search |
| Applebot | Apple | Crawls for Spotlight, Siri and Safari search; Applebot-Extended decides training use |
| Amazonbot | Amazon | Improves Amazon products and may be used to train Amazon AI models |
| Amzn-SearchBot | Amazon | Makes content eligible for Amazon search experiences such as Alexa |
| Amzn-User | Amazon | Fetches live pages to answer an Alexa question for a user |
Amazon, like OpenAI, says each of its settings is independent and that changes can take around 24 hours to take effect.
Which AI crawlers should you allow?
It depends on what you want from AI assistants, and for most businesses that is the same thing SEO is for: to be found by people looking for what they sell. Treat the AI search bots the way you already treat the search engines’ crawlers.
| Your goal | Allow | Disallow |
|---|---|---|
| Be cited and recommended in AI answers | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User | Nothing required |
| Be cited, but keep content out of model training | The six search and user bots above | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended |
| Keep content away from AI entirely | Nothing | All of them, plus a CDN rule, since user-triggered fetchers may ignore robots.txt |
Our own view, for a service business that wants enquiries, is to allow the lot. The training opt-out is a reasonable choice if it matters to you, and it costs little because the vendors keep it separate from search. It will not help your rankings or your citations either way. The row that does real damage is the third one, because a site can end up there without anyone choosing it.
Is Google-Extended a crawler?
No. Google says Google-Extended “doesn’t have a separate HTTP request user agent string”. Pages are fetched by Google’s normal crawlers, and the Google-Extended token in robots.txt only controls whether that content can be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI.
Disallowing Google-Extended does not keep you out of AI Overviews. Google’s guidance on AI features says robots.txt rules for Googlebot are the control for Search, and points to nosnippet, data-nosnippet, max-snippet and noindex if you want to limit what is shown.
The Search Console setting for AI Overviews and AI Mode
Google now has a separate switch for its own AI features. The Search generative AI control sits in Search Console under Settings > Search generative AI, and Google says it had rolled out to all websites by 31 August 2026. It covers AI Overviews, AI Mode and generative AI features in Google Discover.
The default is to include your site. Choosing to exclude it removes your links and content from those features, and Google says you would then get no traffic or impressions from them. It is worth a look as part of any AI visibility check, if only to confirm nobody has switched it off. Apple works the same way: Applebot-Extended “does not crawl webpages” and only decides how data Applebot has already collected may be used.
A robots.txt that allows AI search and opts out of training
If you want the middle row of that table, this is the shape of it. Each group lists several user agents and then the rules that apply to all of them.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /Under the robots.txt standard, RFC 9309, a crawler that finds a group naming it follows that group and only falls back to User-agent: * when nothing names it. Google’s robots.txt documentation describes the same rule. So if your wildcard group blocks an admin area, repeat that Disallow line inside any named group above, or the named bots will be allowed into it.
Robots.txt also works per host. Anthropic’s documentation reminds you to add the rules to every subdomain you want covered, and a blog on a separate subdomain has its own file.
Can you block AI crawlers from part of a site?
Yes. Robots.txt rules work on paths, so you can keep a members’ area or a set of paid downloads away from training crawlers while leaving the rest open:
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /members/
Disallow: /downloads/Anthropic and Common Crawl both say they also honour Crawl-delay, which helps if the worry is server load.
Does llms.txt control AI crawlers?
No. llms.txt is a proposed file that gives language models a clean summary of a site and links to its most useful pages. Who gets in is still decided by robots.txt and your CDN rules. Whether it helps visibility is a separate question, and our guide to whether llms.txt works sets out the evidence.
Your CDN can override robots.txt
Robots.txt asks a crawler to stay out, and a well-behaved crawler checks it before fetching anything. Cloudflare puts it plainly in its managed robots.txt documentation: “robots.txt compliance is voluntary.”
A CDN or firewall rule works differently. It refuses the request at the edge, usually with a 403, before your website ever sees it. Your robots.txt can say Allow: / to GPTBot while your CDN quietly turns GPTBot away, and nothing on the website itself will tell you.
What Cloudflare’s AI settings do
If your site runs through Cloudflare, these are the settings to check. AI Crawl Control (formerly AI Audit) is available on every Cloudflare plan. It shows which AI services are visiting, lets you allow or block individual crawlers, and tracks which of them follow your robots.txt.
On 1 July 2026 Cloudflare announced AI bot policies for every plan, Free included, to replace the single “Block AI bots” switch. They have three categories, Search, Agent and Training, and each can be set to block everywhere, block on pages with ads, or allow. Cloudflare said that from 15 September 2026 new domains would default to blocking Training and Agent bots on pages that display ads, with Search left allowed, and its documentation marks the old switch as deprecating on that date. When we checked on 29 October 2026, neither the documentation nor the changelog had confirmed the change had happened. Either way, the place to look now is Security Settings > Configure AI bot policies. One catch for the “allow search, block training” approach: Cloudflare says crawlers that do both jobs are blocked by every setting that blocks AI training. It can also add its own lines to the top of your robots.txt through a feature called Bot Preference Sync. If your robots.txt has lines you never wrote, that is the likely cause.
Anthropic adds a warning worth taking seriously if you want to opt out cleanly: blocking its IP addresses “may not work correctly or persistently guarantee an opt-out”, because it stops Anthropic reading your robots.txt in the first place.
How to check AI crawlers can get in
Work through these in order. The first two take a few minutes.
- Read your live robots.txt. Open
yourdomain.co.uk/robots.txtin a browser, not the copy in your theme or CMS. Look for groups naming the bots above, and for any block of text you did not write. - Check your CDN. In Cloudflare, look at AI Crawl Control and the AI bot policies under Security settings. Other CDNs and security plugins have their own bot rules, so check whichever you use.
- Look at your server or CDN logs. Search for the user agent names. Then confirm the IP addresses belong to the company, using the lists each one publishes: OpenAI’s GPTBot ranges, Anthropic’s, Perplexity’s. Apple and Common Crawl both support reverse DNS checks.
- Check Search Console. Under Settings, confirm the Search generative AI control is set to include your site, and open the robots.txt report to see the version Google last fetched.
- Run a quick request with a bot’s user agent, as a smoke test only.
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://www.yourdomain.co.uk/Why Google Analytics does not show AI crawlers
Google Analytics automatically excludes traffic from known bots and spiders, and you cannot switch that off. It also records visits through a JavaScript tag, and many crawlers never run JavaScript: Common Crawl, for one, says CCBot does not. An empty Analytics report tells you nothing about AI crawlers, so use your server or CDN logs instead. People who click through from an assistant are a different matter, and you can track AI traffic in GA4 separately.
Why a spoofed curl test is only indicative
That command sends GPTBot’s name from your own computer’s IP address. Real GPTBot requests come from OpenAI’s published ranges, and good firewalls check the address as well as the name. Anyone can claim to be a bot, which is why Google publishes a guide to verifying its crawlers.
So read the result with care. A 403 might mean your firewall blocks GPTBot, or it might mean the firewall spotted a fake GPTBot and did its job. A 200 is better news, though a rule that only applies to verified search bots would still let your fake through while stopping the real one. Treat the curl result as a prompt to look at the CDN settings and logs, which are the evidence that counts.
Why this matters for being recommended
An assistant can only cite a page it was allowed to read. The rest of the work, being clear about what you do, where and for whom, and being mentioned by other sites, is covered in how to get your business recommended by ChatGPT. If you are newer to the subject, our explainer on what a web crawler is covers how crawling, indexing and ranking fit together, and why a website does not show up on Google runs through the same blockages from the search side.
What to do next
Check your robots.txt and your CDN settings this week, and make sure they say the same thing. If they disagree, the CDN is the one that counts.
If you would rather someone else checked, AI search optimisation starts with exactly this audit: who can read the site, what they find when they do, and whether assistants can tell what you sell. Or send us your address for a free SEO audit, which tells you where the site ranks and what is technically holding it back.