Last updated: 10 November 2026
Robots.txt rules for AI crawlers are text directives in a site's robots.txt file that tell bots like GPTBot, ClaudeBot and Google-Extended whether they may access pages. They are a voluntary signal, not a legal barrier: Cloudflare found only about 14% of top-10,000 domains with a robots.txt file had any AI-specific directive at all, out of 3,816 domains checked (Cloudflare Blog / Cloudflare Radar, 2025).
Key Takeaways
- GPTBot is the most blocked AI crawler on the web, disallowed by 312 of the domains Cloudflare analysed — 250 fully and 62 partially (Cloudflare Blog / Cloudflare Radar, 2025).
- GPTBot is simultaneously the most explicitly allowed bot, with 61 domains granting it access (18 fully, 43 partially) — showing the industry is split, not united, on AI crawling (Cloudflare Blog / Cloudflare Radar, 2025).
- Reputable news publishers block AI crawlers far more than low-quality sites do: 60.0% of reputable news sites disallow at least one AI agent, versus just 9.1% of misinformation sites (arXiv, 2025).
- AI crawler traffic is growing fast: between May 2024 and May 2025, overall crawler traffic rose 18%, with GPTBot requests up 305% (Cloudflare Blog, 2025).
- Robots.txt is advisory only — Aether AI advises businesses to note that OpenAI's own documentation states "because these actions are initiated by a user, robots.txt rules may not apply" when a human triggers a live fetch inside ChatGPT.
What is robots.txt and how does it work?
Robots.txt is a plain-text file, placed at the root of a domain (for example yoursite.co.uk/robots.txt), that tells automated crawlers which parts of a site they may or may not request. It implements the Robots Exclusion Protocol, formalised in 2022 as RFC 9309 (Wikipedia: Robots.txt), and it works purely on trust: a well-behaved bot reads the file before crawling and follows its instructions, but nothing forces it to.
Each rule block names a User-agent (the bot it applies to) followed by Disallow or Allow paths. A wildcard User-agent: * applies to any bot not named elsewhere in the file. Search engines such as Google and Bing have honoured robots.txt for over two decades, which is why it became the default tool businesses reach for now that AI crawlers — bots that fetch web content to train large language models or power live AI answers — have multiplied.
The technical limitation is simple: robots.txt cannot authenticate a crawler's identity. A bot can claim to be GPTBot in its user-agent string while ignoring the rule entirely, and robots.txt has no mechanism to stop it. That's why server logs and IP verification matter alongside the file itself, a point we return to below.
Which AI crawlers exist and what user-agent names do they use?
AI crawlers are bots operated by AI companies to collect training data or serve live answers, and each one identifies itself with a distinct User-agent token in robots.txt. Knowing the names is the first practical step, because a rule written for the wrong token simply does nothing.
The most commonly referenced tokens, per vendor documentation:
| Crawler | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Crawls content for model training |
| ChatGPT-User | OpenAI | Fetches pages live when a ChatGPT user asks it to |
| OAI-SearchBot | OpenAI | Powers ChatGPT's search/answer features |
| Google-Extended | Controls use of content for Gemini and AI features, separate from standard Search indexing | |
| ClaudeBot | Anthropic | Crawls content to train Claude models |
| CCBot | Common Crawl | Builds the open dataset many LLMs train on |
| Applebot-Extended | Apple | Controls use of content for Apple Intelligence training |
OpenAI's own documentation lists GPTBot, OAI-SearchBot and ChatGPT-User together and explains how each should be addressed in robots.txt (Overview of OpenAI Crawlers). Google is the trickiest case: its documentation states plainly that "Google-Extended doesn't have a separate HTTP request user agent string" — it's a control token layered over Google's existing crawl infrastructure, not a separate bot you can spot in logs (Google Search Central, cited via Fastly blog). Google adds that "crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity" — meaning Googlebot itself still fetches the page, while Google-Extended governs what happens to that content afterwards.
How do I write robots.txt rules to block or allow specific AI crawlers?
Robots.txt rules for AI crawlers follow the same basic syntax as any other bot rule: name the User-agent, then add Disallow: / to block everything or Disallow: /path/ to block a section. Allowing a bot means either omitting it entirely or adding an explicit Allow: /.
A block-everything rule for three major AI training bots looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
A more surgical approach — common among UK publishers who want AI visibility but not free training data — separates search-facing bots from training bots:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Google-Extended
Disallow: /
This blocks OpenAI's training crawler while still allowing its search-answer bot to cite the page, and opts out of Google's AI training use without touching standard Googlebot indexing. Google states that blocking Google-Extended has no effect on a site's appearance in standard Google Search results — it only governs use by Gemini and AI Overviews training (per Google's common crawlers documentation). Always place AI-specific blocks above or separate from a general User-agent: * block, since crawlers match the most specific applicable rule.
Is blocking AI crawlers via robots.txt legally binding?
Robots.txt is not legally binding in the UK or internationally — it is a voluntary, self-enforced convention with no statute compelling compliance. Nothing in UK law, including the Computer Misuse Act 1990, creates a direct offence for a crawler that ignores a Disallow line, though persistent unauthorised access to protected systems can raise separate legal questions depending on the circumstances.
Compliance rests entirely on the crawler operator's goodwill and published policy. OpenAI, Anthropic and Google all state publicly that their training crawlers respect robots.txt, but OpenAI's own documentation carves out an important exception for user-triggered actions: "because these actions are initiated by a user, robots.txt rules may not apply" — meaning a person asking ChatGPT to fetch a specific URL can sometimes retrieve it even where GPTBot is blocked, because that fetch isn't classed as autonomous crawling.
This gap matters commercially as much as legally. A business relying solely on robots.txt to keep content out of AI training has no contractual or regulatory backstop if a crawler misbehaves or a user-initiated fetch slips through — it has a request, honoured at the operator's discretion.
"Blocking GPTBot to protect your content is like delisting from Google to protect your brochure. Defensible for a paywalled publisher; self-harm for anyone who sells something. If assistants can't read you, they recommend whoever they can read." — Lauren Dawkins, Head of Content, Aether AI
What are the risks of blocking AI crawlers for visibility in AI search?
Blocking AI crawlers can remove a business from the answers that ChatGPT, Perplexity, Gemini and Copilot give to prospective customers, because a bot that cannot read a page cannot cite it. This is the core trade-off UK marketing and SEO leads now have to weigh, and the data shows it's already a majority practice among cautious operators.
The arXiv study found over 50% of reputable sites studied disallow GPTBot outright, and 40-50% restrict other major crawlers including ClaudeBot, ChatGPT-User and Google-Extended (arXiv, 2025). Separately, an analysis cited by Otterly.ai found at least 15% of the top 100 websites and 7% of the top 1,000 websites were blocking GPTBot (Otterly.ai, 2023), while a Medium/LLMO Notes analysis put the figure around 28% of websites studied disallowing GPTBot, with roughly 25% of top-visited sites doing so within a two-month tracking window (Medium / LLMO Notes, 2023).
For a paywalled newspaper, blocking makes sense: the content is the product. For a business that sells a service — security, software, facilities, branding — blocking an AI crawler doesn't protect revenue, it removes the business from consideration entirely when a prospect asks an AI assistant for a recommendation. This measurement illustrates how rare brand citations already are: across 700 live AI engine scans of UK brands' target keywords run in July–August 2026, the tracked brands were named in only 23 answers (3.3%), with the longest-established brand named in 6.3% of its 270 scans (AI-visibility baseline 2026). Blocking crawlers outright only widens that gap further.
What's the difference between robots.txt and other AI-blocking methods?
Robots.txt is a crawl-time request honoured voluntarily by the bot, whereas meta tags, HTTP headers and firewall-level blocking operate at different points and with different levels of enforcement. Understanding the distinction matters because businesses often assume one method covers all three.
A noai or noimageai meta tag sits inside a page's HTML and tells a compliant crawler not to use that specific page's content for training, even if the crawler was allowed to fetch it. Cloudflare-style AI bot blocking works at the network layer: it inspects incoming requests, verifies the crawler's declared IP ranges, and can reject or challenge traffic that spoofs a user-agent string — something robots.txt cannot do, since it simply trusts whatever user-agent a bot presents.
| Method | Enforcement level | Stops spoofed bots? | Covers live user-triggered fetches? |
|---|---|---|---|
| Robots.txt | Crawl-time request, voluntary | No | Not always |
| Meta tags (e.g. noai) | Page-level, voluntary | No | Partially |
| WAF/firewall (e.g. Cloudflare) | Network-level, enforced | Yes, via IP verification | Yes |
| Content-Signal directives | Emerging, voluntary | No | Not yet standardised |
For businesses serious about controlling AI access, robots.txt is the first layer, not the last. A content delivery network or web application firewall that verifies published IP ranges is the only method that can actually stop a crawler ignoring the rules, rather than merely asking it to comply.
Who should be responsible for setting robots.txt rules in a UK business?
Robots.txt decisions sit at the intersection of IT, legal and marketing, and should not be left to whichever department happens to touch the server first. IT typically holds write access to the file and understands the technical syntax; legal should weigh data protection and content-licensing exposure; marketing and SEO leads understand the commercial cost of losing AI visibility.
In practice, a change this consequential needs sign-off from whoever owns organic and AI-search performance, because a single misplaced Disallow: / can silently remove a site from both Google Search and every AI answer engine overnight. The Information Commissioner's Office (ICO) has general guidance on web scraping and data protection that's relevant background for any UK business weighing crawler access, even though robots.txt itself sits outside formal data protection law. A quarterly review, owned jointly by IT and marketing, with legal consulted whenever a new AI crawler policy is published, is a reasonable minimum cadence for most UK organisations.
What common mistakes do site owners make configuring robots.txt?
The most frequent mistake is blocking User-agent: * without realising it silently includes every AI crawler, search engine and accessibility tool, not just the intended target. A second common error is targeting the wrong token entirely — writing a rule for "ChatGPT" when the actual crawler identifies as GPTBot, OAI-SearchBot or ChatGPT-User, each with different behaviour.
Businesses also frequently forget that Google-Extended has no separate crawl user-agent, so attempting to "see" it blocked in server logs is pointless — Google's own documentation confirms the control token and the crawl request are separate things. Another mistake is assuming a robots.txt rule retroactively removes content already used for training; it only governs future crawling, not past collections. Finally, many sites never revisit the file after initial setup, missing new crawlers like Applebot-Extended entirely — Cloudflare's data shows that even among the 14% of domains with any AI-specific directive, coverage of newer bots lags well behind coverage of GPTBot (Cloudflare Blog / Cloudflare Radar, 2025).
How do I check whether my robots.txt is correctly configured?
Checking a robots.txt file means fetching yoursite.co.uk/robots.txt directly in a browser and confirming every intended User-agent block is present, correctly spelled, and ordered so specific rules sit above wildcard rules. Google Search Console includes a robots.txt tester under the Settings menu, useful for confirming Googlebot's own interpretation of the file.
For AI crawlers specifically, cross-reference the live file against current vendor documentation — OpenAI's crawler overview and Google's common crawlers page both list the exact tokens in use, and both are updated periodically as new products launch. Server-log analysis is the only way to confirm a blocked bot has actually stopped requesting pages, since robots.txt compliance is never guaranteed. This is also where Technical GEO platforms add value: Aether AI's citation tracking monitors whether a brand is actually appearing in answers across ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot — a direct way to see whether a robots.txt configuration is helping or hurting visibility, rather than guessing from the file alone.
Your robots.txt for AI crawlers checklist
- Fetch
yoursite.co.uk/robots.txtand confirm it loads without error - List every AI crawler token currently referenced (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, OAI-SearchBot, ChatGPT-User)
- Decide separately for training bots versus search/answer bots — they serve different purposes
- Place specific
User-agentblocks above any wildcardUser-agent: *rule - Cross-check tokens against current OpenAI and Google documentation
- Pair robots.txt with IP-verified firewall rules if spoofed-bot traffic is a concern
- Review the file at least quarterly, and whenever a major AI vendor announces a new crawler
- Track actual AI-answer citations, not just crawl logs, to measure the real commercial effect
FAQ
What is the correct robots.txt syntax to block AI crawlers like GPTBot?
Add a dedicated block with User-agent: GPTBot followed by Disallow: / to block the whole site, or Disallow: /folder/ to block a section only. Place this above any general wildcard rule so it takes precedence.
Does blocking Google-Extended affect my Google Search rankings?
No. Google-Extended only controls use of content for Gemini and AI features; standard Google Search crawling and ranking use Googlebot separately, and Google's own documentation confirms Google-Extended "doesn't have a separate HTTP request user agent string."
Can AI chatbots like ChatGPT-User bypass robots.txt rules?
Sometimes. OpenAI's documentation notes that user-triggered fetches may not follow robots.txt because "these actions are initiated by a user," meaning a person asking ChatGPT to open a specific page can occasionally retrieve it even where the site blocks GPTBot.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot crawls content to train OpenAI's models, OAI-SearchBot powers ChatGPT's search and answer features, and ChatGPT-User fetches pages live in response to a specific user request. Each has its own User-agent token and can be allowed or blocked independently.
How do I allow AI search crawlers but block AI training crawlers?
Write separate blocks: disallow GPTBot and Google-Extended (training), while explicitly allowing OAI-SearchBot (search/answer). This lets a brand appear in AI-generated answers without granting blanket access for model training.
Does robots.txt guarantee AI companies won't use my content for training?
No. Robots.txt is a voluntary request, not an enforcement mechanism, and it has no technical way to verify a crawler's identity or stop one that ignores the file. Content already collected before a rule was added is also unaffected.
How often should I update my robots.txt file for new AI crawlers?
A quarterly review is a reasonable minimum, with an additional check whenever a major AI vendor announces a new bot. New tokens such as Applebot-Extended have appeared well after the file was first set up on most sites, so static configurations drift out of date quickly.
Securing your AI visibility with Aether AI
Getting robots.txt rules for AI crawlers right is only half the problem — the other half is knowing whether the configuration is actually helping a brand appear where customers are asking. That's the gap Aether AI's citation tracking across ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot is built to close, alongside keyword tracking, competitor tracking and GSC integration in one self-service platform.
This measurement shows how much is at stake: across 700 live AI engine scans of UK brand keywords in July–August 2026, tracked brands appeared in just 23 answers overall (3.3%) (AI-visibility baseline 2026). A blocked crawler or a misconfigured robots.txt file can push that number lower still.
Run a free AI-visibility audit at aether-ai.co.uk/audit to see exactly where your site stands with AI crawlers today, and check current plans at the public pricing page before deciding what to change.