Last updated: 11 November 2026
Should you block AI crawlers? For most UK businesses that want to be found, the answer is no — blocking bots like GPTBot removes you from AI training and answers, not just scraping. 1.53 million websites (9.9% of the web) already block at least one AI crawler, according to Suff Digital, so the decision needs weighing carefully, not defaulting to "block everything."
Key Takeaways
- Suff Digital found that GPTBot is blocked by 8.7% of all websites and 23.1% of the most-visited 1,000 sites, based on 15.5 million HTTP Archive websites (Suff Digital, 2026).
- The Register reports around 5.6 million websites have added GPTBot to their robots.txt disallow list, up from roughly 3.3 million at the start of July 2025 (The Register, 2025).
- BuzzStream's analysis of top UK and US news sites found 79% block AI training bots via robots.txt, but only 14% block all AI bots outright (BuzzStream, 2025).
- Conductor data shows AI referral traffic averages just 1.08% of total website traffic, against 25% for traditional organic search (Conductor, 2025).
- Cloudflare, which powers around 20% of all web pages, switched AI crawler access from opt-out to opt-in by default for new customers from 1 July 2025 (Search Engine Roundtable, 2025).
What Are AI Crawlers and How Do They Differ from Googlebot?
An AI crawler is an automated bot that visits web pages to collect text, images or code — either to train a large language model or to retrieve live content for an AI assistant's answer. This differs from Googlebot, Google's long-standing crawler that indexes pages purely to rank them in traditional search results.
The key distinction is purpose. Googlebot indexes your page so it can appear in Google Search; an AI training crawler like GPTBot ingests your content so OpenAI's models can learn patterns from it, potentially without ever sending a visitor back to you. A third category — retrieval bots such as OAI-SearchBot or PerplexityBot — fetches your page in real time specifically to answer a user's question and cite you, which is closer to how a search engine behaves.
Most major AI labs now operate multiple distinct crawlers for these different jobs. OpenAI's own documentation lists GPTBot (training), ChatGPT-User (user-triggered browsing) and OAI-SearchBot (search retrieval) as separate user-agents, each controllable independently in robots.txt — a plain text file at your domain root that tells bots which parts of your site they may or may not visit. Google runs an equivalent split: Googlebot for Search, and Google-Extended, a separate token that controls whether your content trains Gemini models without affecting your Google Search ranking at all.
What Are the Benefits of Allowing AI Crawlers to Access My Site?
Allowing AI crawlers lets your content be read, summarised and cited by assistants like ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot — the channels where a growing share of research-stage buyers now ask questions instead of typing search terms. If a crawler can't read your page, it generally can't quote your page.
This matters commercially even though the current traffic volumes look small. AI referral traffic accounts for just 1.08% of total website traffic on average, compared with 25% for traditional organic traffic, though 87.4% of that AI referral traffic comes from ChatGPT specifically (Conductor, 2025). That 1.08% is a current snapshot, not a ceiling — assistant usage is rising, and the businesses visible in those answers today are building a head start.
Aether AI's Head of Content puts the trade-off bluntly:
"Blocking GPTBot to protect your content is like delisting from Google to protect your brochure. Defensible for a paywalled publisher; self-harm for anyone who sells something. If assistants can't read you, they recommend whoever they can read." — Lauren Dawkins, Head of Content, Aether AI
For a security firm, a software vendor or a facilities contractor, being cited by name when a prospect asks an AI assistant "who handles manned guarding in Manchester" or "best GRC software for UK SMEs" is a direct pipeline opportunity. Blocking the crawler that would surface you removes that path entirely — you simply won't be in the conversation.
What Are the Risks of Letting AI Crawlers Scrape My Content?
The core risk of allowing AI crawlers is uncompensated reuse: your original text, data or analysis can be ingested into a training set and potentially regurgitated without attribution, context or a link back to your site. For publishers whose entire business model is page views and subscriptions, this is existential rather than marginal.
That's precisely why news publishers behave very differently from the average business site. BuzzStream's analysis of the top 50 US and UK news sites by SimilarWeb traffic found 79% block AI training bots via robots.txt and 71% block AI retrieval bots, yet only 14% block all AI bots while 18% block none at all — even among publishers with the strongest incentive to protect content, blanket blocking is the minority position.
There's also a compliance risk on the AI company's side worth knowing about: robots.txt is a voluntary signal, not an enforceable wall, and some crawlers ignore it anyway. TollBit's Q2 2025 report found 13.26% of AI bot requests ignored robots.txt directives, up from 3.3% in Q4 2024 — alongside a 336% year-on-year increase in sites attempting to block AI crawlers in the first place. Blocking reduces exposure; it doesn't guarantee it.
How Do I Technically Block AI Crawlers Using Robots.txt?
Robots.txt is a plain text file placed at the root of your domain (yoursite.co.uk/robots.txt) that issues disallow instructions to named bots by user-agent. To block a specific AI crawler, you add a User-agent line naming the bot followed by Disallow: / to block the entire site, or a specific path to block only that section.
A minimal example to block OpenAI's training crawler while leaving everything else untouched:
User-agent: GPTBot
Disallow: /
The community-maintained ai.robots.txt project on GitHub publishes a regularly updated list of AI crawler user-agents with ready-to-use robots.txt, .htaccess and Nginx configurations, which is worth consulting before you write rules by hand — new bots appear often and misspelling a user-agent string means the block simply does nothing.
Crucially, robots.txt is advisory, not enforced. As Zyte's Sanaea Daruwalla puts it, "this isn't binding law. For the most part, robots.txt are still looked at as advisory and not legally binding. So, it's still a very grey area." Harry Clarkson-Bennett makes the same point more technically: "Worth noting that blocking via the robots.txt (a directive) is almost wholly ineffective and it needs to be done at a CDN level to be effective." A content delivery network (CDN) — infrastructure like Cloudflare that sits between visitors and your server — can actively block bot requests rather than just politely asking them to leave.
Which Specific AI Crawlers Should I Consider Blocking by Name?
The AI crawlers most commonly named in blocking rules are GPTBot (OpenAI's training bot), Google-Extended (Google's Gemini training opt-out), CCBot (Common Crawl, which many AI labs reuse as training data), Bytespider (ByteDance) and ClaudeBot (Anthropic). Each serves a different company and a different purpose, so blanket-blocking "AI" as one category misses the nuance.
Shifter.io's analysis of the top 10,000 sites found that among the top 1,000 specifically, 29.5% block at least one AI crawler and 21.0% block GPTBot by name. Across all crawlers measured, the most-blocked were CCBot (16.2%), Bytespider (15.3%), GPTBot (15.1%) and ClaudeBot (14.0%) — a pattern that broadly tracks which bots site owners perceive as pure training scrapers versus those that drive citations.
| Crawler | Operator | Purpose | Blocking it affects |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Future ChatGPT training data, not live citations |
| OAI-SearchBot | OpenAI | Live search retrieval | ChatGPT's ability to cite you in real-time answers |
| Google-Extended | Gemini training | Gemini training only — Google Search ranking is unaffected | |
| CCBot | Common Crawl | Open dataset crawl | Reused by many labs for training; broad downstream effect |
| ClaudeBot | Anthropic | Model training/retrieval | Claude's training data and answer citations |
| Bytespider | ByteDance | Training (TikTok/Doubao) | Training data for ByteDance's AI products |
The practical takeaway: decide per-bot, not per-category. Blocking Google-Extended while leaving OAI-SearchBot open, for instance, lets you opt out of Gemini training while still being citable in ChatGPT's live search answers.
Will Blocking AI Crawlers Affect My SEO and AI Overviews Visibility?
Blocking training crawlers like GPTBot or Google-Extended does not directly affect your Google Search rankings, because Googlebot — the crawler that determines your position in Search — is a separate user-agent entirely. Google's own documentation confirms Google-Extended controls only Gemini and Vertex AI training, not Search indexing.
Where it gets more consequential is Google AI Overviews, the AI-generated summary panel that appears above traditional results. AI Overviews draws on the same index as Google Search, so if your content is indexable by Googlebot, it generally remains eligible for inclusion regardless of your Google-Extended setting — but blocking retrieval-oriented bots from other providers (like OAI-SearchBot or PerplexityBot) does remove you from those specific engines' live citations.
This is the training-versus-retrieval distinction many businesses blur. Blocking a training bot mainly affects long-term model development; blocking a retrieval bot has an immediate, visible effect on whether you're cited the next time someone asks that assistant a question. If visibility in ChatGPT, Perplexity or Copilot answers matters to your pipeline, blocking their retrieval crawlers is close to self-exclusion from that channel.
Does UK Copyright and Data Protection Law Cover AI Scraping?
UK copyright law, principally the Copyright, Designs and Patents Act 1988, gives you ownership of original text and images the moment you create them, but it does not currently provide a clear statutory right to prevent AI training on publicly published content — the law here is genuinely unsettled. The Information Commissioner's Office (ICO) oversees data protection compliance where scraped content includes personal data, under UK GDPR.
The legal picture internationally reinforces the uncertainty. Dr Bernd Justin Jütte, Associate Professor in Intellectual Property Law at University College Dublin, notes the practical difficulty of enforcement: "that makes it much more complicated for you guys because you have to have a web crawler that not only looks for robots.txt, but also speaks … 27 different languages within the European Union." In the US, Judge Sidney Stein of the Southern District of New York has ruled that robots.txt directives "are merely requests and do not effectively control access to copyrighted works" — a view echoed by legal scholars examining the file's enforceability, including a peer-reviewed analysis on ScienceDirect and a companion paper on arXiv.
For a UK business, the practical position is: robots.txt is your clearest opt-out signal, it carries reputational and some contractual weight, but it is not a guaranteed legal shield. If your content includes client data, case studies naming individuals, or personal information, UK GDPR obligations around lawful processing apply separately and should be reviewed with your data protection officer or the ICO's guidance directly.
How Do I Check Which AI Crawlers Are Accessing My Site?
Checking crawler activity starts with your server or CDN access logs, which record every request's user-agent string — the text identifier a bot sends to announce itself, such as "GPTBot" or "ClaudeBot". Filtering these logs reveals which AI bots are visiting, how often, and which pages they target.
Most mainstream hosting platforms and CDNs, including Cloudflare, provide a bot analytics dashboard that categorises traffic by known crawler type without needing to parse raw logs manually. This matters because Cloudflare, which powers roughly 20% of all web pages, changed its default stance on 1 July 2025: new customers now have AI crawlers blocked by default, shifting the burden from opting out to opting in (Search Engine Roundtable, 2025). If you're on Cloudflare and haven't checked your settings since mid-2025, you may already be blocking bots you didn't intend to.
Before deciding to block anything, audit first: identify which bots visit, how much bandwidth they consume, and whether any of them — OAI-SearchBot, PerplexityBot, ClaudeBot's retrieval variant — are the same ones responsible for citing you in assistant answers. Blocking blind, based on name recognition alone, risks cutting off exactly the traffic you want.
Alternatives to Outright Blocking: Licensing and Selective Access
Outright blocking isn't the only lever available — licensing content directly to AI companies, or using selective per-bot rules, lets you monetise or control access instead of refusing it entirely. Cloudflare's Pay Per Crawl initiative, launched alongside its July 2025 default-block change, lets site owners charge AI companies for crawl access rather than blocking them outright — an early sign that "no AI crawl without compensation" is becoming a viable middle ground for publishers with genuine leverage.
For most non-publisher businesses, the more realistic choice sits between three options:
| Approach | Best suited to | Trade-off |
|---|---|---|
| Block all AI crawlers | Paywalled publishers, subscription content | Full protection, zero AI-assistant visibility |
| Block training bots, allow retrieval bots | Businesses wanting citations without training reuse | Balanced, but requires per-bot technical accuracy |
| Allow all AI crawlers | Businesses selling products/services, seeking discovery | Maximum visibility, no control over reuse |
Licensing deals remain largely the preserve of major publishers with the content volume and legal resources to negotiate per-crawl compensation. For a facilities, security or software business, the more practical equivalent is simply ensuring your retrieval bots stay open while monitoring training-bot access through your CDN dashboard.
Your AI Crawler Decision Checklist
- Pull your server or CDN logs and identify every AI bot currently visiting your site.
- Separate training crawlers (GPTBot, Google-Extended, CCBot) from retrieval crawlers (OAI-SearchBot, PerplexityBot) before making any rule.
- Check your Cloudflare (or equivalent CDN) dashboard for default-block settings changed since July 2025.
- Decide per-bot, not per-category — block what you don't want, leave open what cites you.
- Add or update your robots.txt file, cross-checking user-agent spelling against the ai.robots.txt GitHub list.
- Review whether any scraped content includes personal data requiring UK GDPR consideration, consulting the ICO if unsure.
- Re-audit logs monthly — new AI crawlers appear regularly and blocking rules go stale.
FAQ
Should I block AI crawlers if I run a small UK business website?
Generally no — small businesses selling products or services benefit more from AI-assistant visibility than they risk from content reuse. Blocking retrieval bots like OAI-SearchBot specifically removes you from citations in ChatGPT and similar tools, which is rarely worth it unless you're a paywalled publisher.
Does blocking GPTBot hurt my Google ranking?
No — GPTBot is an OpenAI crawler, entirely separate from Googlebot, which determines your Google Search ranking. Blocking GPTBot affects only whether OpenAI's models can train on your content, not your position in Google Search results.
What's the difference between Google-Extended and Googlebot?
Google-Extended is a separate robots.txt token that controls whether your content trains Gemini and Vertex AI models, while Googlebot is the crawler that indexes your site for Google Search. Google confirms that blocking Google-Extended does not affect Search ranking.
Does robots.txt actually stop AI bots from scraping my site?
Not reliably — robots.txt is a voluntary request, and TollBit's Q2 2025 data found 13.26% of AI bot requests ignored robots.txt directives, up from 3.3% in Q4 2024. For enforcement rather than a polite request, blocking needs to happen at CDN or firewall level.
Is it legal for AI companies to ignore my robots.txt file?
The legal status is unsettled in the UK and genuinely contested elsewhere — a US federal judge has ruled robots.txt directives "are merely requests and do not effectively control access to copyrighted works," while UK copyright law under the Copyright, Designs and Patents Act 1988 doesn't yet provide a clear statutory remedy for AI training specifically.
Should I block all AI crawlers or just some of them?
Most non-publisher businesses should block selectively rather than entirely — blocking training-only bots like GPTBot or CCBot while leaving retrieval bots like OAI-SearchBot and PerplexityBot open preserves your visibility in AI-driven answers while limiting raw training reuse.
How do I know if AI crawlers are already blocked on my site?
Check your robots.txt file at yoursite.co.uk/robots.txt for existing disallow rules, and check your CDN dashboard — Cloudflare in particular began blocking AI crawlers by default for new customers from 1 July 2025, so you may be blocked without having configured anything yourself.
Getting Cited Instead of Blocked, with Aether AI
Deciding whether to block AI crawlers is really a visibility decision, and visibility is exactly what Aether AI's platform is built to measure and improve. Rather than guessing whether GPTBot, PerplexityBot or Google-Extended should be allowed near your site, Aether AI tracks whether your brand is actually being cited across six AI engines — ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot — so the blocking decision is based on evidence, not instinct.
Aether AI's own measured baseline shows how much work most brands have to do here: across 700 live AI engine scans of UK brands' target keywords in July–August 2026, brands were named in only 23 answers (3.3%), with the longest-established brand in the dataset (a security services business) named in just 6.3% of its 270 scans. That's a low starting visibility rate even for brands actively being tracked — a reminder that blocking crawlers when you're barely visible to begin with compounds the problem rather than solving it.
If you're unsure whether your current robots.txt setup is helping or hurting your AI visibility, Aether AI's free AI-visibility audit at /audit checks how your site currently appears — or doesn't — across major AI engines, with public pricing available if you want ongoing citation tracking and GEO-optimised content built around what's actually working.