How to Allow GPTBot and ClaudeBot: UK Guide 2026

Last updated: 26 October 2026

Allowing GPTBot and ClaudeBot means editing your site's robots.txt file to remove Disallow rules against OpenAI's and Anthropic's crawlers, letting them index your content for AI search answers. GPTBot's share of verified bot traffic rose from 4.7% to 11.7% between July 2024 and July 2025, according to Cloudflare Radar, making this a growing visibility decision for UK businesses.

Key Takeaways

• GPTBot's share of verified bot traffic grew from 4.7% in July 2024 to 11.7% in July 2025, while ClaudeBot rose from 6% to nearly 10% over the same period, according to Cloudflare Radar.
• GPTBot was the most blocked crawler on the web, disallowed by 312 domains (250 fully, 62 partially) as of 2025, per the Cloudflare Blog.
• A robots.txt change can be implemented by editing a single plain-text file at yourdomain.co.uk/robots.txt — no plugin purchase or developer licence is strictly required, though CMS access is.
• ClaudeBot traffic fell 46% in requests between May 2024 and May 2025 even as its overall share shifted, according to Cloudflare, showing AI crawler behaviour changes quickly.
• Aether AI's own measurement of 700 live AI engine scans across four UK brands (July–August 2026) found those brands were named in just 23 answers (3.3%), underlining how much AI visibility still depends on being crawlable in the first place.

What is GPTBot and ClaudeBot?

GPTBot is the web crawler operated by OpenAI, the San Francisco-based company behind ChatGPT, used to gather publicly available web content for training its generative AI models. ClaudeBot is the equivalent crawler operated by Anthropic, the AI safety company behind the Claude assistant, used to collect web content that may contribute to training its models.

Both are separate from the user-triggered crawlers each company also runs. OpenAI additionally operates OAI-SearchBot (which powers live ChatGPT search citations) and ChatGPT-User (triggered when a person asks ChatGPT to browse a specific page). Anthropic runs Claude-User, which "supports Claude AI users" directly, according to Anthropic's support documentation, alongside Claude-SearchBot.

This distinction matters commercially. Blocking GPTBot stops your content being used for training future models, but it does not necessarily stop ChatGPT citing your page in a live search answer — that depends on OAI-SearchBot and ChatGPT-User, which use different user-agent tokens and often need separate rules. OpenAI's own guidance states that "disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models" — a narrower claim than blocking AI visibility altogether, per OpenAI's developer documentation.

Anthropic is explicit about the consequence of blocking: "when a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets," according to Anthropic's help centre.

Why would a UK business want to allow these crawlers?

A UK business allows GPTBot and ClaudeBot to increase the chance its content is referenced or cited inside ChatGPT and Claude answers, rather than remaining invisible to the two most widely used consumer AI assistants. Visibility inside AI-generated answers is becoming a genuine referral channel, not just a training concern.

The scale of this shift is significant. From May 2024 to May 2025, overall crawler traffic rose 18%, with GPTBot requests growing 305% compared with Googlebot's 96%, according to Cloudflare's 2026 report. AI model training traffic reached up to eight times the volume of search crawling in 2025, driven largely by GPTBot, per Cloudflare's 2025 Year in Review.

For context on scale, Googlebot still accounted for over 28% of verified bot traffic in 2025, with GPTBot at 7.5% and Bingbot at 6%, according to Cloudflare's report via InfoQ. That ordering matters for a London retailer or a Manchester professional services firm deciding where to prioritise technical effort: Google Search remains the dominant crawler, but AI crawlers are catching up fast enough to warrant a deliberate decision rather than a default block.

There is a trade-off worth naming honestly. Anthropic's crawl-to-refer ratio reached as high as 500,000:1, and OpenAI's peaked at 3,700:1, with Perplexity having the lowest ratio among major AI platforms, according to Cloudflare's 2025 report. In plain terms, AI crawlers visit far more often than they send visible referral clicks back. Businesses allowing these bots are usually optimising for brand mention and citation presence within AI answers, not direct traffic volume.

How do I edit robots.txt to allow GPTBot and ClaudeBot?

Robots.txt is a plain-text file placed at the root of a domain (e.g. yourcompany.co.uk/robots.txt) that tells compliant web crawlers which parts of a site they may or may not access, under the Robots Exclusion Protocol formalised as IETF RFC 9309. Editing it to allow GPTBot and ClaudeBot means removing any existing Disallow lines naming those user agents, or adding explicit Allow rules.

The process typically runs as follows:

1. Locate the file by visiting yourdomain.co.uk/robots.txt in a browser.
2. Identify blocking rules — look for User-agent: GPTBot or User-agent: ClaudeBot followed by Disallow: /.
3. Edit via your CMS or server. WordPress sites often generate robots.txt dynamically through an SEO plugin (such as Yoast or Rank Math); Shopify and Webflow have dedicated robots.txt editing panels; enterprise CMS platforms may require a developer to edit the file directly on the server or via a CDN edge rule.
4. Replace Disallow with Allow, or remove the block entirely if no other restrictions are needed for that path.
5. Re-upload or save, then re-test the live file at the public URL.
6. Submit for re-crawl where the platform supports it, though most crawlers will simply pick up the change on their next scheduled visit.

Because robots.txt is a plain-text instruction, not an enforced technical barrier, compliant crawlers like GPTBot and ClaudeBot choose to honour it voluntarily — which is also why checking actual server logs afterwards (covered below) matters more than trusting the file alone.

What is the exact robots.txt syntax to permit these bots?

The correct syntax to allow GPTBot and ClaudeBot is a User-agent line naming the exact bot, followed by Allow: / or the removal of any conflicting Disallow rule. Precision matters — a typo in the user-agent string, such as "GPT-Bot" instead of "GPTBot", will cause the rule to be silently ignored.

A minimal permissive example:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

If a site has a broad existing rule such as User-agent: * / Disallow: /private/, that wildcard will still apply to GPTBot and ClaudeBot unless a more specific rule is added above it — robots.txt parsers generally apply the most specific matching user-agent block, per Google's own technical specification.

How do I check whether my site is currently blocking these crawlers?

Checking for an existing block means opening yourdomain.co.uk/robots.txt directly in a browser and scanning for User-agent: GPTBot, User-agent: ClaudeBot, or a catch-all User-agent: * rule paired with Disallow: /. Many sites block these crawlers accidentally through a legacy plugin default rather than a deliberate decision.

This is a widespread issue. GPTBot was the most frequently blocked bot on the web, disallowed by 312 domains — 250 fully and 62 partially — making it more commonly blocked than CCBot or Google-Extended, according to Cloudflare's 2026 analysis. ClaudeBot's own traffic fell 46% in requests between May 2024 and May 2025, which Cloudflare's data links to the combined effect of site-level blocks and shifting crawl behaviour across the same period.

Beyond manually reading the file, UK businesses can:

• Use a robots.txt tester tool (many free versions exist, including within Google Search Console's legacy tools).
• Search server access logs for the strings GPTBot and ClaudeBot to see whether requests are arriving at all, regardless of what robots.txt says.
• Check for a CMS-level "discourage search engines" or "block AI crawlers" toggle — WordPress, for instance, has a well-known setting under Settings → Reading that generates a blanket Disallow rule many site owners forget they enabled.

Do I need developer access, and who is responsible for implementing it?

Making this change requires access to the site's content management system or hosting control panel, but it does not necessarily require a professional developer. For a WordPress or Shopify site with an SEO plugin, a marketing lead or site administrator can usually make the edit directly through the dashboard in under ten minutes.

Developer or IT involvement becomes necessary when the site is built on a custom or enterprise CMS, when robots.txt is generated dynamically by server-side code, or when the business also uses a content delivery network (CDN) or firewall that applies its own bot-management rules on top of robots.txt. In those cases, the hosting provider or IT team is typically the only party with the server or CDN dashboard access required.

Responsibility usually sits with whoever manages the site's SEO or technical marketing function, working alongside IT or a hosting provider for the final deployment. For regulated sectors — financial services firms reporting to the Financial Conduct Authority, or healthcare organisations handling NHS-adjacent data — it is also worth looping in a data protection lead before making the change, which brings us to compliance.

Will allowing these bots affect UK GDPR compliance?

Allowing GPTBot and ClaudeBot to crawl a website does not, by itself, breach the UK General Data Protection Regulation (UK GDPR) or the Data Protection Act 2018, provided the crawled pages do not expose personal data that shouldn't be public. The compliance risk lies in what the crawler can access, not in allowing the crawler itself.

The Information Commissioner's Office (ICO) is the UK's data protection regulator, and its general guidance on web scraping and AI training data applies here: content that is already publicly published on a website carries lower risk than content behind a login, a customer portal, or a staff intranet accidentally left crawlable. Businesses should audit:

• Whether any customer account pages, order histories, or staff directories are technically public but unintentionally reachable by crawlers.
• Whether robots.txt rules properly exclude any /account/, /admin/, or /customer-portal/ paths regardless of the GPTBot and ClaudeBot rules added elsewhere.
• Whether any page contains personal data (names, email addresses, case details) published for a narrow original purpose that was never intended for AI training ingestion.

A sensible approach is to allow GPTBot and ClaudeBot broadly while retaining specific Disallow rules for any path containing personal or commercially sensitive data — the two settings are not mutually exclusive, and most CMS platforms support path-level exceptions alongside bot-level permissions.

Are there costs, and how long before changes take effect?

Editing robots.txt to allow GPTBot and ClaudeBot carries no direct cost — it is a free, text-based configuration change available to anyone with CMS or server access. The only cost implication arises indirectly, if a business needs to pay a developer or agency for an hour of implementation time on a complex enterprise site, or if ongoing AI-visibility tracking software is purchased to measure the results.

Timing is straightforward but not instant. There is no fixed "propagation delay" in the way DNS changes work; instead, GPTBot and ClaudeBot simply re-read robots.txt on their next scheduled visit to a domain, which commercial crawlers typically do within a period of hours to a few days depending on how frequently they already crawl that site. A site with no prior crawl history may wait longer for a first visit.

What common mistakes do people make when allowing these bots?

The most frequent mistake is editing the visible robots.txt file while a CDN, firewall, or security plugin continues to block the same user agents at a different layer, leaving the crawler blocked despite the "correct" robots.txt. Cloudflare, Sucuri, and many managed WordPress hosts apply their own bot-management rules independently of robots.txt, and these take precedence because they block the request before it ever reaches the robots.txt file.

Other common errors include:

• Typos in the user-agent string — "GPT-bot" or "Claude bot" (with a space) will not match and the rule is simply ignored.
• Conflicting rules further down the file — a later wildcard Disallow: / block can override an earlier specific Allow rule depending on parser precedence.
• Confusing training bots with search bots — disallowing only GPTBot while leaving OAI-SearchBot untouched (or vice versa) achieves a narrower outcome than intended, since each token controls a different function.
• Assuming robots.txt is an enforced lock rather than a voluntary request — genuinely malicious or non-compliant scrapers will ignore it entirely, so sensitive content still needs server-level or authentication-based protection regardless of the robots.txt entry.
• Forgetting the CMS-generated override — WordPress's "discourage search engines" visibility setting silently regenerates a blocking robots.txt file even after a manual edit, undoing the change on the next save.

How do I verify GPTBot or ClaudeBot are actually accessing my site?

Verification means checking server access logs for genuine requests from the published IP ranges, rather than trusting robots.txt alone. OpenAI publishes its verified GPTBot IP ranges at openai.com/gptbot.json, and Anthropic publishes its equivalent list at claude.com/crawling/bots.json — matching incoming request IPs against these files confirms a crawl is genuine and not a spoofed user-agent string from an unrelated scraper.

Practical verification steps for a UK business without a dedicated DevOps team:

• Ask your hosting provider to pull access log entries containing GPTBot or ClaudeBot over the past 7–30 days.
• Cross-reference the source IPs against OpenAI's and Anthropic's published JSON files.
• Use a server-side reverse DNS lookup on suspicious requests claiming to be GPTBot or ClaudeBot but originating from unexpected IP ranges — a classic spoofing tell.
• Track AI referral patterns in Google Search Console and analytics platforms over several weeks following the robots.txt change, since a single day's absence of logged visits is not conclusive.

This last point reflects a wider visibility gap many UK brands underestimate. Aether AI's own measurement, published at its AI visibility baseline, ran 700 live AI engine scans of four UK brands' target keywords between July and August 2026 and found those brands were named in only 23 answers — 3.3% overall, with the longest-established brand (in security services) named in 6.3% of its 270 scans. Being crawlable is a prerequisite for citation, not a guarantee of it.

Other configuration checks beyond robots.txt

Robots.txt is only one layer of a website's crawler access control, and firewall or CDN-level blocks frequently override it without the site owner realising. Cloudflare, in particular, offers a dedicated "Verified Bots" and AI bot-management category that many sites have enabled by default, blocking GPTBot and ClaudeBot at the network edge before any request reaches the origin server or its robots.txt file.

Checks worth running alongside the robots.txt edit:

• CDN bot-management settings — review Cloudflare, Fastly, or Akamai dashboards for an "AI crawlers" or "Treat llms.txt like you treated meta keywords: cheap to add, unproven as a lever, no substitute for the fundamentals. What measurably matters is whether AI crawlers can fetch and parse your pages at all — server-rendered content, clean structure, no login walls in front of your best answers." toggle, which is separate from the robots.txt configuration.
• Web Application Firewall (WAF) rules — some WAF rulesets block requests based on user-agent pattern matching regardless of robots.txt instructions.
• .htaccess or server-level rules — Apache and Nginx configurations can block user agents by name at the server level, overriding robots.txt entirely.
• Rate-limiting rules — aggressive rate limits can throttle or drop legitimate crawler requests even without an explicit block, particularly relevant given GPTBot's high crawl frequency relative to its referral volume.

Your GPTBot and ClaudeBot access checklist

• Open yourdomain.co.uk/robots.txt and check for User-agent: GPTBot or User-agent: ClaudeBot blocking rules.
• Replace any Disallow: / entries for these bots with Allow: /, keeping path-level exceptions for private or personal data.
• Decide separately on OAI-SearchBot, ChatGPT-User, Claude-User and Claude-SearchBot, since each controls a different function.
• Check CDN and firewall dashboards (Cloudflare, Fastly, Sucuri) for a separate AI bot-blocking toggle.
• Review WordPress's "discourage search engines" setting and any SEO plugin defaults that regenerate robots.txt.
• Cross-check server logs against OpenAI's and Anthropic's published IP ranges to confirm genuine crawl activity.
• Loop in a data protection lead if any customer or staff data sits on technically public but unintentionally crawlable pages.
• Re-check robots.txt and logs again after 7–30 days to confirm the change has taken effect.

FAQ

What is the exact robots.txt syntax to allow GPTBot and ClaudeBot?

Add User-agent: GPTBot followed by Allow: /, and a separate User-agent: ClaudeBot followed by Allow: /, ensuring no earlier wildcard Disallow: / rule overrides them. Each bot needs its own clearly named block rather than relying on a general User-agent: * rule.

Do GPTBot and ClaudeBot actually respect robots.txt rules?

Yes, both OpenAI and Anthropic state they respect robots.txt as a voluntary standard for their official crawlers. Robots.txt is not a technical enforcement mechanism, however, so it only stops crawlers that choose to comply with the Robots Exclusion Protocol as defined in RFC 9309.

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

GPTBot collects data for training OpenAI's models, OAI-SearchBot powers live citations within ChatGPT's search feature, and ChatGPT-User fetches a specific page only when a person asks ChatGPT to browse it directly. Blocking one does not automatically block the others, since each uses a distinct user-agent token.

Will allowing GPTBot and ClaudeBot affect my Google search rankings?

No, allowing or blocking these AI crawlers has no direct bearing on Google Search rankings, since Googlebot and Google-Extended operate on entirely separate robots.txt rules. Google's own crawler documentation treats AI training permissions as a distinct control from standard search indexing.

How do I verify that a GPTBot or ClaudeBot request is genuine and not spoofed?

Cross-reference the requesting IP address against OpenAI's published range at openai.com/gptbot.json or Anthropic's at claude.com/crawling/bots.json. A request claiming to be GPTBot from an IP outside this published range is likely spoofed and should be treated with caution.

Should I allow AI crawlers for search citations but block them from training?

This is achievable by allowing OAI-SearchBot and Claude-SearchBot while disallowing GPTBot and ClaudeBot specifically, since OpenAI confirms that "disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models" without necessarily affecting search citation eligibility, according to OpenAI's documentation.

Does blocking GPTBot or ClaudeBot stop my content appearing in ChatGPT or Claude answers?

Not entirely — blocking the training-specific bots limits future model training on your content, but live search citation bots like OAI-SearchBot and Claude-SearchBot operate under separate rules and may still surface your page in an AI answer if left unblocked.

Allowing AI crawlers is only the first step — Aether AI measures what happens next

Editing robots.txt to allow GPTBot and ClaudeBot solves the access problem, but it doesn't tell a UK business whether those crawlers are actually translating into citations inside ChatGPT, Claude, Perplexity, Gemini or Copilot answers. That gap — between being crawlable and being cited — is exactly what Aether AI's citation tracking is built to close, monitoring brand mentions across all six major AI engines so businesses can see whether a technical fix like this one is actually moving the needle.

Aether AI's own measurement found real, measurable room for improvement even among established UK brands: across 700 live AI engine scans of four UK brands' target keywords in July–August 2026, those brands were named in just 23 answers (3.3%), with the most established of the four named in only 6.3% of its 270 scans. Robots.txt access is necessary but not sufficient — content still needs to be structured, factual, and genuinely citation-worthy once the crawler gets through the door.

If you've made the robots.txt change and want to know whether it's actually working, Aether AI's free AI-visibility audit checks your current citation presence across all six engines in minutes, and the platform's keyword and competitor tracking shows exactly where you stand against rivals targeting the same queries.

Related Reading

• AI Crawlers List 2026: GPTBot, ClaudeBot & More
• Best Content Format for AI Citation (UK Guide 2026)
• Do AI Crawlers Execute JavaScript? The 2026 UK Answer