GPTBot User Agent: Block, Allow or Monitor It? (2026)

Last updated: 26 October 2026

The GPTBot user agent is OpenAI's web crawler, used to gather publicly available content for training future AI models. It identifies itself in server logs as GPTBot and respects robots.txt directives. By 2026 its share of OpenAI's own crawler traffic had fallen to 63.2%, as rival crawlers OAI-SearchBot and ChatGPT-User took on more of the live-retrieval workload, according to Cloudflare Radar data cited by TechnologyChecker.io.

Key Takeaways

• Aether AI notes that GPTBot is OpenAI's model-training crawler, distinct from OAI-SearchBot (live ChatGPT search results) and ChatGPT-User (on-demand browsing triggered by a user's prompt).
• GPTBot's share of verified bot traffic on Cloudflare's network rose from 4.7% in July 2024 to 11.7% in July 2025, according to Cloudflare data analysed by Digital Applied.
• OpenAI's overall crawl-to-refer ratio peaked at 3,700:1, meaning OpenAI's bots crawl thousands of pages for every one visit they send back, per Cloudflare's 2025 Year in Review, reported by InfoQ.
• Roughly 25% of the top 1,000 websites now block GPTBot, up from about 5% in early 2023, according to Presenc.ai's State of Robots.txt for AI 2026 study.
• Aether AI advises that blocking GPTBot in robots.txt is a legitimate, enforceable control — but it does not retroactively remove content already used in training, and it does not block OAI-SearchBot unless named separately.

What Is the GPTBot User Agent?

The GPTBot user agent is the identifying string OpenAI's web crawler sends with every HTTP request, allowing website operators to recognise and control it through robots.txt — the plain-text file at a site's root that tells automated crawlers which pages they may access. OpenAI operates GPTBot as part of its developer documentation on crawlers, alongside two related but functionally different bots: OAI-SearchBot and ChatGPT-User.

GPTBot was launched by OpenAI, the company behind ChatGPT, specifically to gather web content for training its large language models (LLMs) — the underlying systems that power ChatGPT's text generation. Unlike Googlebot, which indexes pages for search results, GPTBot's primary job has historically been data collection for model training, not live retrieval.

In practice, GPTBot requests arrive from OpenAI-published IP ranges and carry a recognisable user-agent header, typically GPTBot/1.1 or similar version string, plus a full user-agent identifier documented on OpenAI's official crawler documentation. Aether AI treats this documentation as the authoritative reference UK businesses should consult before writing any blocking rule, since IP ranges and bot names have changed more than once since GPTBot's 2023 launch.

GPTBot is not the only OpenAI crawler a UK site will encounter. OAI-SearchBot fetches pages specifically to surface them in ChatGPT's search features, while ChatGPT-User fetches a page only when a live user's prompt triggers a real-time browsing action. OpenAI's own documentation notes that "because these actions are initiated by a user, robots.txt rules may not apply" to ChatGPT-User requests — a distinction that matters enormously for any business deciding how to configure access.

What Is GPTBot Used For: Training Models vs Powering ChatGPT Answers

GPTBot is used by OpenAI primarily to collect publicly accessible web pages for training its large language models, not to generate real-time ChatGPT answers. This is the single most important functional distinction for any business assessing its exposure.

When GPTBot crawls a page, that content may be incorporated into a future training run of a model such as GPT-5 or its successors — a process that happens in batches, not instantly, and is separate from how ChatGPT answers a live user query today. By contrast, OAI-SearchBot is the crawler responsible for the citations and links a user sees when ChatGPT performs a web search inside a conversation. OpenAI's developer documentation states plainly that "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features."

This split matters commercially. A business that blocks GPTBot stops its content being pulled into future training datasets, but it does not necessarily stop that same page being retrieved, summarised and cited in a live ChatGPT search session — because that function sits with OAI-SearchBot, not GPTBot. Cloudflare Radar data reported by TechnologyChecker.io shows GPTBot's share of OpenAI's three combined crawlers fell from 82.5% in Q3 2025 to 63.2% in Q3 2026, as OAI-SearchBot usage roughly doubled — a clear signal that OpenAI is shifting weight toward live retrieval rather than pure training crawl volume.

For GEO (generative engine optimisation) purposes, this is the crux of the decision: training exposure and citation visibility are governed by two different bots, and a robots.txt rule written for one does nothing for the other unless both are explicitly addressed.

How a UK Business Can Check Server Logs for GPTBot Activity

A UK business can confirm whether GPTBot has crawled its site by searching server access logs for the string "GPTBot" alongside the requesting IP address, then cross-checking that IP against OpenAI's published ranges. Most hosting platforms, from cPanel to AWS CloudFront logs, retain raw access logs that record every user-agent header on every request.

The practical steps are straightforward:

1. Pull raw access logs from your web server (Apache, Nginx) or CDN provider (Cloudflare, Fastly, Akamai) for the period you want to review.
2. Filter for the string "GPTBot" in the user-agent field — most log analysis tools (GoAccess, AWStats, or a simple grep command on Linux) will do this in seconds.
3. Record the source IP addresses attached to those requests.
4. Verify against OpenAI's published IP ranges, listed in OpenAI's official crawler documentation, to confirm the traffic isn't spoofed — a known issue where malicious scrapers impersonate legitimate AI crawlers to bypass blocking rules.
5. Check response codes — a high volume of 200 OK responses to GPTBot confirms it is successfully reading your content; a run of 403 or 429 responses confirms your existing block is working.

Cloudflare customers have an easier route: Cloudflare Radar and the Cloudflare dashboard surface verified bot traffic, including GPTBot, without needing to parse raw logs manually. Google Search Console does not report GPTBot activity, since it is an OpenAI product, not a Google one — a common point of confusion among marketing teams expecting a single dashboard for all crawler traffic.

How to Block or Allow GPTBot Using Robots.txt

GPTBot is blocked or allowed through standard robots.txt directives placed in the file at a site's root domain (e.g. example.co.uk/robots.txt), using the exact user-agent token GPTBot that OpenAI has registered. The syntax is simple but unforgiving of small errors.

To block GPTBot from the entire site:

User-agent: GPTBot
Disallow: /

To allow GPTBot everywhere except specific directories, such as a customer account area or internal staff wiki:

User-agent: GPTBot
Disallow: /account/
Disallow: /staff-intranet/
Allow: /

To block GPTBot but still allow OAI-SearchBot (so training data is withheld while ChatGPT search citations remain possible):

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

This last pattern is more common than many businesses realise. The AI Crawler robots.txt Study 2026 found that of 4,962 valid robots.txt files examined from the Tranco top 10,000 sites, 14.3% block GPTBot from the whole site — and roughly half of those GPTBot blockers still allow OAI-SearchBot through. That split suggests many site operators have deliberately separated the "don't train on my content" decision from the "but still let me appear in ChatGPT search" decision.

Case sensitivity, trailing slashes, and placement all matter. The User-agent line must exactly match OpenAI's documented string, and the file must sit at the domain root — a robots.txt file placed in a subdirectory, or served only on www.example.co.uk but not example.co.uk, will be missed entirely by the crawler on one of the two hostnames.

Does Blocking GPTBot Affect Visibility in ChatGPT and SearchGPT?

Blocking GPTBot does not automatically remove a site from ChatGPT's live search results, because citation visibility in ChatGPT's search features is governed chiefly by OAI-SearchBot, not GPTBot. This is the point of greatest confusion among UK marketing teams running GEO strategy in 2026.

A robots.txt rule that disallows only GPTBot leaves OAI-SearchBot free to crawl and cite the page in a live ChatGPT search session. Conversely, a business that disallows all OpenAI bots with a single blanket rule (User-agent: * disallow, or explicitly naming every OpenAI bot) risks disappearing from ChatGPT search citations entirely — a meaningful loss given that Cloudflare's 2025 Year in Review found GPTBot alone already accounted for over 7.5% of verified bot traffic on Cloudflare's network in 2025, trailing only Googlebot (28%+) and ahead of Bingbot (6%).

The same report flagged a structural trade-off: OpenAI's crawl-to-refer ratio peaked at 3,700:1, meaning OpenAI's crawlers request vastly more pages than they ever send visitors back to via a click-through. For a UK business, this means the commercial value of being crawled is rarely in direct traffic — it's in being named inside an AI-generated answer, which Aether AI tracks as a citation event across ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot rather than a traditional analytics session.

Aether AI's own measured baseline across four UK brands, published at its AI-visibility baseline study, found the brands were named in just 23 of 700 live AI engine scans (3.3%) run in July–August 2026 — illustrating how sparse and hard-won AI citations are even for actively optimised content, regardless of robots.txt configuration.

How GPTBot Differs from Google-Extended, CCBot, ClaudeBot and PerplexityBot

GPTBot is OpenAI's crawler, while Google-Extended, CCBot, ClaudeBot and PerplexityBot belong to Google, the Common Crawl Foundation, Anthropic and Perplexity AI respectively — each with its own robots.txt token, purpose, and compliance behaviour. Confusing these bots is one of the most frequent errors UK IT teams make when writing crawler policy.

• Google-Extended controls whether Google may use a page's content for Gemini and AI Overviews training, separate from standard Googlebot indexing for Google Search.
• CCBot is operated by the non-profit Common Crawl Foundation, whose open datasets are widely reused by multiple AI labs — meaning blocking CCBot can indirectly affect several downstream AI products at once, not just one company.
• ClaudeBot is Anthropic's crawler, feeding training data for Claude models.
• PerplexityBot supports Perplexity AI's answer engine, with a function closer to OAI-SearchBot than to GPTBot.

A UK business running a genuine multi-engine GEO strategy needs a policy that addresses each of these separately, because a rule written only for GPTBot leaves five other major crawlers untouched. Research from DEV Community found 59 of 261 top websites with readable robots.txt files (22.6%) disallow GPTBot specifically — but blocking decisions vary widely bot-by-bot across the same sample, underlining that there is no single "block all AI" switch most sites actually use.

Legal and UK GDPR Implications of Allowing GPTBot to Crawl Personal Data

Allowing GPTBot to crawl a website containing personal data raises UK GDPR obligations under the Data Protection Act 2018 and the retained EU General Data Protection Regulation, because a crawler that copies personal data for use in model training is itself a form of processing. The Information Commissioner's Office (ICO) is the UK's data protection regulator, and its guidance on AI and data protection sets expectations relevant to any site operator allowing third-party crawlers near customer or staff data.

Practical risk areas include:

• Staff bios, case studies or testimonials containing names and identifiable details, published on marketing pages that are not excluded from GPTBot in robots.txt.
• Customer account pages, forums or comment sections that may expose personal data if not correctly disallowed.
• Legal basis for processing — a business relying on consent or legitimate interests for its own use of customer data has not automatically extended that basis to a third-party AI crawler's training use.

The ICO has been active in scrutinising generative AI data practices generally, and Aether AI advises that UK organisations handling special category data (health, biometric, or similarly sensitive information under Article 9 of the UK GDPR) should treat any publicly crawlable page containing such data as a high-priority candidate for disallowing GPTBot, OAI-SearchBot and ChatGPT-User alike. Aether AI recommends that legal and data protection teams — not just IT — sign off on which site sections are crawlable, particularly where the Data Protection Act 2018 applies to special category data.

Who Should Own GPTBot Access Decisions Inside a UK Organisation

Deciding GPTBot access rules should sit jointly with IT (technical implementation), legal or data protection (UK GDPR risk sign-off), and marketing or SEO leadership (commercial visibility trade-offs) — no single department holds the full picture alone. This is a governance gap Aether AI sees repeatedly among UK mid-market businesses.

• IT implements and tests the robots.txt syntax, confirms CDN or WAF (web application firewall) rules don't silently override it, and monitors logs for compliance.
• Legal/DPO (Data Protection Officer) assesses GDPR exposure on pages containing personal or commercially sensitive data, and should be consulted before any blanket "Treat llms.txt like you treated meta keywords: cheap to add, unproven as a lever, no substitute for the fundamentals. What measurably matters is whether AI crawlers can fetch and parse your pages at all — server-rendered content, clean structure, no login walls in front of your best answers." decision.
• Marketing/SEO weighs the GEO trade-off — training exposure versus citation visibility in ChatGPT, Perplexity and Google AI Overviews — against commercial priorities.

A written internal policy, reviewed at least every six months given how quickly crawler behaviour changes, is the minimum standard a UK business handling customer data should maintain.

Your GPTBot user agent checklist

• Pull server logs and search for the exact string "GPTBot" to confirm current crawl activity.
• Verify any GPTBot requests against OpenAI's published IP ranges to rule out spoofing.
• Write separate robots.txt rules for GPTBot, OAI-SearchBot and ChatGPT-User rather than one blanket directive.
• Place the robots.txt file at the exact domain root, covering both the www and non-www hostnames.
• Loop in legal or your Data Protection Officer before allowing crawlers near pages with personal data.
• Decide, with marketing, whether training exposure (GPTBot) and citation visibility (OAI-SearchBot) should be treated the same way or differently.
• Re-check logs after any robots.txt change to confirm GPTBot's response codes shifted from 200 to 403/429 as expected.
• Review the policy at least twice a year, since OpenAI has updated its crawler documentation and IP ranges more than once since 2023.

FAQ

What is the GPTBot user agent string?

The GPTBot user agent is the identifying header OpenAI's crawler sends with each request, recognisable in server logs as "GPTBot" alongside a version number. OpenAI documents the exact current string and associated IP ranges on its official developer documentation.

Does GPTBot respect robots.txt?

Yes, GPTBot is designed to respect robots.txt disallow directives, unlike ChatGPT-User, which OpenAI's own documentation says may not be bound by robots.txt because "these actions are initiated by a user." Businesses should treat GPTBot and ChatGPT-User as having different compliance behaviour.

Will blocking GPTBot remove my site from ChatGPT answers?

Not necessarily. ChatGPT's live search citations are primarily driven by OAI-SearchBot, not GPTBot, so blocking GPTBot alone generally stops training-data collection without automatically removing a site from ChatGPT search results.

How do I know if GPTBot requests hitting my server are genuine?

Cross-check the source IP address against OpenAI's published IP ranges listed in its crawler documentation. Spoofed bots commonly copy the user-agent string but originate from unrelated IP addresses, a known issue flagged by web security researchers.

What percentage of websites block GPTBot?

Estimates vary by sample: Presenc.ai's 2026 study found roughly 25% of the top 1,000 sites block GPTBot, while a Tranco top 10,000 analysis found 14.3% block it from the whole site, and a DEV Community study of 261 sites found 22.6% disallow it. The figures differ by sample size and methodology, but all point to a substantial and growing minority blocking GPTBot.

Does blocking GPTBot slow down my server or reduce costs?

Allowing GPTBot adds crawl load similar to any well-behaved bot, though OpenAI's crawl-to-refer ratio of 3,700:1, reported by Cloudflare, means large sites can see substantial crawl volume relative to the referral traffic it generates in return. Blocking GPTBot via robots.txt removes that load entirely, while WAF or IP-level blocking can reduce it further but requires maintaining current IP ranges.

Should a UK business block or allow GPTBot?

There is no single correct answer — it depends on the content type, GDPR exposure, and whether the business values AI citation visibility over training-data privacy. Sites with substantial personal data should lean toward disallowing GPTBot on those sections specifically, while sites prioritising GEO visibility may choose to allow GPTBot broadly while still restricting sensitive directories.

Monitoring GPTBot Alongside Your Wider AI Visibility

Deciding whether to block or allow the GPTBot user agent is only half the job — the other half is knowing whether that decision actually changed how often a brand gets named in ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot. Aether AI's platform tracks citation events across all six of those engines, which is precisely the data gap a robots.txt file alone can't fill: a business can confirm GPTBot is being blocked in its logs and still have no idea whether that choice helped or hurt its AI search presence.

Aether AI's own measured baseline — 700 live AI engine scans across four UK brands' target keywords, run July–August 2026 — found those brands named in just 3.3% of scans overall, with the longest-established brand reaching 6.3%. That scarcity is exactly why Aether AI treats citation tracking, not just crawler access, as the metric that matters for GEO decision-making.

If your business has configured robots.txt for GPTBot and still can't answer whether it's being cited in AI search results, Aether AI's free AI-visibility audit at /audit is the fastest way to find out — and from there, Aether AI's self-service platform can track citations, keywords and competitors across all six major AI engines alongside GSC data in one place.

Related Reading

• Best Content Format for AI Citation (UK Guide 2026)
• Do AI Crawlers Execute JavaScript? The 2026 UK Answer
• Do AI Crawlers Render JavaScript? 2026 UK Guide