OAI-SearchBot Explained: UK Business Guide 2026

Last updated: 5 November 2026

OAI-SearchBot is OpenAI's dedicated web crawler for ChatGPT's search feature, separate from the GPTBot crawler used for model training. It indexes pages so ChatGPT can cite and link to them in live search answers, and sites that opt it out via robots.txt will not appear in those answers, though they can still surface as navigational links (OpenAI Developer Documentation, 2026).

Key Takeaways

• Aether AI notes that OAI-SearchBot is operated by OpenAI and crawls the web specifically to power ChatGPT's search and citation feature, not to train OpenAI's language models.
• Blocking OAI-SearchBot removes a site from ChatGPT's search answers, though the page can still appear as a plain navigational link rather than a cited source (OpenAI Developer Documentation, 2026).
• A robots.txt change takes around 24 hours to be reflected in OpenAI's search-facing crawling behaviour (icoda.io, 2026).
• More than half of traffic claiming to be OAI-SearchBot is fake: 54.1% of requests identifying themselves as OAI-SearchBot did not originate from OpenAI's published IP ranges in a production detection corpus (WebDecoy, 2026).
• Blocking AI crawlers is now mainstream, not fringe: about 49% of top sites block OAI-SearchBot, a rise of 104% compared with 12 months earlier (BuzzStream 2026 study via tryrankly.com, 2026).

What is OAI-SearchBot?

OAI-SearchBot is a web crawler operated by OpenAI, the company behind ChatGPT, that is used exclusively to discover and index content for ChatGPT's search and browsing features rather than for training its underlying AI models. It identifies itself with its own user-agent string and respects a dedicated robots.txt directive, separate from OpenAI's other bots.

OpenAI runs three distinct crawlers under one roof, and conflating them is the single most common technical mistake UK marketing and IT teams make. OAI-SearchBot fetches pages so ChatGPT can surface them as live, cited search results — similar in purpose to how Googlebot feeds Google's index. GPTBot, by contrast, collects data that may be used to train future foundation models. ChatGPT-User is a third, separate agent that fires only when a live ChatGPT user asks the assistant to browse a specific page in real time.

Each of these three crawlers can be allowed or blocked independently in robots.txt, using its own named token. A site can, for example, welcome OAI-SearchBot for citation visibility while blocking GPTBot from using its content for model training — a distinction many UK businesses don't realise is available to them until they check their server logs.

OAI-SearchBot vs GPTBot: what's the difference?

OAI-SearchBot and GPTBot are both OpenAI crawlers, but they serve opposite purposes: OAI-SearchBot retrieves content to answer live ChatGPT search queries, while GPTBot gathers data for training OpenAI's generative AI foundation models. As OpenAI's own documentation states, "GPTBot is used to crawl content that may be used in training our generative AI foundation models" — a mandate that has nothing to do with what ChatGPT shows a user searching right now.

This matters commercially. A business that blocks GPTBot but allows OAI-SearchBot can keep its proprietary content out of future model training while still being eligible to appear as a cited source in ChatGPT's search results today. The reverse configuration — blocking OAI-SearchBot but allowing GPTBot — would make a site invisible in ChatGPT search citations while still letting OpenAI train on its content, which is rarely what businesses actually want.

Context matters here too. ChatGPT's own share of worldwide generative AI web traffic fell from roughly 76% to about 53% over the year to 2026, even as citation presence in US ChatGPT prompts rose from about 1.6% in June 2025 to roughly 6.8% by May 2026 (Similarweb, 2026). Fewer total visits, but more weight given to cited sources, makes the OAI-SearchBot decision more consequential, not less.

How OAI-SearchBot crawls and indexes content for ChatGPT search

OAI-SearchBot crawls publicly accessible web pages the same way any search engine spider does: it fetches HTML, follows internal links, and builds an index that ChatGPT's search feature queries at response time. Unlike a one-off GPTBot training pass, OAI-SearchBot recrawls on an ongoing basis to keep ChatGPT's search results current.

When a ChatGPT user asks a question that triggers a search, the system retrieves and ranks indexed pages, then generates an answer that cites the source with a clickable link — much like a featured snippet in Google Search. Pages that load cleanly, use descriptive headings, and present facts in extractable, self-contained sentences are easier for this retrieval layer to quote accurately. That's the same structural discipline Aether AI builds into every article it generates: clear H2/H3 hierarchy, answer-first paragraphs, and named sources an engine can verify.

OAI-SearchBot's measured footprint is still modest relative to the wider AI crawler ecosystem. It accounts for roughly 0.90% share of AI bot traffic tracked via Cloudflare Radar, though that share is up 80% compared with 12 months earlier (Cloudflare Radar AI Insights via tryrankly.com, 2026). Low volume, fast growth — the pattern suggests UK businesses configuring robots.txt today are setting policy for a crawler whose influence is scaling quickly.

How to identify genuine OAI-SearchBot visits and verify them in your logs

A genuine OAI-SearchBot request carries the user-agent string "OAI-SearchBot" and originates from an IP address listed in OpenAI's published range file at openai.com/searchbot.json. Checking the user-agent string alone is not enough, because user-agent headers are trivially spoofed by scrapers, SEO tools, and malicious bots wanting to appear legitimate.

This is not a theoretical risk. In a production detection corpus, 54.1% of traffic claiming to be OAI-SearchBot did not come from OpenAI's published IP ranges at all (WebDecoy, 2026). That means a majority of self-identified OAI-SearchBot hits a typical UK site logs could be impersonators — scrapers harvesting content, competitors benchmarking pages, or security scanners.

To verify genuine traffic, a UK IT team should:

1. Pull the user-agent string "OAI-SearchBot" from server access logs (Apache, Nginx, or CDN logs such as Cloudflare or Fastly).
2. Extract the source IP address for each matching request.
3. Cross-reference each IP against OpenAI's current searchbot.json range file, ideally via an automated reverse-DNS or IP-match script run on a schedule, since the published ranges change.
4. Flag and investigate any request claiming the OAI-SearchBot user-agent from an IP outside the published ranges — this is very likely spoofed traffic, not genuine OpenAI crawling.
5. Repeat the same process separately for gptbot.json and chatgpt-user.json, since each crawler publishes its own distinct range.

Businesses using a Web Application Firewall (WAF) — Cloudflare, Akamai, or AWS WAF are the common UK choices — can usually automate this IP verification as a managed rule, rather than relying on manual log review.

Allowing or blocking OAI-SearchBot in robots.txt

Robots.txt controls are the primary mechanism for permitting or excluding OAI-SearchBot, and OpenAI treats the directive independently from GPTBot and ChatGPT-User. A site allows OAI-SearchBot by default if robots.txt contains no explicit disallow rule for that token; it blocks OAI-SearchBot by adding a specific user-agent block.

To allow OAI-SearchBot (the default state for most sites), no action is needed — simply avoid adding a disallow rule for it.

To block OAI-SearchBot entirely, a site adds this to its robots.txt file:

User-agent: OAI-SearchBot
Disallow: /

To block it from specific sections only (for example, a customer portal or paywalled content):

User-agent: OAI-SearchBot
Disallow: /account/
Disallow: /members/

Once this change is published, it does not take effect immediately. A robots.txt change can take about 24 hours for OpenAI's systems to reflect it in search-facing behaviour (icoda.io, 2026), so teams testing a new rule should wait a full day before drawing conclusions from log data.

It's worth repeating the consequence of blocking, because it's often misunderstood: opting out of OAI-SearchBot removes a site from ChatGPT's cited search answers, but the URL can still appear as an uncited navigational link elsewhere in ChatGPT's interface (OpenAI Developer Documentation, 2026). Blocking does not guarantee total invisibility — it removes citation eligibility specifically.

Should a UK business block OAI-SearchBot? The visibility trade-off

Blocking OAI-SearchBot is now a majority behaviour among larger sites, but that doesn't automatically make it the right call for every UK business. About 49% of top sites block OAI-SearchBot, a figure that has grown 104% against the same period a year earlier (BuzzStream 2026 study via tryrankly.com, 2026). A similar pattern played out with GPTBot: 35.7% of the world's top 1,000 sites were blocking GPTBot by August 2024, up from roughly 5% at its August 2023 launch (Originality.AI via ppc.land, 2024).

The trade-off is straightforward to state, harder to weigh:

Case for allowing OAI-SearchBot: a business becomes eligible for citation in ChatGPT search answers, a channel growing in weight even as ChatGPT's overall web traffic share softens from roughly 76% to about 53% year-on-year (Similarweb, 2026). Citation presence in prompts rose from 1.6% to roughly 6.8% over the same period, suggesting the citation surface itself is becoming more valuable even as raw traffic shifts elsewhere.

Case for blocking OAI-SearchBot: publishers with subscription models, syndicated news content, or data-licensing arrangements may want to withhold free indexing until commercial terms are agreed — the same argument many UK news publishers made against GPTBot in 2023 and 2024.

For most UK SMEs, professional services firms, and B2B companies trying to build discoverability in AI answer engines, blocking OAI-SearchBot removes the one channel most directly aligned with appearing as a cited, clickable source in ChatGPT. This is the exact problem Aether AI's citation tracking exists to monitor across six engines — ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini, and Copilot — because a blocking decision made by an IT team with no visibility into GEO strategy can quietly erase a brand from an entire answer channel.

OAI-SearchBot vs Googlebot and Bingbot: how AI crawlers differ from traditional search spiders

OAI-SearchBot differs from Googlebot and Bingbot in purpose, scale, and the destination of its data: Googlebot and Bingbot build traditional, link-based search indexes ranking billions of pages for organic results, while OAI-SearchBot specifically feeds a generative, conversational answer engine. Googlebot has operated continuously since the late 1990s and processes a vastly larger crawl footprint; OAI-SearchBot is comparatively new and, at roughly 0.90% of tracked AI bot traffic, still a small fraction of overall crawler activity (Cloudflare Radar via tryrankly.com, 2026).

The practical difference for a UK business: ranking well in Google requires decades of accumulated SEO signal across an enormous competitive set, while appearing in ChatGPT search citations is a newer, less saturated opportunity — which is also why blocking rates are rising so sharply as more sites wake up to the channel's existence.

Who should own OAI-SearchBot configuration inside a UK business?

Responsibility for OAI-SearchBot access typically sits across IT, marketing, and sometimes legal functions, and the handoff between them is where most configuration mistakes happen. IT teams usually hold the technical keys to robots.txt and WAF rules; marketing and SEO leads understand the visibility consequences; legal or data protection teams assess licensing and compliance implications.

The most common failure mode in UK organisations is a robots.txt change made unilaterally by a developer or hosting provider, with no sign-off from the team that owns search and AI visibility. A website migration, a new CDN configuration, or a default WAF bot-management rule can silently block OAI-SearchBot without anyone in marketing noticing until citation tracking shows a drop-off.

A workable governance model:

• IT/DevOps owns implementation: editing robots.txt, configuring WAF rules, verifying IP ranges against OpenAI's published JSON files.
• Marketing/SEO owns the decision: whether blocking or allowing aligns with commercial strategy, based on citation tracking data.
• Legal/Data Protection reviews licensing terms and any data protection questions before a large-scale allow decision is finalised, particularly for sites handling customer data.

This is also where a platform like Aether AI's GSC integration and cross-engine citation tracking earns its keep: it gives marketing teams the visibility data needed to make an informed case to IT, rather than discovering a blocking rule only after visibility has already disappeared.

Does allowing OAI-SearchBot raise UK GDPR concerns?

Allowing OAI-SearchBot to crawl a site raises UK GDPR (the UK General Data Protection Regulation, the retained EU law governing personal data processing after Brexit) concerns primarily where crawled pages contain personal data — customer reviews with real names, staff bios, forum posts, or user-generated content. The core question for any UK data controller is whether publicly crawlable pages expose personal data to a third-party AI system in a way the original data subject did not anticipate.

The Information Commissioner's Office (ICO), the UK's data protection regulator, has published general guidance on AI and data protection, though it has not issued crawler-specific guidance naming OAI-SearchBot directly. UK businesses publishing pages containing personal data should apply standard data minimisation principles under the UK GDPR: avoid publishing unnecessary personal data on crawlable public pages, and consider whether sensitive sections (staff directories, customer testimonials with identifiable details) warrant exclusion via robots.txt regardless of the AI citation benefit.

Static company information — services, pricing, locations, blog content — carries minimal GDPR exposure from AI crawling. The practical risk concentrates on pages built around individuals' names, images, or personal circumstances, where a cautious UK business may choose to block OAI-SearchBot from those specific URL paths using the targeted disallow rules shown earlier, while still allowing the rest of the site.

Your OAI-SearchBot configuration checklist

• Confirm whether your current robots.txt file contains any existing rule for OAI-SearchBot, GPTBot, or ChatGPT-User.
• Decide your allow/block policy separately for each of the three OpenAI crawlers, based on citation goals versus training-data concerns.
• Cross-reference any self-identified OAI-SearchBot traffic in your server logs against OpenAI's published searchbot.json IP range before trusting it.
• Check your CDN or WAF's default bot-management settings — many block AI crawlers silently without an explicit robots.txt rule.
• Allow at least 24 hours after any robots.txt change before assessing its effect on crawl or citation behaviour.
• Review any pages containing personal data (staff pages, testimonials, forum content) for UK GDPR exposure before allowing broad AI crawler access.
• Assign clear ownership across IT, marketing, and legal so a future site migration doesn't silently change your AI crawler policy.
• Track citation performance across ChatGPT and other AI engines after any configuration change, rather than assuming the policy is working.

FAQ

What is OAI-SearchBot and what company operates it?

OAI-SearchBot is a web crawler built and operated by OpenAI, used specifically to index content for ChatGPT's search and citation feature. It is distinct from GPTBot, OpenAI's training-data crawler, and from ChatGPT-User, its real-time browsing agent.

What is the difference between OAI-SearchBot and GPTBot?

OAI-SearchBot indexes pages so ChatGPT can cite them in live search results, while GPTBot gathers content that may be used to train OpenAI's generative AI foundation models. The two crawlers can be allowed or blocked independently in robots.txt using separate user-agent tokens.

How do I block OAI-SearchBot using robots.txt?

Add a user-agent block for "OAI-SearchBot" with a "Disallow: /" directive to block it site-wide, or disallow specific folders to block only certain sections. Changes typically take about 24 hours to be reflected in OpenAI's search-facing crawling behaviour (icoda.io, 2026).

How can I verify a crawler claiming to be OAI-SearchBot is genuine?

Cross-reference the request's source IP address against OpenAI's published range file at openai.com/searchbot.json, rather than trusting the user-agent string alone. This matters because 54.1% of traffic claiming to be OAI-SearchBot did not originate from OpenAI's published IP ranges in one detection corpus (WebDecoy, 2026).

What happens if I block OAI-SearchBot on my website?

Blocking OAI-SearchBot removes your site from being cited in ChatGPT's search answers, though your pages may still appear as plain navigational links without citation (OpenAI Developer Documentation, 2026). Roughly 49% of top sites now apply this block, a 104% rise year-on-year (BuzzStream 2026 study via tryrankly.com, 2026).

Is there a performance cost to allowing OAI-SearchBot to crawl my site?

OAI-SearchBot's measured crawl volume is a small share of overall AI bot traffic — roughly 0.90% via Cloudflare Radar tracking — so server load from genuine OpenAI requests is typically modest for most UK business sites (Cloudflare Radar via tryrankly.com, 2026). The bigger performance and security concern tends to be spoofed traffic falsely claiming the OAI-SearchBot identity, which should be filtered via IP verification rather than blanket robots.txt rules.

Does allowing OAI-SearchBot raise UK GDPR issues?

Allowing OAI-SearchBot mainly raises UK GDPR concerns on pages containing personal data, such as staff bios or customer testimonials with identifiable details, rather than on standard commercial content. UK businesses should apply standard data minimisation principles and consider excluding specific personal-data pages via targeted robots.txt rules while allowing the rest of the site.

Securing your AI search visibility with Aether AI

Deciding whether to allow or block OAI-SearchBot is exactly the kind of technical GEO decision that gets made in isolation and then quietly undermines a company's AI visibility months later. Aether AI builds this concern directly into its platform: citation tracking across six AI engines — ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini, and Copilot — means a robots.txt change or a WAF default that silently blocks OAI-SearchBot shows up in the data before it becomes a six-month mystery.

Aether AI's own measured baseline illustrates the scale of the challenge: across 700 live AI engine scans of UK brands' target keywords run in July–August 2026, those brands were named in just 23 answers — 3.3% — with the longest-established brand, in security services, named in 6.3% of its 270 scans (published at aether-ai.co.uk/insights/ai-visibility-baseline-2026). That's a small starting percentage even for a brand actively managing crawler access, which underlines why verifying OAI-SearchBot configuration is a necessary but not sufficient step — it has to sit inside an ongoing visibility strategy, not a one-off robots.txt edit.

If your business isn't sure whether OAI-SearchBot can currently see your site, or whether that's even the right call, Aether AI's free AI-visibility audit at /audit checks your current standing across major AI engines, and the platform's keyword and competitor tracking shows what's achievable once crawler access and content structure are both working in your favour.

Related Reading

• How to Build an Entity for Your Brand in the UK
• GPTBot User Agent: Block, Allow or Monitor It? (2026)
• How AI Reads Your Website: The Technical Reality