Server Logs AI Bot Traffic: A UK Business Guide (2026)

Last updated: 11 November 2026

Server logs AI bot traffic refers to the requests from crawlers like GPTBot, ClaudeBot and PerplexityBot recorded in a website's raw access logs. Throughout 2025, AI bots excluding Googlebot made up an average of 4.2% of HTML requests on Cloudflare's network, peaking at 6.4% in late June, according to Cloudflare Radar 2025 Year in Review.

Key Takeaways

• AI bots excluding Googlebot averaged 4.2% of HTML requests on Cloudflare's network throughout 2025, ranging from 2.4% to 6.4% across the year, per Cloudflare Radar 2025 Year in Review.
• GPTBot's share of verified AI crawler traffic rose from 4.7% in July 2024 to 11.7% in July 2025, according to Cloudflare's crawl-to-click gap report.
• GPTBot requests grew 305% between May 2024 and May 2025, compared with 96% growth for Googlebot, per Cloudflare's "Blocking GPTBot to protect your content is like delisting from Google to protect your brochure. Defensible for a paywalled publisher; self-harm for anyone who sells something. If assistants can't read you, they recommend whoever they can read." analysis.
• Training crawling now drives nearly 80% of all AI bot activity, up from roughly 72% a year earlier, according to Cloudflare.
• Only 14% of top domains currently use robots.txt rules to manage AI and search crawlers, per Cloudflare's 2025 crawler data, leaving most sites unmanaged by default.

What is AI bot traffic in server logs?

AI bot traffic in server logs is the set of HTTP requests recorded by a web server — in files such as Apache's access.log or Nginx's equivalent — that originate from automated crawlers operated by AI companies rather than human visitors or traditional search engines. Each log line typically records a timestamp, requesting IP address, the URL requested, the HTTP status code returned, and a user-agent string — a short piece of text the client sends identifying itself, such as GPTBot/1.1.

Unlike Google Analytics or most JavaScript-based analytics tools, server logs capture every request hitting the web server itself, including bots that never execute JavaScript. This matters because the vast majority of AI crawlers fetch raw HTML and ignore client-side scripts entirely, meaning they are invisible to standard analytics dashboards but fully visible in raw server logs.

Two broad categories of AI bot exist. Training crawlers harvest content to build or refine large language models — GPTBot and ClaudeBot are the best-known examples. Fetcher or retrieval bots make real-time requests when a user asks an AI assistant a question that requires live web data, such as OpenAI's ChatGPT-User or Perplexity's PerplexityBot. Cloudflare data shows training crawling now accounts for nearly 80% of AI bot activity, up from around 72% a year earlier, according to Cloudflare's crawl-to-click gap analysis — so most of what a UK site sees in its logs is model training, not a live user query.

Which AI bots and crawlers are hitting UK business websites right now?

The AI crawlers most commonly recorded in UK website logs today are OpenAI's GPTBot and ChatGPT-User, Anthropic's ClaudeBot, Google-Extended, Common Crawl's CCBot, and Perplexity's PerplexityBot. Their relative volumes have shifted sharply over the past 18 months.

Among verified bots tracked across Cloudflare's network, Googlebot — Google's own search indexing crawler — still accounted for more than 28% of verified bot traffic in 2025, with OpenAI's GPTBot second at approximately 7.5%, according to Cloudflare Radar's Year in Review. But growth rates tell a different story: GPTBot requests grew 305% between May 2024 and May 2025, against 96% growth for Googlebot over the same period, per Cloudflare's analysis. Separately, GPTBot's share of verified crawler traffic climbed from 4.7% to 11.7% between July 2024 and July 2025, while ClaudeBot grew from 6% to nearly 10% over the same window, according to Cloudflare's crawl-to-click gap report.

Fastly's network data tells a notably different story, which is useful for UK IT teams because it shows how much bot mix varies by hosting infrastructure. Fastly's Q2 2025 Threat Insights Report found AI crawlers made up almost 80% of all AI bot traffic it observed, but Meta's AI bots generated 52% of that crawler traffic — more than double Google's 23% or OpenAI's 20% share — according to the Fastly Q2 2025 Threat Insights Report. The same report found ChatGPT generates the most real-time fetcher traffic of any assistant, with 98% of fetcher bot requests attributable to OpenAI's bots, per Fastly.

The practical implication for a UK marketing or SEO lead is that relative bot volume is not static. Aether AI notes that a log analysis done in early 2025 will understate GPTBot's current footprint; a review should be repeated at least quarterly.

How do I identify and verify AI bot traffic, not just trust the user-agent string?

Identifying AI bot traffic starts with filtering server logs for known user-agent substrings, but verifying it requires cross-checking the requesting IP address against the crawler operator's published IP ranges or performing a reverse DNS lookup. A user-agent string alone is self-reported text that any scraper can fake.

The verification process has three steps:

1. Filter logs for known AI user-agent tokens — GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider (ByteDance), Amazonbot, and Applebot-Extended are the most common tokens to search for.
2. Cross-reference the source IP against the operator's published ranges. OpenAI publishes IP ranges and verification guidance in its crawler documentation, and Anthropic publishes equivalent guidance in its crawler support guide. A request claiming to be GPTBot from an IP outside OpenAI's published range is almost certainly spoofed.
3. Run a reverse DNS lookup on the IP and confirm the forward lookup matches — genuine Googlebot traffic, for instance, resolves to a googlebot.com or google.com hostname that matches when looked up in both directions.

This matters commercially as well as technically. Server logs showing "GPTBot" hits are sometimes scrapers or SEO-tool crawlers borrowing a recognisable name to avoid being blocked — treating every string match as genuine risks both over-counting AI crawler load and under-counting genuine malicious scraping.

What tools can I use to analyse server logs for AI bot activity?

Log analysis tools for AI bot traffic range from free command-line utilities to enterprise observability platforms, and the right choice depends on log volume and in-house technical capacity. A small UK business with a single WordPress site has very different needs from a retailer running multiple regional domains.

Raw crawl counts only answer half the question. Knowing that ClaudeBot hit a product page 40 times last month doesn't tell a marketing lead whether that page is actually being cited in Claude's answers. Aether AI's platform tracks citation presence across six AI engines — ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot — alongside keyword and competitor tracking and Google Search Console integration, so crawl activity from server logs can be read against actual citation outcomes rather than treated as a number in isolation. A free AI-visibility audit is available at aether-ai.co.uk/audit for businesses wanting an initial read before investing in deeper log tooling.

How can I block or control unwanted AI bot traffic?

Controlling AI bot traffic is done through three layers — robots.txt directives, server-level firewall or access rules, and CDN-level bot management — and each offers a different degree of enforcement. Only 14% of top domains currently use robots.txt rules to manage AI and search crawlers, according to Cloudflare's 2025 crawler data, meaning most sites are leaving the decision entirely to each AI company's default behaviour.

Robots.txt is the first and lowest-effort layer. A business can disallow GPTBot, Google-Extended or CCBot individually:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

The critical limitation is that robots.txt is a voluntary convention, not an enforcement mechanism. Reputable operators such as OpenAI and Anthropic state they honour it, as documented in OpenAI's crawler documentation and Anthropic's crawler support guide, but a scraper with no reputation to protect can simply ignore the file.

Server-side firewall rules offer actual enforcement. A .htaccess rule on Apache, an Nginx map block, or a web application firewall (WAF) rule can return a 403 status to any request matching a known AI user-agent or IP range, regardless of whether the bot respects robots.txt.

CDN-level bot management, such as Cloudflare's bot rules, sits in front of the origin server entirely and can rate-limit, challenge or block AI crawlers before they ever reach hosting infrastructure — useful where server cost and load, not just data control, is the concern.

What are the legal and data protection implications of AI bots scraping site content?

AI bots scraping website content raises UK GDPR and Data Protection Act 2018 questions wherever the scraped pages contain personal data, because the Information Commissioner's Office (ICO) — the UK's independent data protection regulator — treats web scraping of personal data as processing subject to the same lawful-basis requirements as any other collection method.

A business publishing staff bios, customer testimonials with full names, or case studies naming individuals should treat large-scale AI crawling of those pages as a data processing question, not purely a content-licensing one. The ICO has published general guidance addressing web scraping for AI training purposes as part of its wider work on AI and data protection, reflecting the regulator's view that personal data does not lose its legal protection by being published on a public website.

Separately from personal data, there is a commercial content question: UK copyright law does not currently include a clear exemption for AI training use, which is why many publishers rely on robots.txt disallow rules and emerging machine-readable licensing signals rather than copyright enforcement alone to manage AI access. Businesses with legal teams should treat the decision to allow or block specific crawlers as a joint call between IT, marketing and legal — not a technical setting left entirely to a developer.

How does AI bot traffic affect server costs and hosting bills?

AI bot traffic adds measurable load to hosting infrastructure because every crawl request consumes server compute, bandwidth and, on dynamic sites, database queries — the same resources a paying customer's visit would use. With AI bots excluding Googlebot averaging 4.2% of HTML requests across Cloudflare's network in 2025, and Googlebot adding a further 4.5% on top, according to Cloudflare Radar's 2025 Year in Review, a meaningful share of total server load on an unmanaged site is now non-human.

For a UK business on a fixed-resource hosting plan, the practical risks are:

• Bandwidth overage charges, where a hosting package has a data transfer cap and crawler traffic pushes a site over it.
• Server resource contention, where aggressive crawling by multiple bots simultaneously slows page load for genuine visitors during a traffic spike.
• Database load on dynamic sites, where bots triggering search, filter or pagination endpoints generate expensive queries human visitors rarely run in volume.

E-commerce sites with large product catalogues and faceted navigation are particularly exposed, since crawlers can generate near-infinite URL combinations through filter parameters. The fix is usually architectural rather than purely restrictive: disallowing parameterised URLs in robots.txt, adding noindex to filtered views, and rate-limiting via a WAF tend to reduce wasted crawl load without blocking AI bots from legitimate content pages entirely.

What are the SEO and content-licensing trade-offs of allowing AI bots to crawl my site?

Allowing AI bots to crawl a site trades potential visibility in AI-generated answers against the loss of direct traffic, since most AI training crawls do not send a visitor back to the source page the way a traditional search click does. This is the central trade-off UK businesses now face in technical GEO (Generative Engine Optimisation) strategy.

Training crawling now accounts for nearly 80% of AI bot activity, according to Cloudflare's crawl-to-click gap research — meaning the overwhelming majority of AI bot visits are feeding a model's future answers, not generating an immediate referral. That's the "gap" in Cloudflare's own framing: heavy crawling, light clicking.

The counter-argument for allowing crawling is that citation in an AI answer — ChatGPT, Perplexity or Google's AI Overviews naming a business directly — is itself a visibility and trust signal, even without a click. Aether AI's own measured baseline, published at aether-ai.co.uk/insights/ai-visibility-baseline-2026, found that across 700 live AI engine scans of UK brands' target keywords run in July–August 2026, the tracked brands were named in just 23 answers — 3.3% of scans — with the longest-established brand, in security services, named in 6.3% of its 270 scans. That low baseline is consistent with the wider industry pattern Cloudflare describes: most sites crawled are not yet being cited back, which is exactly why blocking crawlers entirely forecloses an opportunity that is still small but growing.

Allow vs block: the practical decision

Your server logs AI bot traffic checklist

• Pull a 30-day sample of raw server logs and filter for known AI user-agent tokens (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended).
• Cross-check a sample of flagged IPs against the operator's published ranges or run a reverse DNS lookup before trusting the user-agent string.
• Decide, per bot, whether to allow, block or rate-limit — don't apply a single blanket rule to every AI crawler.
• Update robots.txt to reflect that decision, and add a server-side firewall rule for any bot that must be enforced rather than merely requested.
• Check hosting bandwidth and resource usage reports for crawl-driven spikes, particularly around parameterised URLs and search pages.
• Assign clear ownership — IT for enforcement, marketing or SEO for the allow/block strategy, legal for anything touching personal data.
• Re-run the log review quarterly; bot market share shifted sharply through 2025 and will keep moving.
• Run a free AI-visibility audit to see whether crawling is translating into any AI citation presence at aether-ai.co.uk/audit.

FAQ

What exactly counts as AI bot traffic in server logs?

AI bot traffic is any request in a server's access log sent by an automated crawler operated by an AI company, identified by a distinct user-agent string such as GPTBot/1.1 or ClaudeBot/1.0. It covers both training crawlers that harvest content for model development and fetcher bots that retrieve pages in real time to answer a live user query.

Can Google Analytics show me AI bot traffic?

No — most AI crawlers don't execute JavaScript, so they never trigger the tracking code that platforms like Google Analytics rely on. Raw server logs are the only reliable source, since they record every request hitting the server regardless of whether JavaScript runs.

Which AI bots should a UK business website check for first?

GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot (Common Crawl) and PerplexityBot (Perplexity) are the most commonly seen crawlers. GPTBot's share of verified crawler traffic rose from 4.7% to 11.7% between July 2024 and July 2025, according to Cloudflare, making it one of the fastest-growing bots to track.

Does blocking GPTBot in robots.txt actually stop it crawling?

It stops any crawler that honours the file, and OpenAI states it respects robots.txt per its crawler documentation. But robots.txt is voluntary, not enforced — a firewall rule or WAF block is needed to guarantee a bot is stopped regardless of its own compliance.

Who should be responsible for monitoring AI bot traffic in a UK business?

Responsibility typically splits three ways: IT or a hosting provider handles detection and server-side enforcement, marketing or an SEO lead decides the allow/block strategy around AI visibility, and legal reviews anything touching personal data under UK GDPR. Smaller businesses often hand the whole task to an external agency or platform rather than building it in-house.

Is AI bot traffic a GDPR concern even if my site has no login area?

Yes, if any crawled page contains personal data — staff names, customer testimonials, named case studies — because the ICO treats scraping of personal data as processing subject to UK GDPR regardless of whether the site requires a login. Pages containing no personal data carry a copyright and licensing question instead, not a data protection one.

How do I know a bot claiming to be PerplexityBot or GPTBot isn't spoofed?

Check the requesting IP address against the operator's published IP ranges, documented for OpenAI in its crawler guidance and for Anthropic in its crawler support article, then confirm a reverse DNS lookup on that IP resolves back to the operator's domain. A user-agent string alone proves nothing, since any scraper can set it to anything it likes.

Bringing AI bot visibility and citation tracking together with Aether AI

Reading server logs tells a UK business which AI bots are crawling its site, but it doesn't answer the question that actually matters commercially: is that crawling turning into citations in ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini or Copilot answers? Aether AI was built to close that gap — connecting crawl activity to measured citation outcomes rather than leaving businesses to interpret raw log counts alone.

The platform tracks citation presence across all six major AI engines, alongside keyword and competitor tracking and Google Search Console integration, giving marketing and SEO teams a single view of whether their content strategy is actually landing in AI answers. Aether AI's own measured baseline — 700 live AI engine scans of UK brands' target keywords run in July–August 2026 — found brands named in just 3.3% of scans overall, underlining how early-stage AI citation still is for most UK businesses and why treating crawl data and citation data together matters more than watching either in isolation.

Businesses wanting to see where they currently stand can run a free AI-visibility audit at aether-ai.co.uk/audit, with full pricing details available at aether-ai.co.uk/pricing.

Related Reading

• AI Traffic vs Organic Traffic: The UK 2026 Guide
• How to Build an Entity for Your Brand in the UK
• GPTBot User Agent: Block, Allow or Monitor It? (2026)