Last updated: 26 October 2026
AI reads your website by sending automated crawlers — such as GPTBot, ClaudeBot and Google-Extended — to fetch your HTML, strip out navigation and scripts, then break the remaining text into chunks that get converted into numerical representations called embeddings. These chunks are retrieved and cited when a user asks a related question. Headings and early-page content matter most.
Key Takeaways
- Content with clear subheadings achieves double the citation rate in AI Overviews and chatbot answers, based on an analysis of 7,000 AI search results, according to Crispy Content (2026).
- GPTBot's share of AI crawler traffic rose from 2.2% to 7.7% between May 2024 and May 2025 — a 305% increase in requests — according to Cloudflare Radar (2026).
- Only around 14% of the top 10,000 domains have robots.txt rules targeting AI bots specifically, per Cloudflare Radar (2026).
- 78.4% of ChatGPT citations tied to a question come from a heading, and 44.2% come from the first 30% of a page, according to Kevin Indig / Growth Memo (2026).
- Adding JSON-LD schema alone produced no statistically significant citation uplift across Google AI Overviews, AI Mode or ChatGPT, per an Ahrefs study of 1,885 pages, via AuthorityTech (2026).
What methods do AI crawlers use to read and extract content from a website?
AI crawlers read a website through a four-stage pipeline: fetching the page, parsing its HTML, chunking the clean text into passages, and converting those passages into embeddings — numerical vectors that represent meaning rather than exact wording. This process is fundamentally different from how a human reads a page in a browser.
The fetch stage is where most divergence from traditional browsing happens. A crawler such as OpenAI's GPTBot requests the raw server response — often before any client-side JavaScript has run. If your content only appears after a script executes (common on single-page apps built with React or Vue), many AI crawlers never see it at all.
This is measurable. ChatGPT's fetches are overwhelmingly for HTML rather than scripts: 57.7% of ChatGPT's fetches are for HTML content, while 11.5% are for JavaScript files. Anthropic's crawler behaves differently — 23.84% of Claude's requests are for JavaScript files, according to Higoodie (2026). That gap matters: a site that renders its core content via JavaScript may be legible to Claude's crawler but effectively invisible to a large share of ChatGPT's requests.
Once fetched, the parser strips navigation menus, cookie banners, footers and advertising, keeping the body text. That text is then split into chunks — typically paragraph- or section-sized — and each chunk is embedded as a vector. When a user later asks a question, the AI system retrieves the chunks whose vectors are closest in meaning to the query, a process known as retrieval-augmented generation (RAG). Your page is never read "whole" at answer time; it is read in fragments decided months or years earlier at crawl time.
Which AI bots should UK businesses know about, and how do they differ from Googlebot?
AI crawlers are automated bots operated by AI companies — OpenAI, Anthropic, Google and Microsoft — that fetch web pages either to train models or to retrieve live content for chatbot answers, and they behave differently from Googlebot's indexing crawl. Knowing which bot does what determines how you configure access.
OpenAI runs at least three distinct crawlers under different names: GPTBot (used for model training), OAI-SearchBot (used for ChatGPT's search feature) and ChatGPT-User (triggered live when a user asks ChatGPT to browse a specific page). Each respects robots.txt independently, so blocking one does not block the others. Anthropic's ClaudeBot crawls for Claude's training and retrieval. Google-Extended is a separate directive from standard Googlebot, letting you opt a site out of Gemini and AI Overviews training without affecting normal Google Search indexing.
The traffic shift here has been sharp. GPTBot's share of AI crawler traffic increased from 2.2% to 7.7% between May 2024 and May 2025 — a 305% rise in requests — jumping from #9 to #3 among crawlers, according to Cloudflare Radar (2026).
The economics are also lopsided compared with Googlebot, which typically sends meaningful referral traffic back per crawl. ClaudeBot crawls 20,583 pages for every single referral it sends back to a website, according to a Q1 2026 analysis by TechnologyChecker.io (2026). That asymmetry — heavy crawling, minimal direct referral — is why many site owners question whether to allow AI bots at all, a decision covered further down.
| Crawler | Operator | Purpose | Respects robots.txt separately? |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Yes |
| OAI-SearchBot | OpenAI | ChatGPT search retrieval | Yes |
| ChatGPT-User | OpenAI | Live user-triggered browsing | Yes |
| ClaudeBot | Anthropic | Training and retrieval | Yes |
| Google-Extended | Gemini/AI Overviews training (separate from Googlebot) | Yes |
How do I check which AI crawlers have visited my website, and what does my robots.txt file control?
Server log analysis is the only reliable way to confirm which AI crawlers have accessed your website, since most don't appear in standard analytics tools like Google Analytics, which rely on JavaScript execution that bots skip. Your hosting provider or a CDN such as Cloudflare can export raw access logs showing user-agent strings like "GPTBot" or "ClaudeBot" against timestamped requests.
A robots.txt file is a plain-text file at the root of your domain (yoursite.co.uk/robots.txt) that tells compliant crawlers which parts of your site they may or may not fetch. It is the primary lever most businesses have over AI access, but it is advisory — a bot can technically ignore it, though reputable operators like OpenAI and Anthropic state they comply.
Most sites haven't configured it for AI at all. Only about 14% of 3,816 domains analysed from the top 10,000 had robots.txt directives specifically targeting AI bots, according to Cloudflare Radar (2026). A basic AI-aware robots.txt entry looks like this:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
One increasingly discussed alternative is llms.txt, a proposed Markdown file summarising a site's key pages for LLMs, specified at llmstxt.org. It hasn't gained traction with crawlers in practice. Across an 18 AI-crawler-family, 14-day server log study, crawlers fetched /robots.txt 723 times and /llms.txt zero times, according to Saaslinks (2026). For now, robots.txt remains the file that matters; llms.txt is speculative.
What technical steps make a website more readable to AI, and does structured data actually help?
Server-side rendering, clean heading hierarchy and fast, script-light HTML are the technical foundations that make a website readable to AI crawlers, because most AI bots parse the raw server response rather than executing JavaScript. If your WordPress, Webflow or Shopify theme renders key content via client-side scripts, consider server-side rendering (SSR) or pre-rendering so the first HTML response already contains your text.
Heading structure is not cosmetic for AI retrieval — it's structural. 78.4% of citations tied to a question in an analysis of 3 million ChatGPT responses came from a heading, and 44.2% of citations came from the first 30% of the page, according to Kevin Indig / Growth Memo (2026). That means a clear H1, followed by logically nested H2s and H3s that each state a complete idea, does more for AI visibility than almost any other on-page change.
Schema markup — structured data written in JSON-LD format that describes a page's content to machines using vocabulary from schema.org — is often assumed to be essential for AI citation. The evidence doesn't support that assumption on its own. Ahrefs tested 1,885 pages across Google AI Overviews, AI Mode and ChatGPT and found adding JSON-LD schema produced no statistically significant citation uplift on any platform, according to Ahrefs via AuthorityTech (2026). Schema still helps traditional search features like rich results, and it costs little to implement correctly, but treat it as a supporting technical hygiene measure, not a shortcut to AI citations. Clear writing and heading structure do more work.
What it means for structured data vs plain structure
The practical takeaway: invest first in headings, answer-first paragraphs and clean HTML; treat schema markup as a secondary, low-cost addition rather than the primary lever.
"You can't buy your way in, and that's precisely the opportunity. Assistants reward the brand whose content actually answers the question with checkable specifics. Small firms outrank giants in AI answers every day — the shelf space is earned, not rented." — Lauren Dawkins, Head of Content, Aether AI
What are the most common mistakes UK businesses make that block AI from reading their site?
UK businesses most commonly block AI readability through JavaScript-only rendering, missing or overly broad robots.txt rules, and content buried below the fold with no clear heading structure. Each of these is fixable without a full rebuild, but they compound: a page that's technically fetchable but poorly structured still won't be cited.
- Blanket-blocking all bots out of caution, which also blocks legitimate crawlers like OAI-SearchBot that drive referral traffic, without understanding the distinction between training bots and retrieval bots.
- Relying entirely on client-side JavaScript for core content, text that loads via a script after the initial HTML response, with no server-side rendering fallback.
- Burying the answer several paragraphs deep instead of stating it in the opening two sentences of a section — a direct contradiction of how retrieval systems favour early-page content.
- Treating schema markup as a silver bullet and neglecting heading hierarchy and prose clarity, despite the Ahrefs finding above showing schema alone doesn't move citation rates.
- Never checking server logs, so there's no visibility into whether GPTBot, ClaudeBot or Google-Extended have ever successfully crawled the site.
- Ignoring the ICO's position on data scraping (see below) and assuming any content published is automatically fair game for AI training.
Who is responsible for AI readability, and what are the UK GDPR considerations?
Technical GEO sits across marketing, IT and sometimes legal within a UK business, because it involves content strategy, server configuration and, where AI scraping touches personal data, compliance with UK GDPR as enforced by the Information Commissioner's Office (ICO). In smaller organisations, this responsibility often defaults to whoever manages the website — frequently without the server-log access or crawler-configuration knowledge needed to do it properly.
The data protection angle is often overlooked. If a page contains personal data — staff bios, customer testimonials with full names, case studies — and an AI crawler scrapes and stores that data for model training, the ICO's guidance on web scraping and generative AI treats this as a form of processing that can trigger UK GDPR obligations, including lawful basis and transparency requirements under the Data Protection Act 2018. Businesses publishing personal data should review their privacy notices to reflect that AI crawlers may collect it, and consider robots.txt exclusions for pages containing data subjects who haven't consented to AI training use.
Because this spans content, engineering and compliance, many UK businesses now treat it as a distinct discipline — sometimes called Technical GEO — rather than folding it into either traditional SEO or IT security. Aether AI's own platform is built around that overlap: citation tracking across six AI engines (ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot), robots.txt guidance, and a free AI-visibility audit at aether-ai.co.uk/audit designed for exactly this gap in ownership.
Do-it-yourself vs specialist: how much does AI readability auditing cost, and how long until results show?
Whether to audit AI readability yourself or bring in a specialist depends on technical capacity: checking robots.txt and server logs is free and takes an afternoon, while fixing server-side rendering issues or restructuring content at scale typically needs developer or specialist platform support. There's no universal price point in the UK market yet, since Technical GEO is a new enough discipline that costs vary widely between a one-off audit and an ongoing optimisation retainer.
| Approach | Typical effort | Best suited to |
|---|---|---|
| Manual DIY check | A few hours (robots.txt review, log export) | Small sites, single-person teams |
| Agency audit | One-off project, days to weeks | Businesses needing a specific diagnosis |
| Self-serve GEO platform | Ongoing, automated | Teams wanting continuous citation tracking without hiring |
On timing: because AI models retrain and re-crawl on different schedules, changes to your site don't appear in chatbot answers overnight. A robots.txt fix allowing GPTBot access might be picked up within days, but whether that translates into an actual citation depends on when the underlying model's retrieval index refreshes — which can take weeks. Content restructuring (clearer headings, answer-first paragraphs) tends to show up faster in retrieval-based tools like Perplexity or ChatGPT's search mode than in training-data-dependent model knowledge.
Aether AI's own measured baseline illustrates how rare citations currently are even for established brands: across 700 live AI engine scans of four UK brands' target keywords in July–August 2026, those brands were named in 23 answers — 3.3% — with the longest-established brand, a security services business, named in 6.3% of its 270 scans. That's a useful reference point for setting realistic expectations rather than assuming a quick technical fix guarantees frequent citation.
Your AI readability checklist
- Export server logs and search for GPTBot, ClaudeBot, Google-Extended and OAI-SearchBot user-agent strings.
- Add an AI-specific robots.txt block allowing (or deliberately disallowing) each crawler by name.
- Confirm your core content renders in the raw HTML response, not only after JavaScript executes.
- Restructure key pages so each H2 opens with a direct, complete answer in its first two sentences.
- Add JSON-LD schema as supporting hygiene — don't rely on it as your main citation strategy.
- Review pages containing personal data for UK GDPR exposure before allowing AI training crawlers.
- Re-check server logs monthly to track whether crawler visits increase after changes.
FAQ
How does AI read and understand website content?
AI reads website content by crawling the raw HTML, stripping out navigation and scripts, then splitting the remaining text into chunks that are converted into embeddings for retrieval. It doesn't "read" a page the way a human does in a browser — it processes pre-indexed fragments.
Can AI crawlers read JavaScript-rendered websites?
Some can, but many AI crawlers primarily fetch raw HTML and may miss content that only renders after JavaScript executes. ChatGPT's fetches are 57.7% HTML versus 11.5% JavaScript, while Claude's requests are 23.84% JavaScript, according to Higoodie (2026), so server-side rendering reduces risk across both.
What is the difference between how Google and AI chatbots crawl a website?
Googlebot indexes pages for ranked search results using signals built over two decades, while AI crawlers like GPTBot and ClaudeBot fetch content either for model training or live retrieval-augmented generation answers. Google-Extended is a separate directive from Googlebot specifically for AI training opt-outs.
Does schema markup help AI read and cite your website?
Not on its own, according to current evidence. Ahrefs tested 1,885 pages and found adding JSON-LD schema produced no statistically significant citation uplift across Google AI Overviews, AI Mode or ChatGPT, via AuthorityTech (2026); heading structure and clear prose matter more.
What is llms.txt and do AI crawlers actually use it?
llms.txt is a proposed Markdown file summarising a site's key content for LLMs, specified at llmstxt.org. Current evidence suggests crawlers largely ignore it: a 14-day server log study across 18 crawler families recorded 723 fetches of /robots.txt and zero fetches of /llms.txt, according to Saaslinks (2026).
How do I block or allow AI bots like GPTBot and ClaudeBot on my site?
Add user-agent-specific rules to your robots.txt file at the root of your domain, naming each bot (GPTBot, ClaudeBot, Google-Extended, OAI-SearchBot) separately, since each respects directives independently. Only around 14% of top 10,000 domains currently do this, per Cloudflare Radar (2026), meaning most sites have no explicit AI policy set.
How can I tell if AI crawlers are visiting my website?
Check your raw server or CDN logs for user-agent strings matching known AI bots, since standard analytics tools like Google Analytics typically miss non-JavaScript-executing crawlers. Your hosting provider or a service like Cloudflare can usually export this data directly.
Securing your AI visibility with Aether AI
Everything covered above — crawler access, heading structure, robots.txt configuration and citation tracking — is the exact technical surface Aether AI's platform is built to monitor and improve for UK businesses. Rather than guessing whether GPTBot or ClaudeBot has ever reached your pages, Aether AI tracks citations across six AI engines: ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot.
As a proof point, Aether AI runs its own measured baseline across its founding team's brands — Priority First, Pulse Operations, Aether Agency and aether-ai.co.uk itself — rather than relying on theory alone; in one Google Search Console sample from 19 July–15 August 2026, 82.1% of Priority First's clicks and 88.3% of Aether Agency's clicks came from queries Google doesn't name, underlining how much AI-era search traffic now flows through unattributed, AI-influenced channels.
If you want to see where your own site currently stands, run Aether AI's free AI-visibility audit at aether-ai.co.uk/audit and check public pricing before deciding whether to build this capability in-house.