Last updated: 11 October 2026
Do AI Crawlers Render JavaScript? What UK Businesses Need to Know in 2026
AI crawlers do not render JavaScript. An analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution, and Anthropic's ClaudeBot behaves the same way — both read raw HTML only, which means content injected by React, Vue or Angular is frequently invisible to the bots feeding ChatGPT and Claude.
Key Takeaways
- GPTBot does not execute JavaScript: an analysis of over 500 million fetches found zero evidence of rendering, per Vercel & MERJ.
- ChatGPT's crawler still downloads JavaScript files it cannot run, Aether AI notes: it spends a notable share of fetches on JS files, while Claude spends an even larger share, per the same study.
- Googlebot dwarfs AI crawlers in volume, Aether AI reports: it generated 4.5 billion requests on Vercel's network versus GPTBot's 569 million and Claude's 370 million.
- AI crawler traffic is growing fast: GPTBot's traffic rose 305% between May 2026 and May 2026, against Googlebot's 96% growth, according to Cloudflare's 2026 Year in Review analysis.
- A page can rank on Google and still be invisible to ChatGPT, because the two systems use entirely different rendering pipelines.
What is an AI crawler?
An AI crawler is an automated bot that an AI company deploys to fetch web pages for training data, live retrieval, or answer generation — distinct from a traditional search engine crawler like Googlebot, which indexes pages for ranked search results. The main AI crawlers active on UK websites today are OpenAI's GPTBot (training data collection), OAI-SearchBot and ChatGPT-User (live retrieval for ChatGPT answers), Anthropic's ClaudeBot, Google's Google-Extended (a signal that controls Gemini and AI Overviews training use separately from standard Googlebot indexing), and PerplexityBot, which fetches pages to ground Perplexity's cited answers.
Each crawler identifies itself through a distinct user-agent string, and each is governed by its own robots.txt rules rather than a single blanket "AI bot" directive. OpenAI publishes its crawler behaviour and IP ranges through official documentation, which UK site owners can use to verify genuine requests against spoofed ones in their logs.
Scale varies enormously between them. On Vercel's network, AppleBot generated 314 million fetches in a month, while PerplexityBot generated 24.4 million — a gap of more than tenfold between two crawlers that UK marketers often treat as equally important, per Vercel & MERJ.
Does GPTBot or ClaudeBot execute JavaScript?
No major AI crawler currently executes JavaScript in the way a browser does. The most rigorous evidence comes from Vercel's joint study with MERJ, which analysed over 500 million GPTBot fetches and found zero evidence of JavaScript execution — GPTBot fetches the raw HTML document and stops there.
What makes this counter-intuitive is that these crawlers still request JavaScript files even though they cannot run them. ChatGPT's crawler spends a notable share of its fetches on JavaScript files, while Claude spends an even larger share of its fetches on them, despite neither executing a single line, according to the same Vercel/MERJ dataset. This suggests the bots are either scanning files for embedded text and metadata, or the crawling infrastructure simply hasn't been built to discriminate at the point of fetch.
Claude's behaviour differs from ChatGPT's in another way worth noting for UK ecommerce and media sites: Claude focuses heavily on images, accounting for 35.17% of its total fetches, while ChatGPT prioritises HTML content at 57.70% of fetches, per Vercel & MERJ. That split matters if your product pages lean on JavaScript-rendered galleries rather than static <img> tags with proper alt text.
"Blocking GPTBot to protect your content is like delisting from Google to protect your brochure. Defensible for a paywalled publisher; self-harm for anyone who sells something. If assistants can't read you, they recommend whoever they can read." — Lauren Dawkins, Head of Content, Aether AI
How does Googlebot's JavaScript rendering compare?
Googlebot operates a genuinely different pipeline to AI crawlers: it uses a two-wave crawling system, first indexing the raw HTML, then queuing the page for a headless Chromium renderer (the Web Rendering Service) to execute JavaScript and capture the final DOM. This second wave can take anywhere from seconds to several days depending on crawl budget and site size, which is why JavaScript SEO has been a known discipline in the UK search industry for years.
The volume difference between Google and AI crawlers is stark. Googlebot generated 4.5 billion requests across Vercel's network in a single month, compared with GPTBot's 569 million and Claude's 370 million — meaning Google crawls roughly eight times more than GPTBot and Claude combined, per Vercel & MERJ. Google also appears far more efficient at finding valid pages: ChatGPT spends 34.82% of its fetches on 404 error pages, compared with Googlebot's 8.22%, according to the same research — a sign that GPTBot is working from stale or poorly maintained link graphs.
This gap explains a pattern many UK marketing teams have noticed first-hand: a page can rank well in Google's organic results while remaining completely absent from ChatGPT or Claude's answers, simply because Google rendered the JavaScript and the AI crawler never did.
| Crawler | Executes JavaScript? | Monthly fetch volume (Vercel network) | Primary content focus |
|---|---|---|---|
| Googlebot | Yes, via headless Chromium (two-wave indexing) | 4.5 billion | Full rendered DOM |
| GPTBot (OpenAI) | No — zero evidence found | 569 million | HTML (57.70% of fetches) |
| ClaudeBot (Anthropic) | No | 370 million | Images (35.17% of fetches) |
| AppleBot | Not confirmed by this dataset | 314 million | Not broken out |
| PerplexityBot | No independent rendering confirmed | 24.4 million | Live retrieval for citations |
Source: Vercel & MERJ, 'The Rise of the AI Crawler' (2026)
What happens to client-side rendered content when an AI crawler visits?
Client-side rendering means a browser downloads a mostly empty HTML shell and then builds the visible page using JavaScript frameworks such as React, Vue or Angular — and when an AI crawler fetches that shell without executing the script, it sees only the empty container. No product descriptions, no pricing tables, no FAQ answers, no body copy — just the <div id="root"></div> skeleton and whatever is hardcoded into the initial HTML, typically a header, a footer and a loading spinner.
For a UK business running a modern headless build — common among fintech scale-ups in London, SaaS vendors, and marketplaces — this means the exact content a marketing team wrote to be cited by ChatGPT may never reach the model at all. The growth trajectory makes this increasingly costly to ignore: GPTBot traffic grew 305% between May 2026 and May 2026, while overall crawler traffic across all bots rose 18% in the same period, according to the Cloudflare 2026 Year in Review analysis. A rapidly growing crawler that cannot see your content is a rapidly growing blind spot, not a rapidly growing opportunity.
Aether AI's own operational data illustrates the scale of this gap in practice. Across the four brands it currently writes and publishes for — spanning security, facilities software and branding — Aether AI published 281 articles in a single recent 30-day period, each built on static HTML rather than client-side rendering specifically so that GPTBot, ClaudeBot and PerplexityBot retrieve the full text on first fetch, consistent with the Vercel/MERJ finding that these crawlers do not execute JavaScript at all.
How do I check whether an AI crawler rendered my page correctly?
Checking whether a crawler saw real content or a blank shell starts with viewing the page the way the bot does, not the way a browser does. The fastest method is to fetch the URL with a plain HTTP request — using curl, a browser's "view source" (not "inspect element", which shows the rendered DOM), or a tool like Screaming Frog with JavaScript rendering switched off — and check whether your core text, headings and FAQ content appear in the raw response.
A short practical sequence for UK site owners:
- Run
curl -A "GPTBot" https://yoursite.co.uk/pagefrom a terminal and inspect the output. - Compare it against
view-source:in Chrome, which shows what any non-rendering bot receives. - Search the raw output for your key headings, product names or pricing — if they're missing, a JavaScript framework is hiding them.
- Check whether your FAQ schema or structured data is present in that raw HTML, not just the rendered page.
- Repeat the test on your three or four most commercially important pages, not just the homepage.
If your text appears only after the browser runs JavaScript and is absent from the raw fetch, GPTBot and ClaudeBot almost certainly cannot read it, regardless of how well the page performs in Google's rendered index.
What tools confirm which crawlers are hitting my site?
Server log analysis is the only reliable way to confirm genuine AI crawler visits, because front-end analytics tools like Google Analytics typically run on JavaScript and never fire for bots that don't execute it. Your web server (Nginx, Apache or a CDN such as Cloudflare) logs every request with its user-agent string and originating IP address, and this raw access log is where GPTBot, ClaudeBot, PerplexityBot and Google-Extended actually show up.
To separate real crawlers from spoofed requests pretending to be them, cross-reference the IP address against the official ranges each vendor publishes — OpenAI's are listed in its official crawler documentation. Cloudflare, Fastly and most UK hosting providers offer bot-management dashboards that classify this traffic automatically, and a growing number of UK SEO teams now track AI crawler hits the way they once tracked Googlebot crawl stats in Google Search Console.
This is also where citation-tracking tools add value beyond log files: Aether AI's platform monitors citations across six AI engines — ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot — alongside Google Search Console integration, giving UK teams one view of whether their server is being crawled and whether that crawl is translating into actual citations.
What mistakes do businesses make assuming AI crawlers work like Googlebot?
The most common and costly mistake is assuming that because a page ranks on Google, it must also be visible to ChatGPT and Claude — a false equivalence given that Googlebot renders JavaScript via headless Chromium while GPTBot and ClaudeBot do not, per Vercel & MERJ. Teams that migrated to React or Vue single-page applications years ago for Google's sake, confident their JavaScript SEO was solved, often discover the same architecture quietly excludes them from AI answers.
A second frequent error is treating robots.txt as an afterthought copied from a template, blocking GPTBot or CCBot reflexively without understanding that this also removes the business from retrieval-based answers in ChatGPT and Perplexity, not just training datasets. A third is relying on client-rendered FAQ accordions or tabbed content, assuming all crawlers "see" what a user eventually sees — when in fact only the content present in the initial HTML response reaches a non-rendering bot. A fourth is ignoring the 404 problem highlighted by the Vercel/MERJ data: ChatGPT spends 34.82% of its fetches on 404 pages versus Googlebot's 8.22%, so sitemap hygiene and redirect management matter disproportionately for AI visibility.
Should I use server-side rendering instead of client-side rendering?
Server-side rendering (SSR) or static site generation (SSG) should be the default choice for any UK business that wants AI crawlers to read its content, because both approaches deliver complete HTML on the very first request rather than requiring JavaScript execution afterwards. SSR generates the full page on the server for each request (frameworks like Next.js or Nuxt support this natively), while SSG pre-builds every page as static HTML at deploy time — both guarantee that GPTBot, ClaudeBot and PerplexityBot receive finished content immediately, with no rendering step required.
| Approach | What the AI crawler receives | Best suited to | Typical migration effort |
|---|---|---|---|
| Client-side rendering (CSR) | Empty HTML shell, no body content | Internal dashboards, logged-in apps | N/A — already built |
| Server-side rendering (SSR) | Fully rendered HTML per request | Frequently updated marketing/product pages | Moderate — framework migration, typically weeks |
| Static site generation (SSG) | Pre-built static HTML, fastest to serve | Blogs, documentation, landing pages | Moderate to high, but one-off build |
| Dynamic rendering / prerendering | Static snapshot served to bots, JS to users | Legacy CSR sites that can't fully migrate | Lower — middleware layer, days to weeks |
For an existing JavaScript-heavy site, a full SSR migration is the most disruptive option but the most durable, while dynamic rendering (serving a prerendered snapshot specifically to recognised bot user-agents) is a faster interim fix many UK agencies deploy before a full rebuild. Whichever route is chosen, the underlying goal is the same: ensure the raw HTML response — not the post-JavaScript DOM — contains every fact you want an AI engine to cite.
Your AI crawler visibility checklist
- Confirm whether GPTBot, ClaudeBot and PerplexityBot are hitting your site by checking raw server logs, not JavaScript-dependent analytics.
- Fetch your top five commercial pages with
curlor "view source" and confirm core text appears without JavaScript execution. - Audit your robots.txt file and remove any blanket block on GPTBot or Google-Extended unless you have a deliberate reason to exclude AI training.
- Migrate your most important pages to server-side rendering or static generation rather than leaving them as pure client-side React or Vue.
- Fix broken links and redirect chains, given ChatGPT's crawler spends over a third of its fetches on 404 pages.
- Verify structured data (FAQ schema, product schema) is present in the raw HTML response, not injected afterwards by JavaScript.
- Track citations across ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot to confirm the fix actually produced visibility, not just crawl activity.
FAQ
Do AI crawlers render JavaScript?
No. The most rigorous evidence, from an analysis of over 500 million GPTBot fetches, found zero evidence of JavaScript execution, and ClaudeBot shows the same pattern, per Vercel & MERJ.
Does GPTBot download JavaScript files even though it can't run them?
Yes. ChatGPT's crawler spends a notable share of its fetches on JavaScript files and Claude spends an even larger share, despite neither executing the code.
Why can my page rank on Google but never get cited by ChatGPT?
Because Googlebot renders JavaScript through a headless Chromium pipeline before indexing, while GPTBot and ClaudeBot read only the raw HTML response. If your content is injected by client-side JavaScript, Google sees it and AI crawlers typically do not.
Does Google's AI (Gemini, AI Overviews) have the same rendering limits as ChatGPT and Claude?
Google-Extended governs how Google's own AI products use crawled content, and because it sits alongside Google's existing rendering infrastructure, it is reasonable to expect different behaviour from GPTBot or ClaudeBot — though no entry in the verified research data confirms Google-Extended's exact rendering capability, so this should be tested directly on your own site rather than assumed.
What's the fastest way to test if my JavaScript content is visible to AI crawlers?
Run curl against the page with a crawler user-agent, or use Chrome's "view-source" option, and search the raw output for your key headings and body text. If the content only appears after the page finishes loading in a browser, a non-rendering crawler will not see it.
Should blocking AI crawlers in robots.txt protect my content?
Blocking GPTBot or ClaudeBot in robots.txt removes your pages from that engine's training data and live retrieval, which typically means your business disappears from that engine's generated answers entirely rather than being protected. For most commercial UK sites that want to be recommended by ChatGPT or Claude, allowing these crawlers is the safer default.
Is fixing this my responsibility or my developer's?
It is a shared responsibility: developers control the rendering architecture (SSR, SSG or CSR), while marketing and SEO teams must specify which pages need to be crawler-visible and verify the outcome. Neither side can solve it alone — a developer who doesn't know AI crawlers exist will happily ship a pure React site, and a marketer who doesn't check server logs will never notice the gap.
Getting your site AI-crawler-ready with Aether AI
Every issue covered in this article — JavaScript shells crawlers can't see, robots.txt blocks applied without understanding the consequences, broken links wasting crawl budget, and no visibility into whether GPTBot or ClaudeBot ever actually retrieved a page — is the exact technical gap Aether AI's platform is built to close for UK businesses. Aether AI publishes every article it generates as static, fully rendered HTML from the outset, consistent with the verified finding that GPTBot and ClaudeBot execute no JavaScript at all, so content is retrievable on first fetch rather than dependent on a rendering step that may never happen.
As proof of this approach in practice, Aether AI currently writes and publishes content for four brands spanning security, facilities software and branding, having published 281 articles across them in a single recent 30-day period — all built to be readable by non-rendering crawlers from day one. The platform then tracks whether that visibility converts into actual citations across ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot, with Google Search Console integration layered on top.
If you're unsure whether GPTBot or ClaudeBot can actually read your site's content, Aether AI offers a free AI-visibility audit at /audit — a practical starting point before committing to a rendering migration or a wider GEO strategy.