Last updated: 11 October 2026
Do AI Crawlers Execute JavaScript? What UK Businesses Need to Know
Do AI crawlers execute JavaScript? Largely no. A joint analysis by Vercel and MERJ of over 500 million GPTBot fetches found zero evidence of JavaScript execution, and SearchVIU puts the figure at 69% of AI crawlers unable to render JS at all.
Key Takeaways
- GPTBot does not execute JavaScript, according to a Vercel/MERJ analysis of more than 500 million fetches that found zero rendering evidence (Vercel/MERJ).
- Aether AI notes that ClaudeBot downloads JavaScript files in roughly 23.84% of requests but never executes them, per the same dataset cited via dev.to.
- 69% of AI crawlers can't execute JavaScript, according to dataset analysis published by SearchVIU.
- GPTBot's crawl volume grew by 305% between May 2026 and May 2026, reaching a 30% share among AI crawlers, reports Cloudflare.
- A React or Vue page with no server-rendered HTML is effectively invisible to most AI crawlers, meaning a page can rank on Google while returning a blank body to ChatGPT or Claude.
What is an AI crawler?
An AI crawler is an automated bot that an AI company — such as OpenAI, Anthropic or Perplexity — sends out to fetch web pages, either to train a large language model (LLM) or to answer a live user query in real time. Unlike a human browser, an AI crawler typically requests the raw HTML response from a server and does not run a full browser engine.
The best-known examples include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's AI-training signal, separate from Googlebot), PerplexityBot and PerplexityBot-User (Perplexity), and Bytespider (ByteDance, parent of TikTok). Each serves a different purpose: some crawl purely for model training, others fetch pages live when a user asks a chatbot a question, which is why robots.txt rules need to distinguish between them rather than treating "AI crawlers" as one block.
Crucially, "crawling" here does not imply "seeing the page the way a visitor does." Most of these bots fetch the document that the server returns on first request — before any client-side JavaScript has run. If your content only appears after a script executes in the browser, these crawlers simply never see it.
Which AI crawlers render JavaScript versus reading raw HTML only?
Most named AI crawlers read raw HTML only; very few execute JavaScript the way a browser does. The clearest evidence comes from Vercel and MERJ's "Rise of the AI Crawler" study, which analysed over 500 million GPTBot fetches and found no JavaScript execution whatsoever.
The same research quantifies the gap precisely:
- GPTBot fetches JavaScript files in roughly 11.5% of requests but never executes them — it downloads the script as a text file and moves on, according to the Vercel/MERJ analysis.
- ClaudeBot downloads JavaScript files in approximately 23.84% of requests, again without execution or rendering, per the same study cited via dev.to.
- Across the wider dataset, 69% of AI crawlers can't execute JavaScript, according to SearchVIU's analysis.
This matters because fetching a .js file is not the same as running it. GPTBot and ClaudeBot both retrieve script files at measurable rates — likely to check for structured data, redirects or embedded text — but neither builds a DOM (Document Object Model, the browser's in-memory representation of a page) or executes rendering logic. The practical effect is identical to viewing "View Source" in a browser: you get the server's raw HTML, nothing that JavaScript would have added afterwards.
Growth in crawl volume makes this a live commercial issue, not a theoretical one. Cloudflare's 2026 report found GPTBot's crawl volume surged by 305% between May 2026 and May 2026, reaching a 30% share among AI crawlers, while PerplexityBot recorded a 157,490% increase in raw requests over the same period — the fastest growth of any tracked bot. On Vercel's network, GPTBot generated 569 million requests a month and Claude 370 million, together equivalent to roughly 20% of Googlebot's 4.5 billion monthly requests. More bots are visiting UK sites than ever before — and most of them cannot see your JavaScript.
How does this differ from Googlebot's handling of JavaScript?
Googlebot differs from AI crawlers because it runs a full rendering pipeline built on a recent version of Chromium, Google's open-source browser engine, before indexing a page. Google has used this "evergreen" rendering approach since 2019, meaning Googlebot executes JavaScript, builds the DOM, and indexes the resulting rendered page — not just the initial HTML response.
Cloudflare's 2026 Year in Review confirms the scale gap this creates: Googlebot accounted for more than 25% of all Verified Bot traffic observed by Cloudflare in 2026, dwarfing GPTBot's 3.6% and PerplexityBot's 0.06% share of pages crawled. Google also has over two decades of infrastructure investment behind its rendering queue, including a two-wave indexing process where pages are first indexed on raw HTML, then re-crawled and re-indexed once JavaScript has rendered.
AI crawlers have no equivalent two-wave system publicly documented. If GPTBot or ClaudeBot cannot read your content on the first (and only) pass, there is no second wave to catch what was missed. This is the single biggest structural difference UK marketing teams need to internalise: a page can rank perfectly well on Google, built on years of Googlebot's rendering tolerance, while remaining functionally blank to the AI engines now answering an increasing share of buyer questions.
What happens to client-side rendered content like React, Vue or Angular pages?
Client-side rendering (CSR) is a web architecture where the server sends a near-empty HTML shell and JavaScript builds the actual page content inside the visitor's browser. Frameworks like React, Vue and Angular commonly use this pattern by default, which means the raw HTML an AI crawler receives often contains little more than a <div id="root"> tag and a handful of script references.
When GPTBot, ClaudeBot or PerplexityBot fetch a CSR page, they receive that empty shell — not the rendered article, product description or pricing table a human visitor sees. Given the data above, where zero JavaScript execution was observed across 500 million GPTBot fetches, the practical outcome for a typical CSR site is:
- No body text is extracted — headings, paragraphs, prices, FAQs and product specs injected by JavaScript are invisible.
- No internal links are discovered — if navigation menus or footer links are rendered client-side, the crawler cannot follow them to other pages.
- No structured data is read — Schema.org markup added by a JavaScript library (common in React component libraries) never reaches the crawler.
- The page may still rank on Google — because Googlebot renders JavaScript — creating a false sense of security that the site is "crawlable" generally.
This is precisely the gap Aether AI's technical GEO audits are built to catch: a site performing well in traditional search while being structurally absent from ChatGPT, Perplexity and Gemini answers, because nobody checked what the raw server response actually contains.
How can you test whether an AI crawler is executing JavaScript on your site?
Testing JavaScript execution means comparing what your server sends in its raw response against what a browser renders, then checking your server logs for the specific AI crawler's behaviour. The quickest manual check is to disable JavaScript in your browser (via DevTools) and reload the page — if the main content disappears, an AI crawler relying on raw HTML will see the same emptiness.
A more rigorous approach for a UK business with development resources:
- Pull raw HTML via curl or a similar command-line tool, impersonating the crawler's user-agent string (e.g.
GPTBot/1.0), and inspect whether your key content — headings, body text, prices — appears in the response. - Check server access logs for requests from known AI crawler user-agents and confirm whether they request
.jsfiles at all (GPTBot does so roughly 11.5% of the time, ClaudeBot around 23.84%, per the Vercel/MERJ data) — a near-zero rate of JS requests alongside full page visits is a strong signal of raw-HTML-only crawling. - Use Google Search Console's URL Inspection tool to view the rendered HTML Googlebot produces, then compare it against the server's raw response — any content only present in the rendered version is at risk with non-rendering AI crawlers.
- Audit with a dedicated GEO tool. Aether AI's platform includes a free AI-visibility audit at /audit that checks crawlability and citation readiness across the six major AI engines it tracks — ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot — flagging pages where rendering gaps are likely to be suppressing visibility.
Server-side rendering options for making JavaScript content accessible to AI crawlers
Server-side rendering (SSR) is a technique where the server executes the JavaScript framework's rendering logic before sending the response, so the HTML a crawler receives already contains the full page content. This removes the dependency on the crawler itself being able to run scripts.
The main options for a UK business running a modern JavaScript framework:
| Approach | How it works | Typical fit |
|---|---|---|
| Full SSR (Next.js, Nuxt) | Server renders each page on request, sending complete HTML | New builds or sites with development capacity for framework migration |
| Static Site Generation (SSG) | Pages pre-built as static HTML at deploy time | Marketing sites, blogs, documentation with infrequent content changes |
| Dynamic rendering / prerendering services | A middleware layer detects bot user-agents and serves a pre-rendered HTML snapshot while humans still get the CSR app | Existing CSR sites where a full rebuild isn't feasible short-term |
| Hybrid rendering | Critical content (headings, pricing, FAQs) rendered server-side; interactive widgets remain client-side | E-commerce or SaaS sites balancing interactivity with crawlability |
For most UK marketing and SEO teams, the pragmatic starting point is auditing which pages carry commercially important text — product pages, pricing, FAQs, comparison content — and ensuring those specifically are served as rendered HTML, even if interactive dashboard elements elsewhere remain client-side only.
"Blocking GPTBot to protect your content is like delisting from Google to protect your brochure. Defensible for a paywalled publisher; self-harm for anyone who sells something. If assistants can't read you, they recommend whoever they can read." — Lauren Dawkins, Head of Content, Aether AI
Business risks of exclusion from AI chatbot answers
Exclusion from AI chatbot answers means a business's content never gets surfaced when a prospective customer asks ChatGPT, Perplexity or Gemini a buying-related question — even if that same business ranks highly on Google. Given GPTBot's crawl volume grew 305% year-on-year and PerplexityBot's requests grew by 157,490% over the same period, per Cloudflare, these engines are now processing a UK business's content at meaningfully increasing scale — or failing to, if the content is unreachable.
The commercial exposure is straightforward: if a competitor's pricing page is server-rendered and yours is a React shell with no SSR, an AI assistant answering "which supplier offers X in Manchester" can only cite the competitor — not because of relevance, but because of a rendering gap nobody checked. Marketing budget spent on content that AI crawlers cannot parse produces no citation return in ChatGPT, Perplexity or Copilot, regardless of how well the same content performs in Google's organic results.
This is also a governance issue, not just a technical one. Responsibility for AI-crawler accessibility typically sits across three functions in a UK business: developers own the rendering architecture and robots.txt configuration; the SEO or marketing team identifies which pages carry commercial value and need priority fixing; and leadership needs visibility on citation tracking as a KPI alongside traditional organic rankings. Without a named owner, JavaScript rendering gaps tend to persist for months because they sit between departments.
Your JavaScript-and-AI-crawler checklist
- Identify every JavaScript framework in use across your site (React, Vue, Angular, or a hybrid).
- Test raw HTML output with the crawler's user-agent string (e.g.
GPTBot,ClaudeBot,PerplexityBot) using curl or a browser's "disable JavaScript" view. - Check server logs for the proportion of requests from AI crawlers that include
.jsfile fetches versus page-only fetches. - Prioritise SSR, SSG or dynamic rendering for commercially important pages: pricing, product specs, FAQs, comparison content.
- Review robots.txt to confirm you are not unintentionally blocking GPTBot, ClaudeBot, PerplexityBot or Google-Extended on pages you want cited.
- Run a free AI-visibility audit to benchmark current crawlability across ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot.
- Assign a named owner — developer, SEO lead or marketing manager — accountable for rendering fixes and ongoing monitoring.
FAQ
Do AI crawlers execute JavaScript?
Most do not. A Vercel/MERJ analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution, and SearchVIU reports that 69% of AI crawlers in its dataset cannot execute JavaScript at all.
Does GPTBot render JavaScript?
No, GPTBot does not render JavaScript. It fetches .js files in roughly 11.5% of requests but never executes them, according to the Vercel/MERJ analysis.
Does ClaudeBot execute JavaScript?
No. ClaudeBot downloads JavaScript files in approximately 23.84% of requests but does not execute or render them, per the same dataset cited via dev.to.
Why does my site rank on Google but not appear in ChatGPT or Perplexity answers?
This usually happens because Googlebot renders JavaScript before indexing, while most AI crawlers read only the raw, unrendered HTML response. A page built with client-side rendering can look complete to Google and blank to GPTBot or ClaudeBot.
How can I make my React or Vue site visible to AI crawlers?
Implement server-side rendering, static site generation, or a dynamic rendering layer that serves pre-rendered HTML to bot user-agents. Prioritise your commercially important pages first — pricing, product pages and FAQs — rather than attempting a full-site rebuild immediately.
Does blocking AI crawlers in robots.txt protect my content?
It prevents AI engines from citing you in answers, which is a trade-off rather than pure protection. If your business sells something, being unreadable to ChatGPT or Perplexity typically means a competitor gets recommended instead.
How do I check which AI crawlers are visiting my site?
Review your server access logs for known AI crawler user-agent strings such as GPTBot, ClaudeBot and PerplexityBot, and check whether their requests include JavaScript files or only HTML pages. A free audit tool, such as the one Aether AI offers at /audit, can also benchmark this across six AI engines automatically.
Fixing your crawlability gap with Aether AI
Aether AI exists precisely because the gap between "ranks on Google" and "cited by ChatGPT" is now a measurable, fixable technical problem — not a mystery. The platform's free AI-visibility audit at /audit checks whether your pages are structurally readable to the crawlers that matter, then tracks your actual citation performance across ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot.
Aether AI runs the same engine for its own published content as it does for client brands — across security, facilities software and branding sectors, the platform published 281 articles in the last 30 days, each built to be readable by non-rendering crawlers from the first server response onward. That dogfooding is the proof: if the architecture didn't work on Aether AI's own output, it wouldn't be offered to clients.
If your JavaScript-heavy site is ranking on Google but missing from AI answers, start with the free audit at aether-ai.co.uk/audit to see exactly where the rendering gap sits — then explore Aether AI's public pricing to fix it at scale.