Markdown for AI Crawlers: Does It Help UK Sites in 2026?

Last updated: 5 November 2026

Markdown for AI crawlers means publishing a plain-text, lightly-formatted version of a webpage — using symbols like # and * instead of HTML tags — so that AI systems such as GPTBot or ClaudeBot can read it more cheaply. Cloudflare's own blog post shrank from 16,180 tokens in HTML to 3,150 tokens in markdown, an 80% reduction, according to Cloudflare (via SEOteric) — but Google says it isn't required for AI visibility.

Key Takeaways

• Cloudflare's blog post dropped from 16,180 HTML tokens to 3,150 markdown tokens, an 80% cut, per Cloudflare (via SEOteric).
• A single heading costs around 3 tokens in markdown versus 12-15 tokens in HTML once class names and IDs are counted, according to Cloudflare (via officechai.com).
• Only about 10.13% of roughly 300,000 domains had an llms.txt file, and SE Ranking found no statistically significant link between having one and getting cited by AI.
• Aether AI points to Google's own Search Central documentation, which states that markdown files and llms.txt are not needed for visibility in AI Overviews or AI Mode.
• Only 0.1% of AI bot traffic visited a test site's /llms.txt file over 90 days, performing three times worse than an average content page, according to Otterly.ai.

What does 'markdown for AI crawlers' actually mean?

Markdown for AI crawlers is the practice of offering AI bots — automated programs that fetch web content to train or answer questions for large language models (LLMs) — a lightweight, text-based version of a page instead of full HTML. Markdown is a plain-text formatting syntax: headings use #, bold text uses **, and lists use - or *, with none of the <div>, <span>, class names or inline CSS that make up a rendered HTML page.

The theory is straightforward. AI crawlers such as GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended don't render pages visually — they extract text. Every HTML tag, attribute and bit of styling a crawler has to parse adds tokens (the chunks of text an LLM processes) without adding meaning. Strip that scaffolding away and, in theory, the crawler gets the same information for a fraction of the computational cost.

This isn't a new web standard. It's a voluntary technique some sites — led most visibly by content-delivery network Cloudflare — have started offering through two main routes: dedicated .md URLs that mirror existing pages, or HTTP content negotiation, where the server detects a bot's request and returns markdown instead of HTML automatically. Neither is mandated by any search engine, and that distinction matters for everything that follows.

Why would AI crawlers prefer markdown over HTML?

AI crawlers can process markdown using far fewer tokens than equivalent HTML, which matters because every LLM has a finite context window and a per-token processing cost. Cloudflare found that a simple heading costs roughly 3 tokens in markdown but burns 12-15 tokens in HTML once class names, IDs and attributes are counted, according to Cloudflare (via officechai.com).

That efficiency compounds across a whole page. Cloudflare's own blog post needed 16,180 tokens as rendered HTML but only 3,150 tokens as markdown — an 80% reduction, per Cloudflare (via SEOteric). Separately, a developer reported that stripping pages down to raw markdown cut token usage by roughly 95% compared with full HTML, according to Search Engine Journal, reporting on a John Mueller discussion.

For an AI company running millions of crawl requests a day, that's a real infrastructure saving. For a UK business publishing content, it's a reasonable technical courtesy to offer — but it's worth being precise about what it does and doesn't achieve. Cheaper tokens mean a crawler can afford to read more of your page in one pass. It does not automatically mean the crawler will visit more often, rank your content higher, or cite you more in a generated answer. Those are three separate claims, and the evidence for the first is much stronger than for the other two.

Which AI crawlers currently support or benefit from markdown content?

GPTBot, ClaudeBot, PerplexityBot and Google-Extended are the four crawlers most commonly discussed in this context, but none of them require markdown and none publish a stated preference for it over clean HTML. What's actually documented is narrower: Cloudflare built a content-negotiation system — marketed as "Markdown for Agents" — that detects known AI bot user-agents and serves them a markdown version of a page automatically, rather than relying on the site owner to publish separate .md files.

This is a different mechanism from simply dropping .md files on your server and hoping crawlers find them. Crucially, Google has drawn a hard line on the cloaking question — serving different content to bots than to humans. Google Search Central states plainly: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."

Google's search advocate, John Mueller, went further in a public discussion, reportedly saying: "Converting pages to markdown is such a stupid idea. Did you know LLMs can read images?" He also stated separately: "FWIW no AI system currently uses llms.txt." Those are strong, specific positions from the search engine UK businesses most need to stay in good standing with. The practical read: no major crawler currently requires markdown, and Google actively discourages treating it as an SEO or GEO lever.

How do I convert HTML pages into markdown, and should I serve it alongside or instead of HTML?

Converting a page to markdown means stripping its navigation, scripts, styling and decorative HTML while preserving headings, body text, lists and links in plain-text syntax. Most teams use an automated converter — open-source libraries like Turndown or Pandoc handle the bulk conversion — then manually check that tables, code blocks and any structured data haven't been mangled in the process.

The bigger decision is how to serve it. There are two technical routes, and they carry different risk profiles:

The critical rule, regardless of method: the markdown and HTML versions must say the same thing. Serving a bot a different message, different claims or extra promotional text that human visitors never see is cloaking — a violation of Google's webmaster guidelines that can trigger a manual action. If you publish a .md twin of a page, it must be a faithful text extraction, not a rewritten or enhanced version.

For most UK SMEs, the pragmatic path is simpler still: invest in clean, well-structured HTML with a proper H1/H2/H3 hierarchy first, since that serves human readers, Google, and AI crawlers simultaneously, and only consider a markdown layer once that foundation is solid.

What is llms.txt and should UK businesses bother with one?

An llms.txt file is a proposed plain-text file, placed at a site's root (e.g. yoursite.co.uk/llms.txt), that lists a site's key pages in markdown links so an AI system can supposedly find its most important content quickly. It was proposed by Jeremy Howard of Answer.AI via the specification at llmstxt.org, modelled loosely on how robots.txt or sitemap.xml work — except no search engine or AI company has committed to actually reading it.

The adoption and impact data is now substantial enough to draw a firm conclusion. SE Ranking's analysis of roughly 300,000 domains found only about 10.13% had an llms.txt file, with no statistically significant correlation between having one and being cited by AI — removing the llms.txt variable from their predictive model actually improved its accuracy. SE Ranking's research team concluded it "doesn't seem to directly impact AI citation frequency. At least not yet."

Separately, Otterly.ai's 90-day experiment found only 0.1% of AI bot traffic visited a test site's llms.txt file — three times worse than an average content page. And adoption among the biggest sites on the web is effectively nil: llms.txt uptake among the very largest domains was effectively non-existent, even as overall adoption grew substantially over roughly a year. Google's own documentation confirms: "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files" — a deliberately neutral endorsement, not a recommendation.

For a UK business weighing where to spend limited GEO (generative engine optimisation — the practice of optimising content to be cited in AI-generated answers) budget, llms.txt currently sits low on the priority list.

Will offering markdown improve how my business is represented in AI assistant answers?

The honest answer, based on current independent testing, is: probably not on its own. Profound's markdown-versus-HTML test found a marginal increase in bot traffic to markdown pages, but no conclusive evidence of improved citation rates or accuracy — meaning crawlers might visit slightly more, but that didn't translate into better representation in generated answers.

Otterly.ai's wider experiment reinforces this: publishing markdown versions of pages did not attract additional AI crawlers or earn citations, with zero .md citations recorded across the AI search platforms tested, according to Otterly.ai. What actually drives citation in tools like ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot is the substance of your content — clear definitions, named entities, specific data and well-structured answers — not the markup format it's delivered in.

This is the gap Aether AI was built to close. Rather than chasing unproven technical signals like llms.txt, Aether AI's platform focuses on citation tracking across six AI engines, so businesses can see where they're actually being named — or missed — and fix the content, not just the formatting. Aether AI's own measured baseline across 700 live AI engine scans of UK brands' target keywords in July–August 2026 found the brands were named in just 23 answers, or 3.3% — with the longest-established brand (security services) performing best at 6.3% across its 270 scans, published at aether-ai.co.uk/insights/ai-visibility-baseline-2026. That gap between being crawled and being cited is exactly where real GEO effort belongs.

Are there SEO or technical risks from publishing both HTML and markdown versions?

Yes — the main risks are duplicate content signals, cloaking exposure, and ongoing maintenance drift between two versions of the same page that inevitably fall out of sync. Search engines generally handle near-duplicate content with canonicalisation rather than penalties, but a poorly configured .md endpoint without a proper rel=canonical tag pointing back to the HTML original can confuse indexing signals unnecessarily.

The bigger risk is cloaking, which Google treats as a serious violation: if your markdown version contains different claims, extra keywords, or promotional language the HTML visitor never sees, that's a guideline breach regardless of intent. Google Search Central's position — that it doesn't use markdown files or llms.txt at all for its own ranking or AI Overviews — means the compliance risk of getting the implementation wrong currently outweighs any proven benefit for most sites.

There's also a practical maintenance cost that's easy to underestimate: every time a page's HTML is edited, the markdown twin needs regenerating too, or the two drift apart and the markdown version becomes stale or misleading.

Common mistakes UK businesses make when setting up markdown or llms.txt

The most frequent error is treating markdown and llms.txt as a replacement for good content, rather than a formatting layer on top of content that was already strong. A technically perfect .md file full of thin, generic copy won't out-cite a well-structured HTML page with specific data, named sources and clear definitions.

• Publishing an llms.txt file and expecting a citation lift — the evidence from 300,000 domains shows no measurable correlation, per SE Ranking.
• Letting the markdown version diverge from the HTML page after edits, creating an inconsistency that could read as cloaking.
• Ignoring structured data entirely — schema.org markup normally lives in HTML <head> tags and is simply lost in a stripped markdown file, removing a genuinely useful AI and SEO signal.
• Spending developer time on .md endpoints before fixing basic HTML structure — clean H1/H2/H3 hierarchy and semantic markup benefit every crawler, human and AI, without any cloaking risk.
• Assuming all AI crawlers behave like Cloudflare's customers — most UK businesses aren't running Cloudflare's enterprise content-negotiation feature, so a generic .md file sitting unlinked on a server may simply never be found.
• Treating GEO as a one-off technical project rather than an ongoing content and tracking discipline.

Who should be responsible for markdown and llms.txt implementation?

This sits most naturally with whoever already owns technical SEO, because the decisions involved — content negotiation, canonical tags, crawler user-agent detection — are server-level and SEO-adjacent rather than purely a marketing copy task. In most UK SMEs, that means a developer implementing the technical plumbing in consultation with an in-house marketing or SEO lead who decides which pages, if any, justify the effort.

Pure content marketing teams shouldn't own the technical build, but they should absolutely own the underlying question that matters more: is the content itself clear, well-defined and specific enough to be cited? That's a content and GEO strategy question, not a markup question, and it's where tools like Aether AI's self-service platform are designed to sit — generating AI-optimised articles and tracking whether they're actually being cited, rather than leaving teams to guess based on unproven technical signals.

Your markdown for AI crawlers checklist

• Audit your existing HTML structure first — confirm every page has one H1, a logical H2/H3 hierarchy, and no skipped heading levels.
• Check whether your CMS or hosting already supports content negotiation before building a custom .md endpoint from scratch.
• Never let a markdown version say anything different from its HTML twin — treat divergence as a cloaking risk, not a formatting choice.
• Add rel=canonical tags pointing any .md file back to its HTML original to avoid duplicate content confusion.
• Don't publish an llms.txt file expecting a citation lift — treat it as low priority given the 300,000-domain evidence showing no correlation.
• Preserve or replicate schema.org structured data separately, since it's typically lost in a stripped-down markdown file.
• Track actual AI citations, not crawler visits, across engines like ChatGPT, Perplexity, Claude, Gemini and Copilot to see what's working.
• Prioritise content quality and specific, named facts over technical formatting — this is what the evidence actually supports.

FAQ

Does markdown for AI crawlers actually improve AI citations?

No conclusive evidence currently shows this. Profound's testing found only a marginal increase in bot traffic to markdown pages and no proof of better citation rates, while Otterly.ai recorded zero .md citations across the AI platforms it tested.

Why did John Mueller call markdown for bots a "stupid idea"?

Mueller's publicly reported point was that large language models can already process rich content formats, including images, so stripping pages down to plain text markdown isn't necessary for comprehension. He also stated separately that no AI system currently uses llms.txt, undercutting the case for either technique as an optimisation tactic.

Is serving markdown to AI crawlers considered cloaking?

It can be, if the markdown content differs from what human visitors see on the HTML page. Google's cloaking guidelines apply regardless of which crawler receives the altered content, so any markdown or .md version must be a faithful, equivalent copy of the HTML original to stay compliant.

Do GPTBot, ClaudeBot and PerplexityBot actually request markdown versions of pages?

There's no published evidence that these crawlers specifically request or prefer markdown over well-structured HTML. Cloudflare's "Markdown for Agents" feature detects known AI bot user-agents and serves markdown automatically, but this is Cloudflare's own infrastructure choice, not a requirement stated by OpenAI, Anthropic or Perplexity.

What is llms.txt and do I need one?

An llms.txt file is a proposed root-level text file listing a site's key pages for AI systems to reference, based on the specification at llmstxt.org. Current evidence suggests it isn't necessary: SE Ranking's 300,000-domain study found no statistically significant link to AI citation frequency, and adoption among the top 1,000 domains sits at 0%.

How much should a UK SME budget for markdown or llms.txt implementation?

Given the current evidence, most UK SMEs should spend minimal budget here — a few hours of developer time to ensure clean HTML heading structure delivers more proven benefit than building custom markdown endpoints. Resources are better directed towards content quality and ongoing AI citation tracking.

Are there UK data protection considerations when AI crawlers access site content?

Content published publicly on a website is generally accessible to any crawler that respects (or ignores) your robots.txt file, and the Information Commissioner's Office (ICO) has increasingly scrutinised how AI companies use scraped web data for training. UK businesses should check their own privacy policy and any personal data on public pages before assuming AI crawlers won't process it, and robots.txt directives remain the primary control point for limiting bot access.

Getting your AI visibility right with Aether AI

Deciding whether to build markdown endpoints or an llms.txt file is exactly the kind of technical GEO question that distracts from the bigger issue: most UK businesses have no visibility into whether AI engines are citing them at all. Aether AI's platform was built to answer that question directly, tracking citations across six AI engines — ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot — so businesses can see where they stand before spending developer time on unproven technical signals.

Aether AI itself runs on this approach: its measured baseline of 700 live AI engine scans across UK brands' target keywords found those brands named in just 3.3% of answers overall, with the best-performing brand reaching 6.3%. Aether AI is built and operated by Aether Agency Ltd. Priority First, Pulse Operations and Aether Agency are businesses connected to the founding team, and the same platform and tracking used for them is available to any UK business through self-serve plans.

If you want to see where your own business currently stands before investing in markdown, llms.txt or any other technical GEO tactic, start with Aether AI's free AI-visibility audit or review public pricing to see what ongoing citation tracking looks like.

Related Reading

• AI Crawlers List 2026: GPTBot, ClaudeBot & More
• Do AI Crawlers Execute JavaScript? The 2026 UK Answer
• Do AI Crawlers Render JavaScript? 2026 UK Guide