Last updated: 6 November 2026
PerplexityBot is the web crawler operated by Perplexity AI that fetches page content to power its answer engine's search index and real-time citations. UK site owners typically encounter it in server logs identified by the user-agent string "PerplexityBot", and the decision to allow or block it hinges on crawl volume versus referral value — a ratio that reached roughly 289:1 in July 2026, according to technologychecker.io (via Cloudflare Radar).
Key Takeaways
- PerplexityBot's crawl-to-refer ratio climbed substantially between January 2025 and July 2025, a marked increase, according to Cloudflare Radar.
- By July 2026 the ratio had widened further to approximately 289:1, per technologychecker.io (via Cloudflare Radar).
- One independent log analysis recorded PerplexityBot reading robots.txt daily for 71 days and making 2,517 requests, while Perplexity referred just 16 sessions over the same period — see Sunil Pratap Singh's analysis.
- Perplexity's share of worldwide AI chatbot referral traffic fell from a 12.07% peak in April 2025 to 5.85% in September 2026, according to StatCounter Global Stats.
- Cloudflare has alleged that Perplexity used undeclared crawlers to bypass robots.txt rules, activity it says spanned tens of thousands of domains, per the Cloudflare / ActuIA report.
What is PerplexityBot and what is its purpose?
PerplexityBot is an automated web crawler, a software agent that systematically requests and reads web pages, built by Perplexity AI to index content for its AI-powered answer engine. Perplexity's own documentation describes the crawler as "designed to surface and link websites in search results on Perplexity" rather than for general-purpose model training.
Perplexity maintains a second, related agent called Perplexity-User, which fetches pages in real time when a user's query specifically requires reading a cited page, distinct from PerplexityBot's background indexing crawls. Perplexity has stated in a corporate response that "Perplexity, as an application-layer company, does not train AI models on content. Never has," and its documentation separately states the crawler "is not used to crawl content for AI foundation models."
Context matters here: Perplexity reported 780 million queries in May 2025, growing more than 20% month-on-month at the time according to CEO Aravind Srinivas, as reported by TechCrunch via AEOVision. That scale of query demand is the direct driver behind the crawling volumes UK webmasters now see in their logs.
How can I identify PerplexityBot in my server logs?
PerplexityBot identifies itself in HTTP requests through a user-agent string containing "PerplexityBot", which any standard web server log (Apache, Nginx, IIS) or CDN dashboard (Cloudflare, Fastly, Akamai) will record. Checking your access logs for this string is the first step; Perplexity-User requests carry a separate, related identifier for real-time fetches.
Beyond the user-agent, Perplexity publishes a machine-readable JSON list of IP addresses used by the genuine crawler, updated regularly. Cross-referencing the source IP of any request claiming to be PerplexityBot against this list is the only reliable verification method, because user-agent strings alone can be spoofed by any script.
Practical steps for a UK IT or SEO team:
- Filter server logs by the string "PerplexityBot" over a rolling 30-90 day window.
- Export matching IPs and diff them against Perplexity's published range.
- Note request timestamps and paths crawled to understand which sections of the site draw attention.
- Cross-check against Google Search Console and your CDN's bot-analytics panel, where available.
Community projects such as the ai.robots.txt GitHub repository maintain an up-to-date classification of PerplexityBot alongside other AI crawlers, useful for building firewall rules.
How do I block or allow PerplexityBot using robots.txt?
Robots.txt is a plain-text file placed at a website's root (e.g. example.co.uk/robots.txt) that tells compliant crawlers which paths they may or may not request. To block PerplexityBot specifically, add a dedicated user-agent block rather than relying on a wildcard rule aimed at all bots.
A minimal blocking rule looks like this:
User-agent: PerplexityBot
Disallow: /
To allow it fully, either omit any PerplexityBot-specific block or explicitly permit it:
User-agent: PerplexityBot
Allow: /
To restrict it to certain sections — a common middle ground for UK publishers protecting paywalled or client-confidential content — disallow specific directories:
User-agent: PerplexityBot
Disallow: /account/
Disallow: /internal/
Allow: /
| Approach | Robots.txt effect | Typical use case |
|---|---|---|
| Full block | Disallow: / under PerplexityBot |
Sites wanting zero AI crawler load, or with licensing concerns |
| Full allow | No restrictive rule, or explicit Allow: / |
Sites prioritising visibility in Perplexity's answers |
| Partial block | Disallow specific paths only | Protecting account areas, paywalled content, internal tools |
| No action | Default — most crawlers treated as allowed | Sites unaware of the issue, or deliberately neutral |
Robots.txt is a voluntary standard, not an enforcement mechanism. Perplexity's documentation states it respects robots.txt for PerplexityBot's indexing crawls, but Cloudflare's August 2025 technical report alleged that associated crawling activity continued to "impersonate Google Chrome on macOS when their declared crawler was blocked" — a claim Perplexity has disputed. The practical implication for UK site owners is that robots.txt should be treated as one control layer, not the only one.
Is it legal under UK law for PerplexityBot to crawl my website?
Crawling a public website is not automatically unlawful under UK law, but the legal position depends on copyright, the terms of use published on the site, and — where personal data is involved — UK GDPR obligations. The Copyright, Designs and Patents Act 1988 governs how original content may be reproduced or used, and rights holders retain the ability to object to unauthorised reproduction of substantial copyrighted material, regardless of whether a crawler respects robots.txt.
The Information Commissioner's Office (ICO) is the UK regulator responsible for data protection, and if a crawled page contains personal data — customer reviews naming individuals, staff bios, or user-generated content — UK GDPR principles around lawful basis and data minimisation can apply to how that data is subsequently processed, not just collected. A website's own terms of service can also expressly prohibit automated scraping, which matters in any contractual or civil dispute even where criminal law does not apply.
The Cloudflare-versus-Perplexity dispute, covered by PPC Land, illustrates the live tension here: Cloudflare alleged undeclared crawling bypassing access rules across "tens of thousands of domains", while Perplexity countered that Cloudflare "fundamentally misattributed 3-6M daily requests from BrowserBase's automated browser service to Perplexity". There is no UK court ruling settling AI-crawler legality at the time of writing, which means UK businesses should treat robots.txt, terms of use, and server-side blocking as risk-management tools rather than guaranteed legal protections, and consult legal counsel where licensing revenue or sensitive data is at stake.
Does blocking PerplexityBot affect visibility in Perplexity's answers?
Blocking PerplexityBot via robots.txt generally prevents Perplexity from indexing new content from your site, which can reduce or eliminate the chance of being cited in Perplexity's answers going forward. However, the relationship between being crawled and being cited is weaker than most UK marketing teams assume.
One independent log analysis found that a sample site was crawled by PerplexityBot every day for 71 days, racking up 2,517 requests from Perplexity's published IP ranges, yet Perplexity referred only 16 sessions to that site over the same window, according to Sunil Pratap Singh's analysis. That imbalance mirrors the wider trend Cloudflare has tracked: Perplexity's crawl-to-refer ratio climbed substantially between January 2025 and July 2025, per Cloudflare Radar, and reportedly widened further to around 289:1 by July 2026, per technologychecker.io.
For UK businesses, this means heavy crawling does not guarantee proportionate referral traffic, and blocking the crawler entirely trades away a currently small but non-zero visibility channel. Perplexity's overall share of worldwide AI chatbot referral traffic fell from a 12.07% peak in April 2025 to 5.85% in September 2026, per StatCounter Global Stats — a useful reality check against over-investing defensive effort against a single engine whose referral share is itself shrinking relative to competitors. This is precisely the kind of citation-versus-crawl gap that Aether AI's citation tracking across six AI engines (ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini and Copilot) is built to surface for UK marketing teams deciding where to focus technical GEO effort.
How does PerplexityBot differ from GPTBot, Google-Extended and ClaudeBot?
PerplexityBot is one of several named AI crawlers now operating alongside the long-established search bots like Googlebot and Bingbot, and each is governed by its own robots.txt user-agent token and declared purpose. GPTBot is OpenAI's crawler, primarily associated with gathering data that may inform model training and ChatGPT's browsing features. Google-Extended is a separate control token Google introduced specifically so site owners can opt out of having content used for Gemini and AI Overviews training, independent of standard Googlebot indexing. ClaudeBot is Anthropic's crawler, serving a comparable function for Claude.
| Crawler | Operator | Stated purpose | Robots.txt token |
|---|---|---|---|
| PerplexityBot | Perplexity AI | Indexing for answer citations, not foundation-model training (per Perplexity docs) | PerplexityBot |
| GPTBot | OpenAI | Content gathering associated with model development and browsing | GPTBot |
| Google-Extended | Opt-out control for Gemini/AI Overviews training use | Google-Extended |
|
| ClaudeBot | Anthropic | Crawling to support Claude's training and retrieval | ClaudeBot |
The practical distinction for a UK webmaster is that blocking one of these bots has no effect on the others — each requires its own robots.txt entry, and each operator has made different public statements about whether crawled content feeds model training versus live answer retrieval. The ai.robots.txt community project maintains a consolidated, regularly updated list covering all of these, which is a practical starting point for building a single robots.txt policy that addresses multiple AI crawlers at once rather than managing each individually.
What are the risks to server load from PerplexityBot crawling my site?
Server load and bandwidth cost are legitimate operational concerns given how disproportionate PerplexityBot's crawl volume can be relative to the traffic it sends back. Cloudflare's dataset during its stealth-crawling investigation showed Perplexity-related crawler activity occurring across tens of thousands of domains and millions of requests per day, according to the Cloudflare / ActuIA report, though Perplexity has disputed the attribution of a significant portion of that volume to its own infrastructure.
For a mid-sized UK business site on shared or modestly provisioned hosting, repeated daily crawling — as in the 2,517-request, 71-day pattern logged by Sunil Pratap Singh — can contribute to increased CPU usage, higher CDN bandwidth billing, and in extreme cases degraded page-load times for genuine human visitors during peak crawl windows. Smaller WordPress or Shopify-hosted sites without a CDN layer such as Cloudflare or Fastly in front of them are typically more exposed, because every crawl request hits origin infrastructure directly rather than being served from an edge cache.
Mitigation options include rate-limiting rules at the CDN or web application firewall level, scoping robots.txt disallows to the heaviest or least valuable sections of a site (search results pages, filtered product listings, internal archives), and monitoring server response times during known crawl windows rather than blocking outright.
What should I check to verify a crawler claiming to be PerplexityBot?
Verification requires checking the requesting IP address against Perplexity's official published range, because a user-agent string reading "PerplexityBot" can be typed into any scraping script with no connection to the real service. Perplexity publishes this list as a machine-readable JSON file that is updated periodically, and it is the single authoritative source for confirming genuine requests.
A practical verification checklist for a UK IT or SEO team:
- Extract the source IP from any log entry carrying the PerplexityBot user-agent.
- Fetch the current Perplexity IP range file and compare.
- For CDN-protected sites, configure a firewall rule that only treats requests as genuine PerplexityBot traffic if the IP matches the published range, challenging or blocking all others regardless of user-agent.
- Repeat the IP check periodically, since published ranges can change.
- Cross-reference against community trackers like ai.robots.txt for a second opinion on classification.
This matters because Cloudflare's August 2025 report alleged crawler activity that would "impersonate Google Chrome on macOS when their declared crawler was blocked" — behaviour that, if accurate, would render user-agent-only blocking ineffective and make IP verification the only dependable layer of defence.
Your PerplexityBot management checklist
- Search server logs for the "PerplexityBot" user-agent string across the last 30-90 days.
- Download Perplexity's published IP range file and verify any suspicious requests against it.
- Decide on a policy — full allow, full block, or partial block of sensitive paths — and write it explicitly into robots.txt under a dedicated
User-agent: PerplexityBotblock. - Configure CDN or WAF-level IP verification if server load or spoofing is a concern, rather than relying on robots.txt alone.
- Check Google Search Console and CDN bot-analytics dashboards monthly to track crawl volume trends.
- Assign clear ownership — typically IT or dev-ops for implementation, SEO/marketing for strategy, and legal/compliance for sign-off where licensing or data protection questions arise.
- Reassess the policy quarterly, since crawl-to-refer ratios and market share for AI engines are shifting quickly.
FAQ
What is PerplexityBot used for?
PerplexityBot is Perplexity AI's web crawler, used to index publicly available web pages so Perplexity's answer engine can cite and link to them in response to user queries. Perplexity's own documentation states it is not used to train the company's foundation models.
How do I block PerplexityBot from crawling my site?
Add a dedicated block to your robots.txt file reading User-agent: PerplexityBot followed by Disallow: /, placed at your site's root address (yourdomain.co.uk/robots.txt). For stronger enforcement against spoofed requests, pair this with IP verification against Perplexity's published range at CDN or firewall level.
Does PerplexityBot respect robots.txt?
Perplexity states that PerplexityBot respects robots.txt disallow rules for its indexing crawls, but Cloudflare's 2025 technical report alleged associated stealth-crawling activity that bypassed declared access rules, a claim Perplexity has disputed. UK site owners should treat robots.txt as one layer of control rather than a guaranteed block.
Will blocking PerplexityBot hurt my SEO or AI visibility?
Blocking PerplexityBot specifically affects visibility within Perplexity's own answers, not Google Search rankings, since the two are entirely separate crawlers and indexes. Given that Perplexity's crawl-to-refer ratio reportedly reached around 289:1 by July 2026 according to technologychecker.io, many sites already receive minimal referral traffic relative to crawl volume, so the practical visibility cost of blocking is often smaller than businesses expect.
Is PerplexityBot the same as Perplexity-User?
No. PerplexityBot performs background indexing crawls to build Perplexity's search index, while Perplexity-User fetches a specific page in real time only when a live user query requires reading that page's current content. They carry distinct identifiers in server logs.
How can I tell if PerplexityBot has already crawled my site?
Search your server access logs or CDN bot-analytics dashboard for the "PerplexityBot" string, checking both the frequency and the dates of requests. If your hosting platform does not retain raw logs, a CDN such as Cloudflare typically provides bot-traffic reporting that separates PerplexityBot from other crawlers.
Who should decide whether to allow or block PerplexityBot on a UK business site?
Responsibility is usually shared: the IT or development team implements the technical robots.txt and firewall rules, the SEO or marketing team sets the strategic policy around AI-engine visibility, and legal or compliance input is advisable where the site handles licensed content, personal data, or paywalled material, given the unresolved copyright and data protection questions around AI crawling.
Managing AI crawler visibility with Aether AI
Deciding whether to allow, restrict or block PerplexityBot is only useful once you can see whether that crawling is actually translating into citations — and most UK businesses have no visibility into that gap at all. Aether AI's citation tracking monitors how and where a brand is actually named across six AI engines, including Perplexity, ChatGPT, Google AI Overviews, Claude, Gemini and Copilot, so technical GEO decisions like robots.txt policy can be based on evidence rather than guesswork.
As a concrete reference point, Aether AI's own measured baseline — 700 live AI engine scans of UK brands' target keywords run July–August 2026 — found those brands named in only 23 answers, or 3.3% of scans, with the longest-established brand appearing in 6.3% of its 270 scans. That kind of low, measurable citation rate is exactly what a crawl-to-refer ratio in the hundreds-to-one range, as tracked by Cloudflare Radar, would predict, and it's the reason crawler access decisions should sit alongside citation monitoring rather than being made in isolation.
If you want to see where your own site currently stands with AI engines before deciding how to treat PerplexityBot and similar crawlers, Aether AI's free AI-visibility audit at /audit is the starting point, with public pricing available for teams ready to track citations and manage technical GEO on an ongoing basis.