Can AI Crawlers Actually Reach Your Pages?
Reviewed August 2026.
On-page HTML is free. Crawler checks run in the Fix Report.
The Answer Patch Technical Access pillar scores on-page HTML on the free scan. The Fix Report adds crawler allow-block for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and PerplexityBot, plus sitemap, llms.txt, and live-fetch. Server-rendered HTML is free; crawler rules ship in the report.
On-page HTML scores on the free scan. A site can rank well in classic search and still be invisible to AI systems if robots.txt blocks the crawlers those systems send, JavaScript-only rendering hides the HTML, or redirect chains break the fetch. The Fix Report adds crawler allow-block, sitemap, llms.txt, and live-fetch comparison, and checks whether the HTML a non-browser client receives is actually readable.
What does the Fix Report check for Technical Access?
- On-page HTML on the free scan. Fix Report Technical Access: crawler allow-block, sitemap, llms.txt, and live-fetch comparison
- robots.txt rules: are AI crawlers like GPTBot, ClaudeBot, and PerplexityBot allowed or blocked?
- llms.txt presence and quality: does the site publish a curated summary for AI systems?
- Server-side rendering: does JavaScript-only rendering hide content from non-browser fetches?
- Response codes and redirects: do crawled pages return clean 200s or break on redirect chains?
- Sitemap availability: can crawlers discover the important pages?
- Per-crawler fetch comparison: do different AI user agents get different responses?
Why does crawler access decide whether AI can cite you?
None of the other pillars matter if AI systems cannot reach the page. ChatGPT, Perplexity, Claude, and Google AI Overviews do not all use one crawler. OpenAI publishes GPTBot for training, OAI-SearchBot for search indexing, and ChatGPT-User for on-demand browsing when a person asks ChatGPT to look at a URL. Anthropic publishes ClaudeBot for training and Claude-User plus Claude-SearchBot for fetch. Perplexity documents PerplexityBot as a search-index crawler for live results, not a foundation-model training crawl. A robots.txt rule that names the wrong user-agent leaves the live citation path open, or closes it by accident.
The Fix Report distinguishes training crawlers from live-fetch crawlers because those are different decisions. Blocking GPTBot is a training-consent choice. Blocking ChatGPT-User means ChatGPT cannot fetch the page when a user asks a live question. The report reads the robots.txt groups the scoring engine actually evaluates, then runs a separate live-fetch comparison that requests the homepage as a smaller set of user-agents. It does not fetch the page as GPTBot, and it does not operate those vendors' crawlers.
How do GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and PerplexityBot differ?
People treat "the ChatGPT bot" as one user-agent. OpenAI documents three. GPTBot crawls to collect training data. OAI-SearchBot crawls to build OpenAI's search index. ChatGPT-User fetches a page in response to a person asking ChatGPT to browse. Anthropic documents ClaudeBot as a training crawler, with Claude-User and Claude-SearchBot as fetch agents. Perplexity documents PerplexityBot as a search-index crawler that is not used to crawl content for AI foundation models; the scoring engine records its purpose as fetch.
The table below is the product roster, not a marketing nickname list. User-agent strings and purposes come from the same crawler table the Fix Report scoring engine uses. Live-fetch comparison is a second check: it requests the homepage as Googlebot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, and Claude-SearchBot. GPTBot and ClaudeBot appear in the robots.txt check. They are not in the live-fetch set, so the report does not request your homepage as GPTBot or ClaudeBot.
A copy-paste Disallow under User-agent: GPTBot is a training opt-out. It is not the same as blocking ChatGPT-User. The reverse is also true: allowing GPTBot does not mean ChatGPT can live-fetch the page if ChatGPT-User is disallowed. Purpose values in the table are training, fetch, or both, the same labels the engine records.
| User-agent | Platform | Purpose | Robots.txt check | Live-fetch comparison |
|---|---|---|---|---|
| GPTBot | ChatGPT / OpenAI | training | Yes | No |
| OAI-SearchBot | OpenAI Search | fetch | Yes | Yes |
| ChatGPT-User | ChatGPT browsing | fetch | Yes | Yes |
| ClaudeBot | Anthropic / Claude | training | Yes | No |
| PerplexityBot | Perplexity AI | fetch | Yes | Yes |
Live-fetch comparison requests the homepage as Googlebot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, and Claude-SearchBot. GPTBot and ClaudeBot are training crawlers: the Fix Report reads robots.txt for them and does not fetch the page as GPTBot or ClaudeBot.
What does a blocked live-fetch crawler look like in practice?
Imagine a local service site that ranks in Google and wants to stay out of model training. The owner pastes a robots.txt snippet from a blog that names GPTBot and ClaudeBot, then Disallow: /. That snippet does what it says: it opts the site out of those training crawlers. It does not mention ChatGPT-User, OAI-SearchBot, or PerplexityBot. A visitor can still ask ChatGPT to open the homepage. ChatGPT-User is allowed, so the live fetch proceeds.
A different site copies a broader block, User-agent groups for GPTBot, ChatGPT-User, and OAI-SearchBot, all Disallow: /, and expects ChatGPT browsing to fail. The Fix Report's robots.txt table would show GPTBot blocked (training) and ChatGPT-User blocked (live fetch) as different rows. The live-fetch comparison would request the homepage as ChatGPT-User and record the response it actually received.
Neither site is a named customer. The pattern is the one the product is built to catch: the user-agent string in robots.txt is the rule, not the brand name on the blog. Training consent and live citation fetch are separate switches. Mixing them up is how a site stays "open to ChatGPT" in conversation and closed to ChatGPT-User in the file that matters.
What is llms.txt, and how does the Fix Report use it?
llms.txt is a proposed plain-text file at the site root that publishes a curated map of useful pages for language models, rather than asking a crawler to infer that map from a full HTML crawl. The convention is documented at llmstxt.org. It is not a replacement for robots.txt, and it is not a sitemap. robots.txt says who may fetch. A sitemap lists URLs for discovery. llms.txt is a short, deliberate summary, typically Markdown, of the pages and facts the publisher wants a model to read first.
The Fix Report checks whether llms.txt and llms-full.txt are present at the expected URLs and whether the file is a usable curated summary rather than an empty stub. Presence alone is not a ranking guarantee. Absence is not a penalty invented by Answer Patch; it is a missed chance to hand AI systems a clean table of contents. If you already publish /llms.txt, the report is the place that confirms a crawler can actually fetch it.
A useful llms.txt names the business in the first lines, links to the pages that state what you do, and skips boilerplate navigation. A file that only says "welcome to our website" does not give a model anything it could not guess from the homepage title. The quality check looks for that curated-summary job, not for a magic keyword list.
Why does JavaScript-only rendering hide a page from AI crawlers?
Most AI crawlers fetch raw HTML. They do not run the JavaScript bundle that fills in a client-rendered app. If the server returns a near-empty shell and the real copy appears only after hydration, a browser user sees a full page and a crawler sees almost nothing. Googlebot can render JavaScript, at a delay and a cost. GPTBot, ChatGPT-User, ClaudeBot, and PerplexityBot are not browser users sitting in front of Chrome.
The Technical Access pillar flags server-side rendering risk when the fetched HTML has too little text or too few headings to be a real page. That on-page check runs on the pages in scope for the audit. Crawler allow-block, sitemap, llms.txt, and live-fetch comparison are Fix Report features on top of it.
Redirect chains and non-200 responses break the same fetch. A 301 to a 302 to a 404 is a page the crawler never reads. Clean 200s with the copy in the HTML are what live-fetch needs. If the live-fetch comparison shows one user-agent getting a full document and another getting a login wall or an empty shell, that difference is evidence, not a vibe.
How does sitemap discovery fit Technical Access?
A crawler that is allowed in robots.txt still has to find the pages that matter. The Fix Report looks for a sitemap the site advertises, usually from robots.txt Sitemap directives, and whether that sitemap lists indexable URLs. An empty sitemap, a sitemap that 404s, or no sitemap at all means discovery depends on following homepage links.
That can be enough for a small site whose service pages are linked from the homepage. It is a problem when the important pages live three clicks down a JavaScript menu the crawler never sees. The report records what it could fetch. It does not promise that every AI system used the same discovery path.
Pair the sitemap check with the robots.txt table. Allowing OAI-SearchBot and PerplexityBot is incomplete if the only URLs they can discover are the homepage and a thin about page. Technical Access is reach plus readability: can the crawler enter, can it find the page, and is the HTML it receives the page you think you published.
What might a Technical Access finding look like?
- High
robots.txt blocks ChatGPT-User, live citation fetch is prevented
- Medium
No llms.txt file found at the expected URL
- Medium
Homepage returns only 120 words of server-rendered text, content may be hidden behind JavaScript
- Low
Sitemap discovered but contains zero indexable URLs
Illustrative examples, not from a real scan.
Where do these claims come from?
Claims on this page cite the sources below. Numbers that are not in those sources are omitted.
Frequently asked questions
What is Technical Access in AI Search Readiness?
Technical Access is one of six Answer Patch audit pillars. On-page HTML scores on the free scan. The Fix Report adds crawler allow-block, sitemap, llms.txt, and live-fetch. A site can rank well in classic search and still be invisible to AI systems if robots.txt blocks their crawlers, JavaScript-only rendering hides content, or redirect chains break the fetch.
Which AI crawlers does Answer Patch check?
The Fix Report's Technical Access pillar checks robots.txt against the same AI crawler roster the scoring engine uses, including GPTBot, ChatGPT-User, and OAI-SearchBot (OpenAI); ClaudeBot, Claude-User, and Claude-SearchBot (Anthropic); and PerplexityBot. It distinguishes training crawlers from live-fetch crawlers. A separate live-fetch comparison then requests the homepage as Googlebot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, and Claude-SearchBot, not as GPTBot or ClaudeBot.
Does Answer Patch tell me if my robots.txt is blocking ChatGPT?
Yes. The Fix Report flags a ChatGPT-User Disallow as blocking the live-fetch path it checks. ChatGPT search citations go through a separate OAI-SearchBot row. OpenAI's bots documentation notes that robots.txt may not apply to ChatGPT-User, so a Disallow is not a guarantee that ChatGPT never fetches the page, and it is not the same as blocking ChatGPT search.
Other audit pillars
Read more on the blog
Also see: Technical SEO, llms.txt
Get crawler and llms.txt checks in a Fix Report
Start with a free homepage score. No account. The $19 Fix Report adds robots.txt, sitemap, llms.txt, and live-fetch comparison.