Technical Access
Can AI Crawlers Actually Reach Your Pages?
A site can rank well in classic search and still be invisible to AI systems if robots.txt blocks their crawlers, JavaScript-only rendering hides content, or redirect chains break the fetch. Answer Patch checks whether AI crawlers like GPTBot, ClaudeBot, and PerplexityBot can actually reach and read your pages, and whether you publish an llms.txt file to make your site deliberately legible to AI.
The audit
What Gets Checked
- robots.txt rules: are AI crawlers like GPTBot, ClaudeBot, and PerplexityBot allowed or blocked?
- llms.txt presence and quality: does the site publish a curated summary for AI systems?
- Server-side rendering: does JavaScript-only rendering hide content from non-browser fetches?
- Response codes and redirects: do crawled pages return clean 200s or break on redirect chains?
- Sitemap availability: can crawlers discover the important pages?
- Per-crawler fetch comparison: do different AI user agents get different responses?
Why it matters
Why Technical Access Matters for AI Search
None of the other pillars matter if AI crawlers can't reach your pages in the first place. A single robots.txt rule blocking GPTBot means ChatGPT's live-fetch crawler — the one that retrieves your page when a user asks a direct question — returns empty-handed.
Answer Patch distinguishes training crawlers from live-fetch crawlers because blocking training is a policy choice, but blocking live fetch means your site can't be cited in a live AI answer at all.
Example findings
What a Scan Might Find
- High
robots.txt blocks ChatGPT-User — live citation fetch is prevented
- Medium
No llms.txt file found at the expected URL
- Medium
Homepage returns only 120 words of server-rendered text — content may be hidden behind JavaScript
- Low
Sitemap discovered but contains zero indexable URLs
Illustrative examples — not from a real scan.
Other audit pillars
See how your site scores on Technical Access
The scan is free. No account, no credit card.
Run a free scan