What Is llms.txt? How to Create One for Your Website
llms.txt is the emerging plain-text file AI systems read instead of crawling your whole site. What it is, what goes in it, and how to write your own.
llms.txt is a plain-text Markdown file, served at your website's root path (/llms.txt), that gives AI systems a curated overview of your most important content. Proposed by Jeremy Howard of Answer.AI in September 2024, it works like a table of contents written for large language models. It tells ChatGPT, Claude, Perplexity, and similar systems where your most useful pages are and what each one covers, so they don't have to crawl and parse your entire HTML site to understand it. The format is simple: a heading with your site name, a short summary, and organized sections of links with brief descriptions. The whole file is valid CommonMark Markdown — no XML, no JSON schema, no special tooling required. The specification lives at llmstxt.org. This post covers what goes in an llms.txt file, how it differs from robots.txt, and how to write one for your own site.
Why llms.txt Exists
Web pages are built for people. An HTML page wraps its actual information in navigation menus, cookie banners, ads, footers, and JavaScript that has to execute before content appears. When an AI system reads a page, it gets all of that noise along with the content — and has to figure out which parts matter.
That's workable for a single page. It breaks down at scale. An AI coding assistant trying to understand a framework's API needs to read dozens of documentation pages. A chat assistant answering a question about your product needs to find the right page among hundreds. Context windows — the amount of text a language model can process at once — are large but not unlimited, and every token wasted on HTML boilerplate is a token not spent on your actual content.
llms.txt solves this by giving AI systems a single, curated entry point. Instead of crawling a sitemap and parsing every HTML page, a model can fetch one Markdown file and immediately know what a site is about, which pages matter most, and where to go for specific topics. The idea came directly from Jeremy Howard's work on FastHTML, a Python web framework whose technical documentation served as the reference implementation for the spec.
IDE-based coding agents already use it. Cursor, Windsurf, Claude Code, GitHub Copilot, Cline, and Aider all look for /llms.txt when pointed at a documentation site. A study by SE Ranking analyzing 300,000 domains in November 2025 found that roughly 10% already publish an llms.txt file. Adopters include Anthropic, Stripe, Cloudflare, Vercel, Cursor, Supabase, and Mintlify — mostly developer-facing companies, which tracks with the spec's origins in programming documentation.
llms.txt vs robots.txt
Both files live at the root of your website and guide how automated systems interact with your content. That's where the similarities end. robots.txt has been a web standard since 1994 (formalized as RFC 9309) and controls crawl access — it tells bots what not to fetch. llms.txt does the opposite: it tells AI systems what is most useful and worth reading.
| Aspect | robots.txt | llms.txt |
|---|---|---|
| Created | 1994 | September 2024 |
| Format | Custom directive syntax | CommonMark Markdown |
| Purpose | Tells crawlers what NOT to access | Tells AI what IS most useful |
| Audience | Search engine crawlers (Googlebot, Bingbot) | LLMs and AI agents (ChatGPT, Claude, Perplexity) |
| Function | Access control — block or allow paths | Content curation — highlight and describe key pages |
| Enforcement | Widely respected by search engines | Voluntary — no enforcement mechanism yet |
| Standardization | RFC 9309 | Proposed convention, not ratified by IETF or W3C |
The two files are complementary. A site needs robots.txt to manage crawl access. llms.txt adds a layer on top: once a crawler is allowed in, here's where the good stuff is. One important detail: if your robots.txt blocks AI crawlers like GPTBot or ClaudeBot, those bots will never reach your llms.txt file either. Check your technical access configuration before investing time in llms.txt.
What Goes in an llms.txt File
The specification at llmstxt.org defines a fixed structure. The whole file is valid Markdown, but the sections follow a specific order:
- H1 — your site or project name. This is the only required element.
- Blockquote — a one-to-two-sentence summary of what your site or project does. Optional but strongly recommended.
- Free-form details — paragraphs or lists providing additional context about the project. No headings allowed in this section.
- H2 sections — groups of links organized by topic. Each link follows the format: Page Name: Brief description of the page.
- "Optional" section — an H2 titled exactly "Optional" signals that the links beneath it can be skipped when context space is tight.
The spec explicitly disallows images, HTML, tables, code blocks, and headings beyond H1 and H2. The constraint is deliberate — the simpler the file, the more reliably any parser can read it. It's structured data in Markdown form: machine-readable metadata about your site's most important content.
Example llms.txt File
Here's what an llms.txt file looks like for a hypothetical SaaS product: # Acme Analytics > Acme Analytics is a real-time product analytics platform for SaaS teams. It tracks user behavior, measures feature adoption, and surfaces retention insights. Acme Analytics serves product managers and growth teams at B2B SaaS companies. ## Docs - Getting Started: Installation, SDK setup, and sending your first event - Event Tracking: How to define and send custom events - Dashboards: Creating and sharing real-time dashboards ## API - REST API Reference: Authentication, endpoints, rate limits, and response formats - Webhooks: Configuring event-driven notifications ## Optional - Changelog: Recent product updates - Terms of Service: Legal terms and privacy policy
Notice the pattern: the H1 names the product, the blockquote explains what it does in one sentence, each H2 groups related pages, and every link includes a short description after a colon. The "Optional" section contains pages a model can skip if it's short on context space. The whole file is under 1 KB.
llms.txt vs llms-full.txt
The specification defines two companion files. llms.txt is the index — a curated list of links with descriptions. llms-full.txt is the expanded version — it inlines the full Markdown text of the linked pages so a model can load your content in a single fetch without following any links.
Think of llms.txt as a table of contents and llms-full.txt as the entire book. llms-full.txt makes sense for documentation-heavy sites: API references, developer guides, knowledge bases with dozens of pages. If a model can read your full documentation in one request instead of following dozens of links, it gets better context and gives more accurate answers. The tradeoff is file size — llms-full.txt files can run to hundreds of kilobytes.
For most marketing sites and product pages, llms.txt alone is enough. Add llms-full.txt only when your content is text-heavy enough that inlining the full text genuinely helps a model answer questions about your product. A ten-page marketing site doesn't need it. If you publish both, treat llms.txt as the primary file — it should stand on its own. llms-full.txt is a companion, not a replacement.
How to Create Your llms.txt
Creating an llms.txt file takes about 30 minutes for most sites. Here's the process, step by step.
Step 1: Identify Your Most Important Pages
List the 10 to 30 pages that best represent what your site is and does. Start with the pages you'd want an AI system to cite when someone asks about your product: your homepage, product or service pages, documentation, pricing, FAQ, and key guides. Skip tag pages, paginated archives, and anything that duplicates content found on another page.
Step 2: Write Your Summary
Draft a one-to-two-sentence blockquote that precisely describes your business. This is not a tagline or marketing copy. It's a factual statement of what you do, who you serve, and what distinguishes you. An AI model will use this summary to decide whether your site is relevant to a query — write it the way you'd describe your company to a knowledgeable colleague, not a billboard audience.
Step 3: Organize Links into Sections
Group your pages under H2 headings that match your site's natural structure. Common sections: Docs, Products, API, Guides, Company. Use the heading "Optional" for pages that are helpful but not essential — changelogs, legal notices, press kits. An AI model short on context tokens can skip that entire section.
Step 4: Write Link Descriptions
After each URL, add a colon and a brief note explaining what the page covers. Keep descriptions to one sentence. Be specific — "Pricing for teams of 5 to 500, with annual and monthly options" tells a model more than "Our pricing page." Good descriptions turn llms.txt from a list of URLs into something that helps AI systems surface the right page for the right question — the same principle behind strong answer readiness.
Step 5: Host and Test
Save the file as llms.txt and serve it at your domain root: yourdomain.com/llms.txt. Verify it returns HTTP 200 with the correct content. Confirm that your robots.txt doesn't block the path. Then test it the way an AI would: paste the file into a language model and ask it to summarize your site based on the file alone. If the summary is accurate and complete, the file is working. You can also run a free scan to check whether your llms.txt is detectable and properly formatted.
Step 6: Maintain It
Review the file quarterly. Add new pages, remove deprecated URLs, and update descriptions that no longer match their content. Trigger an off-cycle update after major content launches, URL migrations, or site redesigns — stale links break the file silently. Version-control the file in your repository so changes are tracked.