Fix llms.txt: publish a short Markdown guide at /llms.txt

What the grade checks

• GET https://example.com/llms.txt (and /ai.txt) with a 7-second timeout. It counts when the answer is HTTP 200, text rather than an HTML page from a catch-all route, and not empty.
• Either file earns the llms.txt point of the machine-readable surfaces area. The area adds one point for a sitemap and one for an agent surface (agent-card.json, openapi.json, mcp-registry-auth or x402), capped at 2.

1 of the 2 points in the machine-readable surfaces area. The full scoring rules are on the methodology page.

Why it matters for AI agents and crawlers

The llms.txt proposal, v2 defines the file: an H1 with the site or project name is the only required part, followed by a blockquote summary, optional notes, and H2 sections that list links as [name](url): notes. A section named Optional marks links an agent can skip when it is short of context.

It is past the experiment stage: the proposal notes that Chrome's Lighthouse audits sites for one in its agentic-browsing checks, that documentation platforms generate it, and that OpenAI, Anthropic and Google publish llms.txt files for their own developer docs.

It is not a crawler control. Access is still decided by robots.txt; llms.txt guides an agent that is already allowed in. Point its links at pages that answer questions (pricing, docs, product pages, policies). The v2 proposal also recommends clean Markdown copies of pages (page.md or page.html.md) and a Link: </llms.txt>; rel="describedby" header where you can add one.

How to fix it

1. Write the file

   Start with the name and one sentence a stranger would understand, then list the pages that answer the questions people ask about you. Keep it short enough to read in one go: the detail lives behind the links. The llms.txt generator drafts one from your home page and sitemap.

   llms.txt

   # Example Co
   
   > Example Co makes invoicing software for small agencies: send invoices, track payments and chase late clients automatically. Plans from $12 a month.
   
   Example Co is a web app; there is no desktop client. Support answers within one business day.
   
   ## Product
   
   - [Pricing](https://example.com/pricing): plans, limits and what each includes
   - [Features](https://example.com/features): invoicing, reminders, payment links
   - [Integrations](https://example.com/integrations): Stripe, QuickBooks, Xero
   
   ## Docs
   
   - [Getting started](https://example.com/docs/start.md): create an account and send the first invoice
   - [API reference](https://example.com/docs/api.md): REST endpoints and authentication
   
   ## Optional
   
   - [Blog](https://example.com/blog): product updates and guides
   - [Privacy policy](https://example.com/privacy)

2. Publish it at /llms.txt

   Serve it at the root as text/plain (or text/markdown). What matters is that the answer is the file, not an HTML page.

   Static site

   Save it as llms.txt in the web root. Web servers send .txt files as text/plain.

   WordPress

   Yoast SEO and AIOSEO can generate and maintain the file. To write it by hand, upload llms.txt to the WordPress root folder (the one with wp-config.php) by SFTP or your host's file manager: the web server returns a real file before WordPress routing runs.

   Next.js

   Put it at public/llms.txt, or build it in a route handler so it stays in sync with your content:

   app/llms.txt/route.ts

   export async function GET() {
     const body = `# Example Co
   
   > Invoicing software for small agencies.
   
   ## Product
   
   - [Pricing](https://example.com/pricing): plans and limits
   `
     return new Response(body, {
       headers: { 'Content-Type': 'text/plain; charset=utf-8' },
     })
   }

   Cloudflare

   With Workers static assets or Pages, put llms.txt in the directory you deploy (often public/). From Worker code, return it with a text content type:

   Cloudflare Worker

   const LLMS = `# Example Co
   
   > Invoicing software for small agencies.
   `;
   
   export default {
     async fetch(request, env) {
       if (new URL(request.url).pathname === "/llms.txt") {
         return new Response(LLMS, { headers: { "content-type": "text/plain; charset=utf-8" } });
       }
       return env.ASSETS.fetch(request); // your static assets binding, if the Worker has one
     },
   };

3. Point agents to it (optional)

   The v2 proposal recommends the describedby link relation so a client can find the file from any page, as an HTML tag or an HTTP header.

   HTML head, or response header

   <link rel="describedby" href="/llms.txt">
   
   Link: </llms.txt>; rel="describedby"

Free generator: llms.txt generator. Drafts an llms.txt from a site's home page and sitemap.

How to verify

Check the status and the content type, then read the first lines.

shell

curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -5

200 text/plain and a first line starting with # . If the content type is text/html, a catch-all route is answering and the grade will not count it.

Then re-grade your site: the result lists the evidence for this check.

Questions

Is llms.txt an official standard?

It is a public proposal by Jeremy Howard of Answer.AI, dated 2024-09-03 and revised as v2, not an IETF or W3C standard. It is widely used in practice: Lighthouse audits for it and documentation platforms generate it.

Does llms.txt replace robots.txt or sitemap.xml?

No. robots.txt says what crawlers may access, sitemap.xml lists every indexable page, and llms.txt is a short, curated guide an agent reads when it needs to understand the site. The proposal describes them as complementary.

What goes in the Optional section?

By convention, links an agent can skip when it needs a shorter context: blogs, changelogs, long references, policies.

Can I generate it automatically?

Yes. The llms.txt generator on this site drafts one from your home page and sitemap, and WordPress plugins (Yoast SEO, AIOSEO) and docs platforms (Mintlify, GitBook) generate and maintain one. Edit the summary line by hand either way: it is the line agents quote.

Sources

Primary documentation, read 2026-09-30. Vendors change these pages; follow the link before relying on a detail.

• The /llms.txt file, v2 (llmstxt.org (Jeremy Howard, Answer.AI))
• Lighthouse: llms.txt audit (agentic browsing) (Chrome for Developers)
• Yoast SEO: llms.txt (Yoast)
• AIOSEO: llms.txt generator (AIOSEO)
• Route handlers (route.ts), including non-UI responses (Next.js)
• Workers static assets (Cloudflare)

Related

• robots.txt for AI crawlers: robots.txt groups for eight AI crawler tokens; a blanket Disallow: / counts as blocked.
• sitemap.xml: An XML sitemap at /sitemap.xml, or a Sitemap: line in robots.txt pointing anywhere.
• ai.txt: An alternative to llms.txt for the same point; no crawler vendor documents reading it.
• Generator: llms.txt generator
• All fix guides · All generators · Graded sites · How the grade works