ai.txt: the grade accepts it, but publish llms.txt first

What the grade checks

• GET https://example.com/ai.txt with a 7-second timeout. It counts when the answer is HTTP 200, text rather than an HTML page, and not empty.
• It earns the same point as /llms.txt, never a second one: one of the two files is enough.

Shares 1 point with llms.txt in the machine-readable surfaces area. The full scoring rules are on the methodology page.

Why it matters for AI agents and crawlers

The crawler documentation we read for these guides (OpenAI, Anthropic, Google, Perplexity, Apple, Amazon, Common Crawl, Meta) documents robots.txt tokens and never mentions ai.txt (checked 2026-09-30).

Spawning, a company that makes AI-training opt-out tools, published an ai.txt proposal for stating AI-training permissions; its pages were under maintenance when we checked on 2026-09-30, so we could not re-read it. A separate academic paper, ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet (May 2025), defines a different format under the same name.

llms.txt has a published format, a Lighthouse audit and generator support in WordPress plugins and docs platforms. robots.txt remains the only file the crawler vendors say they obey.

How to fix it

1. Prefer llms.txt

   If you have neither file, publish llms.txt: it earns the same point, and agents know what to do with it.

2. If you keep ai.txt, make it plain text

   State your AI policy in plain sentences and point to the files that carry the machine rules. Serve it as text/plain at /ai.txt (web root on a static site or WordPress, public/ai.txt in Next.js, your static assets on Cloudflare).

   ai.txt

   # ai.txt for example.com
   # Plain-language AI policy. Crawler rules: https://example.com/robots.txt
   # Site guide for language models: https://example.com/llms.txt
   
   AI search and answer engines may read and cite this site.
   Content on this site may not be used to train AI models.
   Questions: ai-policy@example.com

Free generator: llms.txt generator. Drafts an llms.txt from a site's home page and sitemap.

How to verify

Check the status and content type.

shell

curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/ai.txt

200 text/plain. A text/html answer is your site's catch-all page, which does not count.

Then re-grade your site: the result lists the evidence for this check.

Questions

Do AI crawlers read ai.txt?

None of the eight vendors whose crawler documentation we read mentions it (checked 2026-09-30). Put crawler rules in robots.txt.

Should I publish both ai.txt and llms.txt?

Only if ai.txt says something llms.txt does not. For the grade, either one earns the point.

Is ai.txt the same as llms.txt?

No. llms.txt is a Markdown guide to your pages under a published proposal; the ai.txt proposals are about permissions and share no format.

Sources

Primary documentation, read 2026-09-30. Vendors change these pages; follow the link before relying on a detail.

• The /llms.txt file, v2 (llmstxt.org (Jeremy Howard, Answer.AI))
• ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet (arXiv 2505.07834) (arXiv)
• Overview of OpenAI Crawlers (OpenAI)
• Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic)
• Google's common crawlers (Google)

Related

• llms.txt: A Markdown guide to your key pages at /llms.txt, served as text, not as your HTML 404.
• robots.txt for AI crawlers: robots.txt groups for eight AI crawler tokens; a blanket Disallow: / counts as blocked.
• Generator: llms.txt generator
• All fix guides · All generators · Graded sites · How the grade works