Agent-Readiness Grade / Crawlers / ClaudeBot

ClaudeBot

Anthropic · training. Source: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler, read 2026-09-29. Where the vendor page does not state a fact, this page says "not published".

Facts from the vendor page

Operator Anthropic

What it is for Training crawler. Anthropic says ClaudeBot collects web content that could contribute to training its generative AI models, and that restricting it signals the site's future material should be excluded from those training datasets.

Published user-agent string Not published: the vendor page gives the robots.txt token but no full user-agent string.

Honors robots.txt Yes. Anthropic says its bots honor industry-standard robots.txt directives, support the non-standard Crawl-delay extension, and do not try to bypass CAPTCHAs. It asks that opt-outs be set in robots.txt on every subdomain, and warns that blocking its IP addresses is not a reliable opt-out because it stops the bot from reading robots.txt.

Verification (IP ranges or reverse DNS) Published IP list at https://claude.com/crawling/bots.json; Anthropic says a source address on that list indicates the crawler is theirs.

robots.txt token ClaudeBot

Operator	Anthropic
What it is for	Training crawler. Anthropic says ClaudeBot collects web content that could contribute to training its generative AI models, and that restricting it signals the site's future material should be excluded from those training datasets.
Published user-agent string	Not published: the vendor page gives the robots.txt token but no full user-agent string.
Honors robots.txt	Yes. Anthropic says its bots honor industry-standard robots.txt directives, support the non-standard Crawl-delay extension, and do not try to bypass CAPTCHAs. It asks that opt-outs be set in robots.txt on every subdomain, and warns that blocking its IP addresses is not a reliable opt-out because it stops the bot from reading robots.txt.
Verification (IP ranges or reverse DNS)	Published IP list at https://claude.com/crawling/bots.json; Anthropic says a source address on that list indicates the crawler is theirs.
robots.txt token	`ClaudeBot`

robots.txt lines

To allow ClaudeBot explicitly (so your policy survives a change to the User-agent: * group):

User-agent: ClaudeBot
Allow: /

To block it:

User-agent: ClaudeBot
Disallow: /

What Agent-Readiness Grade checks for ClaudeBot

The crawler-access area parses your robots.txt by user-agent group and reports ClaudeBot as allowed, disallowed or unmentioned (unmentioned means allowed by default). A group for "ClaudeBot" wins over the "*" group; "Disallow: /" or "Disallow: /*" without a matching "Allow: /" counts as blocked; narrower path rules are reported as partial. ClaudeBot is one of the 8 tokens in that score (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, Amazonbot, CCBot, Bytespider).

Context: in the 30-day census of one Cloudflare zone (2026-08-29 to 2026-09-28), 5,568 requests carried a user agent matching "ClaudeBot". See the census report.

Other crawlers

GPTBot · OAI-SearchBot · ChatGPT-User · Claude-User · PerplexityBot · Google-Extended · Amazonbot · CCBot · Bytespider

Free agent-readiness checkers are a crowded category: Goodie, Frase, aiwebreadiness.com, Arobis, Rank Prompt, Otterly and Writesonic each offer one. Of the checkers we looked at, Agent-Readiness Grade is the only one we found that grades payment and agent-checkout surfaces (/.well-known/x402, agent-card.json, mcp-registry-auth, openapi.json, the MCP endpoint) and legal notices. Statement dated 2026-09-29.