Agent-Readiness Grade / Crawlers / Claude-User
Claude-User
Anthropic · user-triggered fetch. Source: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler, read 2026-09-29. Where the vendor page does not state a fact, this page says "not published".
Facts from the vendor page
| Operator | Anthropic |
|---|---|
| What it is for | User-triggered fetch. Anthropic says that when a person asks Claude a question, Claude may access a website with this agent; disabling it prevents retrieval of your content in response to user queries, which may reduce your visibility in user-directed web search. |
| Published user-agent string | Not published: the vendor page gives the robots.txt token but no full user-agent string. |
| Honors robots.txt | Yes, per the vendor's general statement that Anthropic's bots honor robots.txt directives; the page says Claude-User lets site owners control which sites these user-initiated requests can access, and makes no exception for it. |
| Verification (IP ranges or reverse DNS) | Published IP list at https://claude.com/crawling/bots.json (shared by Anthropic's bots). |
| robots.txt token | Claude-User |
robots.txt lines
To allow Claude-User explicitly (so your policy survives a change to the User-agent: * group):
User-agent: Claude-User Allow: /
To block it:
User-agent: Claude-User Disallow: /
Anthropic documents Claude-User as a user-triggered fetch; treat these lines as a preference, not a guarantee.
What Agent-Readiness Grade checks for Claude-User
Claude-User is not one of the 8 tokens in the crawler-access score, because the vendor documents it as a user-triggered fetch rather than an automatic crawl. What a Claude-User fetch sees is what the readability area measures: the raw HTML of your home page without JavaScript, plus the structured-data area's JSON-LD and Open Graph checks.
Context: in the 30-day census of one Cloudflare zone (2026-08-29 to 2026-09-28), 323 requests carried a user agent matching "Claude-User"; most of them probed scanner paths, and only 18 reached a real endpoint. See the census report.
Other crawlers
GPTBot · OAI-SearchBot · ChatGPT-User · ClaudeBot · PerplexityBot · Google-Extended · Amazonbot · CCBot · Bytespider
Free agent-readiness checkers are a crowded category: Goodie, Frase, aiwebreadiness.com, Arobis, Rank Prompt, Otterly and Writesonic each offer one. Of the checkers we looked at, Agent-Readiness Grade is the only one we found that grades payment and agent-checkout surfaces (/.well-known/x402, agent-card.json, mcp-registry-auth, openapi.json, the MCP endpoint) and legal notices. Statement dated 2026-09-29.