Agent-Readiness Grade / Crawlers
AI crawlers: one page per bot
For each crawler: what the vendor says it is for (training, search or a user-triggered fetch), its published user-agent string, whether it honors robots.txt, how to verify it, the exact robots.txt lines to allow or block it, and what our grader checks. Facts come from the vendor's own documentation, read 2026-09-29; anything the vendor does not state is marked "not published".
| Crawler | Operator | Kind | robots.txt | Token | In the grade |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | Yes. | GPTBot | yes |
| OAI-SearchBot | OpenAI | search | Yes. | OAI-SearchBot | yes |
| ChatGPT-User | OpenAI | user-triggered fetch | May not apply. | ChatGPT-User | no (user-triggered) |
| ClaudeBot | Anthropic | training | Yes. | ClaudeBot | yes |
| Claude-User | Anthropic | user-triggered fetch | Yes, per the vendor's general statement that Anthropic's bots honor robots. | Claude-User | no (user-triggered) |
| PerplexityBot | Perplexity | search | The vendor page recommends allowing PerplexityBot in robots. | PerplexityBot | yes |
| Google-Extended | training and grounding control token | Yes. | Google-Extended | yes | |
| Amazonbot | Amazon | training | Yes. | Amazonbot | yes |
| CCBot | Common Crawl | open web archive | Yes. | CCBot | yes |
| Bytespider | ByteDance | not published | Not published. | Bytespider | yes |
How the grade uses these
The crawler-access area of every grade parses robots.txt for 8 tokens (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, Amazonbot, CCBot, Bytespider) and reports each as allowed, disallowed or unmentioned. The two user-triggered fetchers (ChatGPT-User, Claude-User) are not scored because their vendors document them as acting on a person's request; what they see is measured by the readability area instead.
Free agent-readiness checkers are a crowded category: Goodie, Frase, aiwebreadiness.com, Arobis, Rank Prompt, Otterly and Writesonic each offer one. Of the checkers we looked at, Agent-Readiness Grade is the only one we found that grades payment and agent-checkout surfaces (/.well-known/x402, agent-card.json, mcp-registry-auth, openapi.json, the MCP endpoint) and legal notices. Statement dated 2026-09-29. See the comparisons.