Agent-Readiness Grade / Crawlers / CCBot
CCBot
Common Crawl · open web archive. Source: https://commoncrawl.org/ccbot, read 2026-09-29. Where the vendor page does not state a fact, this page says "not published".
Facts from the vendor page
| Operator | Common Crawl |
|---|---|
| What it is for | Open-archive crawler. Common Crawl, a non-profit, uses CCBot to build a free, open repository of web crawl data that anyone can analyze. The vendor page does not describe a search product or model training of its own. |
| Published user-agent string | CCBot/2.0 (https://commoncrawl.org/faq/)Full string as published by the vendor. |
| Honors robots.txt | Yes. The vendor page gives the robots.txt group that stops Common Crawl from crawling a site (User-agent: CCBot, Disallow: /). |
| Verification (IP ranges or reverse DNS) | Dedicated IP ranges with reverse DNS under crawl.commoncrawl.org (not yet over IPv6), published as JSON at https://index.commoncrawl.org/ccbot.json. Common Crawl warns that other crawlers falsely identify as CCBot. |
| robots.txt token | CCBot |
robots.txt lines
To allow CCBot explicitly (so your policy survives a change to the User-agent: * group):
User-agent: CCBot Allow: /
To block it:
User-agent: CCBot Disallow: /
What Agent-Readiness Grade checks for CCBot
The crawler-access area parses your robots.txt by user-agent group and reports CCBot as allowed, disallowed or unmentioned (unmentioned means allowed by default). A group for "CCBot" wins over the "*" group; "Disallow: /" or "Disallow: /*" without a matching "Allow: /" counts as blocked; narrower path rules are reported as partial. CCBot is one of the 8 tokens in that score (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, Amazonbot, CCBot, Bytespider).
Context: in the 30-day census of one Cloudflare zone (2026-08-29 to 2026-09-28), 950 requests carried a user agent matching "CCBot". See the census report.
Other crawlers
GPTBot · OAI-SearchBot · ChatGPT-User · ClaudeBot · Claude-User · PerplexityBot · Google-Extended · Amazonbot · Bytespider
Free agent-readiness checkers are a crowded category: Goodie, Frase, aiwebreadiness.com, Arobis, Rank Prompt, Otterly and Writesonic each offer one. Of the checkers we looked at, Agent-Readiness Grade is the only one we found that grades payment and agent-checkout surfaces (/.well-known/x402, agent-card.json, mcp-registry-auth, openapi.json, the MCP endpoint) and legal notices. Statement dated 2026-09-29.