Agentic web census · AI access lookup · rank #234 · fetched 2026-09-30
Can AI assistants read pages.dev?
pages.dev serves no robots.txt, so it asks no AI agent to stay away. (robots.txt served from www.cloudflare.com)
- Training crawlers blocked: none
- llms.txt: published (plus llms-full.txt)
- A2A agent card: published
- security.txt: published
| AI agent | Operator and purpose | robots.txt verdict |
|---|---|---|
| User-triggered assistants | ||
| ChatGPT-User | OpenAI (a user asked ChatGPT to open a page) | no robots.txt |
| Claude-User | Anthropic (a user asked Claude to open a page) | no robots.txt |
| Perplexity-User | Perplexity (a user asked Perplexity to open a page) | no robots.txt |
| MistralAI-User | Mistral (user-requested fetch) | no robots.txt |
| DuckAssistBot | DuckDuckGo (DuckAssist answers) | no robots.txt |
| Search crawlers | ||
| OAI-SearchBot | OpenAI (ChatGPT search) | no robots.txt |
| Claude-SearchBot | Anthropic (Claude search) | no robots.txt |
| PerplexityBot | Perplexity (search index) | no robots.txt |
| Googlebot | Google Search | no robots.txt |
| Bingbot | Microsoft Bing (also feeds ChatGPT search) | no robots.txt |
| Training crawlers | ||
| GPTBot | OpenAI (training) | no robots.txt |
| ClaudeBot | Anthropic (training) | no robots.txt |
| Google-Extended | Google (Gemini training and grounding control) | no robots.txt |
| CCBot | Common Crawl (open dataset used to train many models) | no robots.txt |
| Bytespider | ByteDance | no robots.txt |
| Applebot-Extended | Apple (AI training control) | no robots.txt |
| meta-externalagent | Meta (AI crawler) | no robots.txt |
| anthropic-ai | Anthropic (legacy token) | no robots.txt |
| cohere-ai | Cohere | no robots.txt |
| Amazonbot | Amazon | no robots.txt |
"Blocked" means the site's robots.txt disallows the whole site for that agent, by name or through the * group. It is a request: OpenAI says robots.txt may not apply to user-initiated ChatGPT requests, and sites can also block agents at the network edge, which robots.txt does not show.
Show it on your site
updates with each census
<a href="https://grade.agentexchange.work/access/pages.dev"><img src="https://grade.agentexchange.work/access/pages.dev.svg" alt="AI access: pages.dev" height="20"></a>
Is this your site?
Get the full agent-readiness grade Change these rules · robots.txt generator for AI crawlers