Agentic web census · AI access lookup · rank #4961 · fetched 2026-09-30
Can AI assistants read turbify.com?
turbify.com's robots.txt lets ChatGPT, Claude and Perplexity open pages when a user asks. (robots.txt served from www.turbify.com)
- Training crawlers blocked: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot
- llms.txt: published
- A2A agent card: not published
- security.txt: not published
- Content-Signal:
search=yes, ai-input=yes, ai-train=no - robots.txt looks like Cloudflare's managed file
| AI agent | Operator and purpose | robots.txt verdict |
|---|---|---|
| User-triggered assistants | ||
| ChatGPT-User (named) | OpenAI (a user asked ChatGPT to open a page) | allowed |
| Claude-User (named) | Anthropic (a user asked Claude to open a page) | allowed |
| Perplexity-User | Perplexity (a user asked Perplexity to open a page) | partly blocked |
| MistralAI-User | Mistral (user-requested fetch) | partly blocked |
| DuckAssistBot | DuckDuckGo (DuckAssist answers) | partly blocked |
| Search crawlers | ||
| OAI-SearchBot (named) | OpenAI (ChatGPT search) | allowed |
| Claude-SearchBot (named) | Anthropic (Claude search) | allowed |
| PerplexityBot (named) | Perplexity (search index) | allowed |
| Googlebot (named) | Google Search | allowed |
| Bingbot (named) | Microsoft Bing (also feeds ChatGPT search) | allowed |
| Training crawlers | ||
| GPTBot (named) | OpenAI (training) | blocked |
| ClaudeBot (named) | Anthropic (training) | blocked |
| Google-Extended (named) | Google (Gemini training and grounding control) | blocked |
| CCBot (named) | Common Crawl (open dataset used to train many models) | blocked |
| Bytespider (named) | ByteDance | blocked |
| Applebot-Extended (named) | Apple (AI training control) | allowed |
| meta-externalagent (named) | Meta (AI crawler) | blocked |
| anthropic-ai | Anthropic (legacy token) | partly blocked |
| cohere-ai | Cohere | partly blocked |
| Amazonbot (named) | Amazon | blocked |
"Blocked" means the site's robots.txt disallows the whole site for that agent, by name or through the * group. It is a request: OpenAI says robots.txt may not apply to user-initiated ChatGPT requests, and sites can also block agents at the network edge, which robots.txt does not show.
Show it on your site
updates with each census
<a href="https://grade.agentexchange.work/access/turbify.com"><img src="https://grade.agentexchange.work/access/turbify.com.svg" alt="AI access: turbify.com" height="20"></a>
Is this your site?
Get the full agent-readiness grade Change these rules · robots.txt generator for AI crawlers