Reports · fetched 2026-09-30 · Tranco V349N
The agentic web census, September 2026: how the top 10,000 sites treat AI crawlers and AI assistants
We fetched the files AI crawlers and AI assistants read, on every one of the top 10,000 registrable domains in the Tranco V349N research ranking: robots.txt, llms.txt, llms-full.txt, the A2A agent card, security.txt, mcp.json and the x402 manifest. 5,716 domains answered on the web (the rest are CDN, API and tracking hosts with no website); 4,308 of those serve a robots.txt. Every file was validated, not just checked for a 200: an HTML page at /llms.txt is not an llms.txt.
What we found
- 296 sites tell ChatGPT, Claude and Perplexity not to open their pages even when a user asks. 10.9% block ChatGPT-User, 8.7% block Claude-User and 8.9% block Perplexity-User, mostly through a blanket rule rather than by name (324 sites name Claude-User). Anthropic documents that Claude-User follows robots.txt; OpenAI says robots.txt may not apply to user-initiated ChatGPT requests.
- Training crawlers: 16.8% block GPTBot, 15.8% ClaudeBot, 17.8% Common Crawl's CCBot and 13.7% Google-Extended. 359 sites block GPTBot but allow OAI-SearchBot: no training, yes to ChatGPT search.
- Mixed signals: 30 sites publish an llms.txt (an invitation to language models) while blocking GPTBot or ClaudeBot.
- llms.txt: 730 valid files (12.8% of reachable sites; 115 in the top 1,000). A further 509 sites answer /llms.txt with an HTML page and a 200, which a status-code count would add to adoption. 109 also publish llms-full.txt.
- A2A agent cards: 11 sites publish a valid agent card at /.well-known/agent-card.json or the legacy agent.json. 24 serve an mcp.json and 0 an x402 manifest.
- Content-Signal: 130 robots.txt files (3.0%) carry a Content-Signal line; 146 look like Cloudflare's managed robots.txt. security.txt: 1014 (17.7%).
- Top 100: 17 of 53 with a robots.txt block GPTBot; 10 publish llms.txt; 2 publish an agent card.
Blocked share by AI agent
Share of sites with a robots.txt whose rules disallow the whole site for that agent (its own group, or the * group when it is not named). Partial path blocks are not counted as blocked.
| Agent | Operator and purpose | Top 10,000 | Top 1,000 | Sites naming it |
|---|---|---|---|---|
| Training crawlers | ||||
| GPTBot | OpenAI (training) | 16.8% | 23.4% | 857 |
| ClaudeBot | Anthropic (training) | 15.8% | 23.4% | 730 |
| Google-Extended | Google (Gemini training and grounding control) | 13.7% | 20.2% | 668 |
| CCBot | Common Crawl (open dataset used to train many models) | 17.8% | 26.0% | 721 |
| Bytespider | ByteDance | 16.9% | 25.6% | 617 |
| Applebot-Extended | Apple (AI training control) | 11.9% | 17.3% | 570 |
| meta-externalagent | Meta (AI crawler) | 13.8% | 21.5% | 512 |
| anthropic-ai | Anthropic (legacy token) | 13.2% | 18.5% | 478 |
| cohere-ai | Cohere | 13.3% | 20.3% | 444 |
| Amazonbot | Amazon | 12.5% | 17.9% | 468 |
| Search crawlers | ||||
| OAI-SearchBot | OpenAI (ChatGPT search) | 8.6% | 13.7% | 497 |
| Claude-SearchBot | Anthropic (Claude search) | 8.5% | 14.9% | 327 |
| PerplexityBot | Perplexity (search index) | 12.5% | 19.0% | 632 |
| Googlebot | Google Search | 2.8% | 4.7% | 445 |
| Bingbot | Microsoft Bing (also feeds ChatGPT search) | 3.3% | 5.3% | 348 |
| User-triggered assistants | ||||
| ChatGPT-User | OpenAI (a user asked ChatGPT to open a page) | 10.9% | 15.8% | 587 |
| Claude-User | Anthropic (a user asked Claude to open a page) | 8.7% | 15.6% | 324 |
| Perplexity-User | Perplexity (a user asked Perplexity to open a page) | 8.9% | 16.6% | 310 |
| MistralAI-User | Mistral (user-requested fetch) | 8.3% | 15.4% | 211 |
| DuckAssistBot | DuckDuckGo (DuckAssist answers) | 9.0% | 15.4% | 265 |
Every site with an A2A agent card
- cloudflare.com (#2): Cloudflare Site Agent, 2 skills (agent.json)
- workers.dev (#80): Cloudflare Site Agent, 2 skills (agent.json)
- pages.dev (#234): Cloudflare Site Agent, 2 skills (agent.json)
- forter.com (#276): forter, 25 skills (agent-card.json)
- cookieyes.com (#2698): CookieYes, 0 skills (agent.json)
- medal.tv (#3052): Medal.tv Sales Agent, 0 skills (agent.json)
- monday.com (#3226): monday.com MCP Server, 1 skill (agent-card.json)
- razorpay.com (#3315): Razorpay, 8 skills (agent-card.json)
- opensea.io (#4460): OpenSea, 7 skills (agent-card.json)
- glance.com (#4985): GlanceAI Fashion Agent, 11 skills (agent-card.json)
- withbuddies.com (#5806): Scopely, 0 skills (agent.json)
Mixed signals: llms.txt published, AI crawlers blocked
- gravatar.com (#154): publishes llms.txt, blocks GPTBot, Google-Extended, CCBot, Bytespider, Amazonbot
- unesco.org (#628): publishes llms.txt, blocks GPTBot, ClaudeBot, CCBot, Bytespider, Applebot-Extended, meta-externalagent, Amazonbot
- repubblica.it (#967): publishes llms.txt, blocks GPTBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, anthropic-ai, cohere-ai, DuckAssistBot
- flightradar24.com (#999): publishes llms.txt, blocks GPTBot, CCBot
- jusbrasil.com.br (#1259): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot, ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, DuckAssistBot
- ilmeteo.it (#1395): publishes llms.txt, blocks GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent, cohere-ai, Amazonbot
- oreilly.com (#1933): publishes llms.txt, blocks GPTBot, CCBot, Bytespider, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot
- meetup.com (#1992): publishes llms.txt, blocks GPTBot
- sendbird.com (#2325): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot
- prezi.com (#2915): publishes llms.txt, blocks GPTBot, ClaudeBot
- zalando.de (#3020): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, Bytespider, Applebot-Extended
- loom.com (#3090): publishes llms.txt, blocks GPTBot
- newsbreak.com (#3108): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot, ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, DuckAssistBot
- paypal.me (#3135): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot, ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, DuckAssistBot
- chosun.com (#3673): publishes llms.txt, blocks GPTBot
- wiki.gg (#3813): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot
- coindesk.com (#4025): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot
- vinted.co.uk (#4620): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Amazonbot
- onlamp.com (#4814): publishes llms.txt, blocks GPTBot, CCBot, Bytespider, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot
- turbify.com (#4961): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot
- infobip.com (#5003): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot
- ora.com (#5193): publishes llms.txt, blocks GPTBot, CCBot, Bytespider, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot
- milanuncios.com (#5472): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot
- ziprecruiter.com (#5805): publishes llms.txt, blocks GPTBot, CCBot, Bytespider, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot, Perplexity-User, MistralAI-User, DuckAssistBot
- groupon.com (#5807): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, Amazonbot
- lesoir.be (#6152): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot, ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot
- lavoixdunord.fr (#6274): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot, ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot
- hackernoon.com (#7062): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot, Perplexity-User, MistralAI-User
- oilprice.com (#7101): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, anthropic-ai, cohere-ai, Amazonbot
- rpp.pe (#7323): publishes llms.txt, blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, anthropic-ai, cohere-ai, Amazonbot
Top sites that block all three assistants
facebook.com · gstatic.com · instagram.com · fbcdn.net · twitter.com · googlevideo.com · amazon.com · x.com · pinterest.com · tiktok.com · myfritz.net · msn.com · baidu.com · chatgpt.com · reddit.com · amazonvideo.com · registrar-servers.com · nytimes.com · cnn.com · t.co · meraki.com · imdb.com · weibo.com · skybridge.click · amazon.co.uk · naver.com · amazon.de · demdex.net · sohu.com · kwai.net · quora.com · amazon.co.jp · gosuslugi.ru · amazon.co.za · usatoday.com · threads.com · imgsmail.ru · telegraph.co.uk · amazon.fr · amazon.ca
Content-Signal values seen
search=yes,ai-input=yes,ai-train=yes: 67ai-train=no,search=yes: 57ai-train=no,search=yes,ai-input=yes: 47ai-train=yes,search=yes,ai-input=yes: 42search=yes,ai-input=yes,ai-train=no: 26ai-train=no,search=yes,ai-input=no: 12search=yes,ai-input=no,ai-train=no: 8search=yes,ai-input=yes: 7search=yes,ai-train=no,use=reference: 7search=yes,ai-train=no: 6search=yes,ai-input=yes,ai-train=no,use=reference: 3ai-train=yes,ai-input=yes,search=yes: 3
Look up any site
How this compares
Other counts use different populations and rules. Casey Burridge measured llms.txt at 5.61% of the top 10,000 in June 2026; Rankability counted 8.7% of the top 1,000 the same month; Cloudflare Radar reported Content-Signal on about 4% of the top 200,000 domains in April 2026; API Evangelist found 421 agent cards among 27,840 API providers in its catalog (September 2026) and made the same point about 200s that are not cards. Our counts validate file contents, so they can run lower than status-code counts.
Method and limits
Population: Tranco V349N top 10,000 registrable domains. Fetch: https, following redirects, 8-second timeout, 256 KB cap, user agent agentexchange-census/1.0 (+https://grade.agentexchange.work/methodology), one pass on 2026-09-30; robots.txt retried on www. when the bare domain failed. Robots groups parsed per RFC 9309; a bot is "blocked" when its group (or * if unnamed) disallows / without allowing /. llms.txt counts only non-HTML text that starts with a Markdown heading. Agent cards count only JSON with a name or skills. Limits: one snapshot; sites can serve crawlers different robots.txt files; Cloudflare and other edges can block by network rules the file does not show; "blocked" in robots.txt is a request, not an enforcement. Data: JSON (CC BY 4.0).
Grade your own site How to set robots.txt rules for AI crawlers · llms.txt generator · agent card generator