Agentic web census · fetched 2026-09-30
Can ChatGPT read websites?
Yes, when a user asks it to open a link, unless the site says no. 470 of the 4,308 top-10,000 sites with a robots.txt (10.9%) tell ChatGPT not to open their pages; 296 block ChatGPT, Claude and Perplexity at once. Look up any of the top 10,000 sites for the exact verdict for 20 AI agents: user-triggered assistants, search crawlers and training crawlers (robots.txt fetched 2026-09-30).
Top sites that block all three assistants
facebook.com (#3) · gstatic.com (#4) · instagram.com (#11) · fbcdn.net (#13) · twitter.com (#15) · googlevideo.com (#19) · amazon.com (#26) · x.com (#50) · pinterest.com (#53) · tiktok.com (#54) · myfritz.net (#70) · msn.com (#71) · baidu.com (#81) · chatgpt.com (#82) · reddit.com (#109) · amazonvideo.com (#118) · registrar-servers.com (#122) · nytimes.com (#157) · cnn.com (#206) · t.co (#244) · meraki.com (#251) · imdb.com (#269) · weibo.com (#311) · skybridge.click (#337) · amazon.co.uk (#338) · naver.com (#356) · amazon.de (#400) · demdex.net (#409) · sohu.com (#481) · kwai.net (#486) · quora.com (#496) · amazon.co.jp (#497) · gosuslugi.ru (#510) · amazon.co.za (#525) · usatoday.com (#597) · threads.com (#609) · imgsmail.ru (#612) · telegraph.co.uk (#625) · amazon.fr (#626) · amazon.ca (#645) · sharethrough.com (#648) · primevideo.com (#670) · yelp.com (#676) · vkontakte.ru (#682) · amazon.in (#690) · hihonorcloud.com (#691) · dailymail.com (#698) · dailymail.co.uk (#703) · fb.com (#716) · amazon.es (#723) · pinimg.com (#724) · cnet.com (#729) · amazon.it (#733) · quantserve.com (#739) · creativecdn.com (#750) · crwdcntrl.net (#771) · adform.net (#823) · theverge.com (#854) · daum.net (#855) · amazon.com.br (#864)
Questions people ask
Can ChatGPT read websites?
Yes, when you ask it to open a link: ChatGPT fetches the page with the user agent ChatGPT-User. Whether it may is up to each site's robots.txt. Of 4,308 top-10,000 sites that publish a robots.txt, 470 (10.9%) tell ChatGPT-User to stay out, so ChatGPT can say it cannot access those pages. OpenAI states that robots.txt may not apply to user-initiated requests, so behaviour varies. Pages behind logins, paywalls or heavy JavaScript can also fail to read.
Can ChatGPT read links I paste?
Usually. A pasted link is fetched live by ChatGPT-User. It fails when the site disallows ChatGPT-User in robots.txt, blocks it at the network edge, requires a login, or renders the content only with JavaScript. Look the site up above to see its robots.txt rule.
Which websites block ChatGPT, Claude and Perplexity?
296 of the top 10,000 sites tell all three assistants not to open pages even on a user's request (fetched 2026-09-30). Claude-User is blocked by 373 sites (8.7%), Perplexity-User by 383 (8.9%). The full lists are in the census report.
Can Claude and Perplexity read websites?
Yes, the same way: Claude-User and Perplexity-User fetch a page when a user asks. Anthropic documents that Claude-User follows robots.txt. Most blocks come from a blanket "User-agent: *" rule rather than naming the assistant, so a site that blocks all bots blocks these too.
Source: the agentic web census (method, limits, JSON download). Robots.txt is a request to crawlers, not an enforcement; sites can also block at the network edge.