AI crawlers at the edge of one zone, Aug 29 to Sep 28 2026

Method

Source: Cloudflare GraphQL httpRequestsAdaptiveGroups for the zone agentexchange.work, grouped by user-agent string, for the 30 days from 2026-08-29 to 2026-09-28. The top 100 user agents account for 1,468,345 requests of about 1.66 million in total. Requests were classified by their user-agent string, so every count here is a lower bound for spoofing: a client can claim any name. "AI crawlers and answer engines" counts 15,403 requests within the top-100 sample and 34,052 when the crawler names are matched across all user agents. The zone had 29 subdomains during the window. Three new hosts (inference, tools, memory) created on 2026-09-28 were watched separately for their first 48 hours (2026-09-26T22:40Z to 2026-09-28T22:40Z) to see who finds a hostname nobody has linked to.

Table 1. Who sends the requests (top-100 user agents, 30 days)

Table 2. Named crawlers matched across all user agents (34,052 requests)

Name matching catches every string containing the token, including search and SEO crawlers that were matched alongside the AI ones. The kind column links to the vendor-documented purpose where we have a rule page.

Findings

1. Real crawlers spend their budget on robots.txt and sitemaps

ClaudeBot fetched the store host's three sitemaps 1,029 times and its robots.txt 333 times in the window. OAI-SearchBot fetched robots.txt on 14 subdomains, between 66 and 108 times each. GPTBot's most-fetched content page was try.agentexchange.work/, with 113 requests. With 29 subdomains, most of them without a checkout, the crawl budget is spread across pages that cannot convert a visit into anything.

2. The user-triggered agent strings are spoofed

Requests carrying ChatGPT-User, Claude-User and Perplexity-User had, as their top paths, /.env, /.ssh/id_ed25519, /actuator/... and /gradle.properties, all answered 429 or 404. Only 36 ChatGPT-User hits reached a real page (the root) and 18 Claude-User hits reached the aivisibility host's /mcp. There is no measurable "an assistant fetched our page for a user" traffic in this zone; the strings are worn by vulnerability scanners. OpenAI, Anthropic and Perplexity all publish IP lists for their fetchers (see the rule pages), which is the only way to tell the real ones apart.

3. New hostnames are found within hours, by security scanners and ClaudeBot

Certificate Transparency logs publish every new custom hostname. Within hours of creation, leakix, ForestEngine, TrashHound and ClaudeBot arrived on the new hosts. ClaudeBot crawled both product hosts inside a day with no link pointing at them. The x402-economy monitors (CarbonMonitor, SentinelOracle, mcpbeat, 402explorer) did not find the new hosts; only x402-list-monitor came, because of the one directory submission made.

Table 3. First 48 hours of three new hostnames (created 2026-09-28)

4. AI crawlers are a rounding error next to the agent-economy monitors

Monitors and directory crawlers built around x402 and MCP sent 511,520 requests (34.8% of the sample). They probe liveness and prices; they do not buy: the store issued 567,251 payment challenges in the window and settled 54 payments (0.01%). Our own Sentinel added 289,532 requests (19.7%) verifying a store that sold nothing. On the aivisibility host, the free MCP tool received 56,968 /mcp requests against 221 human /check calls.

Table 4. Named monitors and scripted clients (requests, 30 days)

Limitations

• Classification is by user-agent string only; no IP or reverse-DNS verification was applied, so the AI-crawler counts include spoofed strings and the monitor counts include anything that named itself honestly.
• Table 1 covers the top 100 user agents (1,468,345 of about 1.66 million requests); the long tail is not classified.
• One zone, one operator, thirty days. The zone sells to agents and runs its own probes, which inflates machine traffic relative to a content site.
• The 48-hour discovery measurement covers three hosts created on one day.

What to do with this if you run a site

• Verify user-triggered fetchers by IP list, not by name: ChatGPT-User, Claude-User.
• Keep robots.txt and sitemaps small and correct; they are the pages real crawlers fetch most. Check yours with a free Agent-Readiness Grade.
• Expect a new hostname to be crawled within a day of its certificate being issued, whether or not you link to it.