Agent-Readiness Grade / Crawlers / Bytespider
Bytespider
ByteDance · not published. Source: https://www.bytedance.com/en/, read 2026-09-29. Where the vendor page does not state a fact, this page says "not published".
Facts from the vendor page
| Operator | ByteDance |
|---|---|
| What it is for | Not published. ByteDance's site publishes no documentation for Bytespider: the home page does not mention a crawler and /en/bytespider returns the site's 404 page (checked 2026-09-29). |
| Published user-agent string | Not published. |
| Honors robots.txt | Not published. |
| Verification (IP ranges or reverse DNS) | Not published. |
| robots.txt token | Bytespider |
robots.txt lines
To allow Bytespider explicitly (so your policy survives a change to the User-agent: * group):
User-agent: Bytespider Allow: /
To block it:
User-agent: Bytespider Disallow: /
These lines follow the robots.txt convention for the token; ByteDance publishes no statement that Bytespider reads them.
What Agent-Readiness Grade checks for Bytespider
The crawler-access area parses your robots.txt by user-agent group and reports Bytespider as allowed, disallowed or unmentioned (unmentioned means allowed by default). A group for "Bytespider" wins over the "*" group; "Disallow: /" or "Disallow: /*" without a matching "Allow: /" counts as blocked; narrower path rules are reported as partial. Bytespider is one of the 8 tokens in that score (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, Amazonbot, CCBot, Bytespider).
Context: in the 30-day census of one Cloudflare zone (2026-08-29 to 2026-09-28), 521 requests carried a user agent matching "Bytespider". See the census report.
Other crawlers
GPTBot · OAI-SearchBot · ChatGPT-User · ClaudeBot · Claude-User · PerplexityBot · Google-Extended · Amazonbot · CCBot
Free agent-readiness checkers are a crowded category: Goodie, Frase, aiwebreadiness.com, Arobis, Rank Prompt, Otterly and Writesonic each offer one. Of the checkers we looked at, Agent-Readiness Grade is the only one we found that grades payment and agent-checkout surfaces (/.well-known/x402, agent-card.json, mcp-registry-auth, openapi.json, the MCP endpoint) and legal notices. Statement dated 2026-09-29.