substack.com

8 of 8 AI crawlers may read the site, 0 disallowed

• GET /robots.txt -> HTTP 200, 585 B, 4 user-agent groups
• Allowed: GPTBot (via *) (14 path rules), ClaudeBot (via *) (14 path rules), OAI-SearchBot (via *) (14 path rules), PerplexityBot (via *) (14 path rules), Google-Extended (via *) (14 path rules), Amazonbot (via *) (14 path rules), CCBot (via *) (14 path rules), Bytespider (via *) (14 path rules)
• Sitemap declared: https://substack.com/sitemap.xml, https://substack.com/news_sitemap.xml

Fix: Nothing to fix here.

1 of 7 surfaces present (/sitemap.xml)

• /llms.txt: HTTP 429 (the site rate-limited our fetch, even after a retry)
• /sitemap.xml: HTTP 200, sitemap index (905 B)
• /.well-known/x402: HTTP 404
• /.well-known/agent-card.json: HTTP 404
• /.well-known/mcp-registry-auth: HTTP 404
• /openapi.json: HTTP 429 (the site rate-limited our fetch, even after a retry)
• /ai.txt: HTTP 429 (the site rate-limited our fetch, even after a retry)

Fix: Publish /llms.txt (a short Markdown guide to your key pages); one agent surface: /openapi.json, /.well-known/agent-card.json, or /.well-known/x402 if you sell to agents.

Home page is readable

• GET https://substack.com/ -> HTTP 200, text/html, 189 KB
• 1,426 characters of visible text with scripts, styles and tags removed; <title>: "Substack"
• 41 external scripts, 6 inline; <noscript> present
• first words: "Home Subscriptions Chat Activity Explore Profile Create Make money doing the work you believe in Start your Substack Learn more For you Rosi..."

Fix: Nothing to fix here.

No JSON-LD, Open Graph complete

• 0 JSON-LD blocks
• Open Graph: og:title "Substack"; og:description present
• meta description: present

Fix: add a JSON-LD block (Organization or WebSite, plus Product, FAQPage or Article where relevant) so answer engines describe you accurately.

Unavailable

• takeitdown.agentexchange.work/check.json: refused takeitdown.agentexchange.work: hosts on agentexchange.work cannot be graded from inside the same Cloudflare zone; every check would - unavailable

Fix: Re-run in a few minutes; the notice check could not be completed.

Exposes /mcp but is not in the census

• census unavailable (refused index.agentexchange.work: hosts on agentexchange.work cannot be graded from inside the same Cloudflare zone; every check would time )
• GET /mcp -> HTTP 429 (text/plain): an endpoint answers
• Index MCP probe record: refused index.agentexchange.work: hosts on agentexchange.work cannot be graded from inside the same Cloudflare zone; every check would time

Fix: Publish /.well-known/x402 with your paid routes so payment indexers can list you.