lobste.rs

0 of 8 AI crawlers may read the site, 8 disallowed

• GET /robots.txt -> HTTP 200, 653 B, 3 user-agent groups
• Disallowed: GPTBot (via *), ClaudeBot (via *), OAI-SearchBot (via *), PerplexityBot (via *), Google-Extended (via *), Amazonbot (via *), CCBot (via *), Bytespider (via *)
• "User-agent: *" disallows the whole site; only crawlers named explicitly may read it
• Content-Signal: ai-input=no, ai-train=no, search=yes
• Sitemap declared: https://lobste.rs/sitemap.xml.gz

Fix: Add a group for GPTBot, ClaudeBot, OAI-SearchBot and the others with "Allow: /" (or drop the blanket "Disallow: /") if you want answer engines to cite you.

None of the 7 machine-readable surfaces is published (a sitemap is declared in robots.txt)

• /llms.txt: HTTP 404
• /sitemap.xml: HTTP 404
• /.well-known/x402: HTTP 404
• /.well-known/agent-card.json: HTTP 404
• /.well-known/mcp-registry-auth: HTTP 404
• /openapi.json: HTTP 404
• /ai.txt: HTTP 404
• Sitemap declared in robots.txt instead: https://lobste.rs/sitemap.xml.gz

Fix: Publish /llms.txt (a short Markdown guide to your key pages); one agent surface: /openapi.json, /.well-known/agent-card.json, or /.well-known/x402 if you sell to agents.

Home page is readable

• GET https://lobste.rs/ -> HTTP 200, text/html, 56 KB
• 3,786 characters of visible text with scripts, styles and tags removed; <title>: "Lobsters"
• 1 external script, 1 inline; <noscript> absent
• first words: "Lobsters Active Recent Comments Search Login 73 Bill Gates tries to install Movie Maker windows techemails.com via nrposner 2026-09-28 11:18..."

Fix: Nothing to fix here.

No JSON-LD, Open Graph complete

• 0 JSON-LD blocks
• Open Graph: og:title "Lobsters"; og:description present
• meta description: absent

Fix: add a JSON-LD block (Organization or WebSite, plus Product, FAQPage or Article where relevant) so answer engines describe you accurately.

0 of 6 required notices complete (desk score 1/12)

• Desk score 1/12 across 6 checks (5 of 12 pages read)
• Missing or partial: TAKE IT DOWN Act notice (0/2); DMCA / copyright notice (0/2); DSA notice-and-action (EU users) (0/2); Privacy request method (1/2); Accessibility statement and feedback (0/2); Marketing opt-out (0/2)

Fix: Publish the missing notices with a working request mechanism. Start with TAKE IT DOWN Act notice: Publish a plain-language notice and a request mechanism usable without an account; act within 48 hours.

Not selling to agents: no census entry, no /mcp endpoint

• Not in the census of 2,031 hosts selling to agents (2026-09-29)
• GET /mcp -> HTTP 404: no MCP endpoint at the root

Fix: Expose a Streamable HTTP MCP endpoint at /mcp (or a paid x402 route) so agents can call you, not just read you.