Crawler access (robots.txt)
2/2 · passNo robots.txt (HTTP 404): every crawler is allowed by default
- GET /robots.txt -> HTTP 404 after 1 redirect(s)
- Unmentioned (allowed by default): GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, Amazonbot, CCBot, Bytespider
Fix: Optional: publish a robots.txt that names the AI crawlers you allow and points to your sitemap; without one your policy is implicit.