Crawler access (robots.txt)
2/2 · pass8 of 8 AI crawlers may read the site, 0 disallowed
- GET /robots.txt -> HTTP 200, 2.4 KB, 8 user-agent groups
- Allowed: GPTBot (11 path rules), OAI-SearchBot (11 path rules)
- Unmentioned (allowed by default): ClaudeBot, PerplexityBot, Google-Extended, Amazonbot, CCBot, Bytespider
- Sitemap declared: https://www.jpost.com/GoogleNewsSiteMap/GNSiteMap, https://www.jpost.com/jpgooglesitemap/sitemapindex.xml, https://www.jpost.com/jpgooglesitemap/latestarticles
Fix: Name the AI crawlers you allow explicitly so your policy survives a future change to the "User-agent: *" group.