Fix readability: make your home page readable without JavaScript

What the grade checks

• GET https://example.com/ (falling back to http) with a 7-second timeout: a plain server-side fetch with no JavaScript and no cookies, user agent AgentReadyBot/1.0, HTML read up to 1.5 MB.
• Scripts, styles and tags are stripped and the visible text is measured. 2/2: at least 1,200 characters (or 300 with two or fewer external scripts) and no empty root container. 0/2: under 300 characters, an empty root container (#root, #app, #__next, #__nuxt and similar) with under 800 characters, a bot challenge, or a non-200 answer. 1/2 otherwise.
• Challenges are recognised by Cloudflare's cf-mitigated: challenge header or by pages such as "Just a moment..." and "Verify you are human".

2 of the 12 points. The full scoring rules are on the methodology page.

Why it matters for AI agents and crawlers

Vercel's analysis of AI crawler traffic (December 2024) found that none of the major AI crawlers rendered JavaScript: OAI-SearchBot, ChatGPT-User and GPTBot from OpenAI, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. Gemini uses Googlebot's infrastructure and does render, as does Applebot.

Google renders JavaScript for Search in a later phase, and still calls server-side or pre-rendering "a great idea" because not all bots can run JavaScript.

Anthropic says its crawlers do not try to bypass CAPTCHAs, so a challenge shown to every non-browser client hides your pages from them.

How to fix it

1. See what a crawler sees

   Fetch the home page without a browser and look for a sentence you expect to be there, then check the status code a non-browser client gets.

   shell

   curl -s -A "Mozilla/5.0 (compatible; AgentReadyBot/1.0; +https://agentexchange.work/)" https://example.com/ | grep -c "a sentence from your home page"
   curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; AgentReadyBot/1.0; +https://agentexchange.work/)" https://example.com/

2. Fix it on your stack

   Put what you do, prices, contact details and links to key pages in the HTML the server sends.

   Static site

   Plain HTML is already readable. If you ship a single-page app (React, Vue or Svelte through Vite, for example), prerender the routes that matter at build time so each HTML file carries its text, or serve a server-rendered landing page.

   WordPress

   Themes render on the server, so the usual culprits are page builders that load sections by script, sliders that hold the only copy of key text, and security plugins that challenge bots. Keep the essentials in ordinary blocks and let verified crawlers through the security plugin.

   Next.js

   App Router pages are Server Components by default. Keep the main content there; do not fetch it in a client component after load, and do not wrap it in dynamic(() => import(...), { ssr: false }).

   app/page.tsx (a Server Component: its text is in the HTML response)

   export default async function Home() {
     const plans = await getPlans() // runs on the server
     return (
       <main>
         <h1>Invoicing for small agencies</h1>
         <p>Send invoices, track payments and chase late clients automatically.</p>
         <ul>
           {plans.map((p) => <li key={p.id}>{p.name}: {p.price}</li>)}
         </ul>
       </main>
     )
   }

   Cloudflare

   If Bot Fight Mode, a WAF rule or AI Crawl Control challenges non-browser clients, crawlers get the challenge instead of your page. Exempt verified crawlers, or at least your public pages, from the challenge (Perplexity documents a WAF allow rule combining user agent and IP ranges), then re-test with curl.

How to verify

Re-run the curl check, then re-grade. The evidence lists the character count, script counts and the first words the fetch saw.

shell

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0 (compatible; AgentReadyBot/1.0; +https://agentexchange.work/)" https://example.com/

200 and a size well above a few kilobytes of real content. A 403 or 503 usually means a challenge.

Then re-grade your site: the result lists the evidence for this check.

Questions

Does Googlebot need server rendering?

Google renders JavaScript for Search, though in a separate, later phase. The AI crawlers in Vercel's analysis did not render at all, so server-rendered HTML serves both.

What counts as an empty root container?

A div, main or section with an id such as root, app, __next, __nuxt, __gatsby or __svelte and nothing inside it (a noscript note inside is ignored). It is the signature of a page assembled entirely in the browser.

My site shows a Cloudflare challenge to bots. Is that a problem?

For AI visibility, yes: the crawler receives the challenge, not your content, and Anthropic says its crawlers do not try to get past CAPTCHAs. Challenge abusive traffic, not every non-browser client.

Sources

Primary documentation, read 2026-09-30. Vendors change these pages; follow the link before relying on a detail.

• The rise of the AI crawler (December 2024) (Vercel)
• Understand the JavaScript SEO basics (Google Search Central)
• Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic)
• Perplexity crawlers (Perplexity)
• AI Crawl Control (Cloudflare)

Related

• Structured data: One JSON-LD block that parses plus og:title and og:description on the home page.
• robots.txt for AI crawlers: robots.txt groups for eight AI crawler tokens; a blanket Disallow: / counts as blocked.
• llms.txt: A Markdown guide to your key pages at /llms.txt, served as text, not as your HTML 404.
• All fix guides · All generators · Graded sites · How the grade works