# Fix readability: make your home page readable without JavaScript > An AI crawler reads the HTML your server sends and nothing else. If your home page is an empty `
` filled in by JavaScript, or a bot challenge, the crawler sees no text and an answer engine has nothing to quote. Checked 2026-09-30 against Agent-Readiness Grade 1.3.0. HTML version: https://grade.agentexchange.work/fix/readable-without-javascript ## What the grade checks - `GET https://example.com/` (falling back to http) with a 7-second timeout: a plain server-side fetch with no JavaScript and no cookies, user agent `AgentReadyBot/1.0`, HTML read up to 1.5 MB. - Scripts, styles and tags are stripped and the visible text is measured. **2/2**: at least 1,200 characters (or 300 with two or fewer external scripts) and no empty root container. **0/2**: under 300 characters, an empty root container (`#root`, `#app`, `#__next`, `#__nuxt` and similar) with under 800 characters, a bot challenge, or a non-200 answer. **1/2** otherwise. - Challenges are recognised by Cloudflare's `cf-mitigated: challenge` header or by pages such as "Just a moment..." and "Verify you are human". 2 of the 12 points. ## Why it matters for AI agents and crawlers [Vercel's analysis of AI crawler traffic](https://vercel.com/blog/the-rise-of-the-ai-crawler) (December 2024) found that none of the major AI crawlers rendered JavaScript: OAI-SearchBot, ChatGPT-User and GPTBot from OpenAI, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. Gemini uses Googlebot's infrastructure and does render, as does Applebot. [Google](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) renders JavaScript for Search in a later phase, and still calls server-side or pre-rendering "a great idea" because not all bots can run JavaScript. [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) says its crawlers do not try to bypass CAPTCHAs, so a challenge shown to every non-browser client hides your pages from them. ## How to fix it ### 1. See what a crawler sees Fetch the home page without a browser and look for a sentence you expect to be there, then check the status code a non-browser client gets. shell: ```sh curl -s -A "Mozilla/5.0 (compatible; AgentReadyBot/1.0; +https://agentexchange.work/)" https://example.com/ | grep -c "a sentence from your home page" curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; AgentReadyBot/1.0; +https://agentexchange.work/)" https://example.com/ ``` ### 2. Fix it on your stack Put what you do, prices, contact details and links to key pages in the HTML the server sends. #### Static site Plain HTML is already readable. If you ship a single-page app (React, Vue or Svelte through Vite, for example), prerender the routes that matter at build time so each HTML file carries its text, or serve a server-rendered landing page. #### WordPress Themes render on the server, so the usual culprits are page builders that load sections by script, sliders that hold the only copy of key text, and security plugins that challenge bots. Keep the essentials in ordinary blocks and let verified crawlers through the security plugin. #### Next.js App Router pages are Server Components by default. Keep the main content there; do not fetch it in a client component after load, and do not wrap it in `dynamic(() => import(...), { ssr: false })`. app/page.tsx (a Server Component: its text is in the HTML response): ```tsx export default async function Home() { const plans = await getPlans() // runs on the server return (

Invoicing for small agencies

Send invoices, track payments and chase late clients automatically.

) } ``` #### Cloudflare If Bot Fight Mode, a WAF rule or [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) challenges non-browser clients, crawlers get the challenge instead of your page. Exempt verified crawlers, or at least your public pages, from the challenge ([Perplexity documents a WAF allow rule](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) combining user agent and IP ranges), then re-test with curl. ## How to verify Re-run the curl check, then re-grade. The evidence lists the character count, script counts and the first words the fetch saw. ```sh curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0 (compatible; AgentReadyBot/1.0; +https://agentexchange.work/)" https://example.com/ ``` `200` and a size well above a few kilobytes of real content. A 403 or 503 usually means a challenge. Re-grade: https://grade.agentexchange.work/grade?url=example.com&fresh=1 ## Questions ### Does Googlebot need server rendering? Google renders JavaScript for Search, though in a separate, later phase. The AI crawlers in Vercel's analysis did not render at all, so server-rendered HTML serves both. ### What counts as an empty root container? A div, main or section with an id such as root, app, __next, __nuxt, __gatsby or __svelte and nothing inside it (a noscript note inside is ignored). It is the signature of a page assembled entirely in the browser. ### My site shows a Cloudflare challenge to bots. Is that a problem? For AI visibility, yes: the crawler receives the challenge, not your content, and Anthropic says its crawlers do not try to get past CAPTCHAs. Challenge abusive traffic, not every non-browser client. ## Sources - [The rise of the AI crawler (December 2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler) (Vercel) - [Understand the JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) (Google Search Central) - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Anthropic) - [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) (Perplexity) - [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) (Cloudflare) ## Related - [Structured data](https://grade.agentexchange.work/fix/structured-data.md): One JSON-LD block that parses plus og:title and og:description on the home page. - [robots.txt for AI crawlers](https://grade.agentexchange.work/fix/robots-txt-ai-crawlers.md): robots.txt groups for eight AI crawler tokens; a blanket Disallow: / counts as blocked. - [llms.txt](https://grade.agentexchange.work/fix/llms-txt.md): A Markdown guide to your key pages at /llms.txt, served as text, not as your HTML 404. - [All fix guides](https://grade.agentexchange.work/fix)