Agent-Readiness Grade / Crawlers / Google-Extended
Google-Extended
Google · training and grounding control token. Source: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers and https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers, read 2026-09-29. Where the vendor page does not state a fact, this page says "not published".
Facts from the vendor page
| Operator | |
|---|---|
| What it is for | Not a crawler but a control token. Google says Google-Extended has no user-agent string of its own: crawling is done by Google's existing user agents and the token works in robots.txt as a control. It governs whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. It does not affect a site's inclusion or ranking in Google Search. |
| Published user-agent string | Not applicable: Google states Google-Extended has no separate HTTP user-agent string. |
| Honors robots.txt | Yes. Google says its common crawlers always obey robots.txt rules when crawling automatically, and Google-Extended exists only as a robots.txt user-agent token. |
| Verification (IP ranges or reverse DNS) | Google's common crawlers use the IP ranges in https://developers.google.com/static/crawling/ipranges/common-crawlers.json and reverse-DNS hostnames matching crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com; verification guide: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests. |
| robots.txt token | Google-Extended |
robots.txt lines
To allow Google-Extended explicitly (so your policy survives a change to the User-agent: * group):
User-agent: Google-Extended Allow: /
To block it (Google says this does not affect Google Search inclusion or ranking):
User-agent: Google-Extended Disallow: /
What Agent-Readiness Grade checks for Google-Extended
The crawler-access area parses your robots.txt by user-agent group and reports Google-Extended as allowed, disallowed or unmentioned (unmentioned means allowed by default). A group for "Google-Extended" wins over the "*" group; "Disallow: /" or "Disallow: /*" without a matching "Allow: /" counts as blocked; narrower path rules are reported as partial. Google-Extended is one of the 8 tokens in that score (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, Amazonbot, CCBot, Bytespider). Because Google-Extended never appears in a user-agent string, the grader can only read the token in robots.txt; it does not expect to see it in logs.
Context: in the 30-day census of one Cloudflare zone (2026-08-29 to 2026-09-28), 347 requests carried a user agent matching "Google-Extended"; since Google says that token never appears in a user-agent string, those requests did not come from Google's documented crawlers. See the census report.
Other crawlers
GPTBot · OAI-SearchBot · ChatGPT-User · ClaudeBot · Claude-User · PerplexityBot · Amazonbot · CCBot · Bytespider
Free agent-readiness checkers are a crowded category: Goodie, Frase, aiwebreadiness.com, Arobis, Rank Prompt, Otterly and Writesonic each offer one. Of the checkers we looked at, Agent-Readiness Grade is the only one we found that grades payment and agent-checkout surfaces (/.well-known/x402, agent-card.json, mcp-registry-auth, openapi.json, the MCP endpoint) and legal notices. Statement dated 2026-09-29.