Agent-Readiness Grade / Crawlers / GPTBot

GPTBot

OpenAI · training. Source: https://platform.openai.com/docs/bots, read 2026-09-29. Where the vendor page does not state a fact, this page says "not published".

Facts from the vendor page

Operator OpenAI

What it is for Training crawler. OpenAI describes GPTBot as the crawler that collects content which may be used to train its generative AI foundation models, and says that disallowing it signals a site's content should not be used for that training.

Published user-agent string Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Example string from the vendor page; the version number may change. When fetching robots.txt, OpenAI may add a "robots.txt" marker to the string.

Honors robots.txt Yes. OpenAI documents GPTBot as a robots.txt tag site owners use to manage how their content is used, independent of the OAI-SearchBot tag. If both are allowed, one crawl may serve both purposes.

Verification (IP ranges or reverse DNS) Published IP ranges at https://openai.com/gptbot.json.

robots.txt token GPTBot

Operator	OpenAI
What it is for	Training crawler. OpenAI describes GPTBot as the crawler that collects content which may be used to train its generative AI foundation models, and says that disallowing it signals a site's content should not be used for that training.
Published user-agent string	`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot` Example string from the vendor page; the version number may change. When fetching robots.txt, OpenAI may add a "robots.txt" marker to the string.
Honors robots.txt	Yes. OpenAI documents GPTBot as a robots.txt tag site owners use to manage how their content is used, independent of the OAI-SearchBot tag. If both are allowed, one crawl may serve both purposes.
Verification (IP ranges or reverse DNS)	Published IP ranges at https://openai.com/gptbot.json.
robots.txt token	`GPTBot`

robots.txt lines

To allow GPTBot explicitly (so your policy survives a change to the User-agent: * group):

User-agent: GPTBot
Allow: /

To block it:

User-agent: GPTBot
Disallow: /

What Agent-Readiness Grade checks for GPTBot

The crawler-access area parses your robots.txt by user-agent group and reports GPTBot as allowed, disallowed or unmentioned (unmentioned means allowed by default). A group for "GPTBot" wins over the "*" group; "Disallow: /" or "Disallow: /*" without a matching "Allow: /" counts as blocked; narrower path rules are reported as partial. GPTBot is one of the 8 tokens in that score (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, Amazonbot, CCBot, Bytespider).

Context: in the 30-day census of one Cloudflare zone (2026-08-29 to 2026-09-28), 1,957 requests carried a user agent matching "GPTBot". See the census report.

Other crawlers

OAI-SearchBot · ChatGPT-User · ClaudeBot · Claude-User · PerplexityBot · Google-Extended · Amazonbot · CCBot · Bytespider

Free agent-readiness checkers are a crowded category: Goodie, Frase, aiwebreadiness.com, Arobis, Rank Prompt, Otterly and Writesonic each offer one. Of the checkers we looked at, Agent-Readiness Grade is the only one we found that grades payment and agent-checkout surfaces (/.well-known/x402, agent-card.json, mcp-registry-auth, openapi.json, the MCP endpoint) and legal notices. Statement dated 2026-09-29.