Skip to content
goflag
Documentation
Rule catalogue

robots.blocks-ai-answers

robots.txt must not keep AI answer engines out of a site Googlebot may crawl

warningsite-widevendor-spec

Why it matters

An AI answer engine cites what its agents may read. OpenAI says a site that disallows OAI-SearchBot is not shown in ChatGPT search answers; Anthropic, that disallowing Claude-SearchBot stops Claude indexing it for search, and disallowing Claude-User stops Claude reading a page someone asks it about; Perplexity, that a site appears in its results by allowing PerplexityBot. The usual cause is not a decision: a User-agent: * disallow with a Googlebot exception, or a list of AI bots pasted from somewhere, catches these agents along with the training crawlers. Training crawlers (GPTBot, ClaudeBot) are left alone — they decide what a model learns from, not what an engine cites — and so are ChatGPT-User and Perplexity-User, which their operators say robots.txt may not bind. Read at /, and silent when the file keeps Googlebot out as well: that is robots.blocks-site.

What the finding looks like

The message goflag prints, with example values substituted.

warn robots.blocks-ai-answers robots.txt lets Googlebot in and keeps 2 AI answer engine agents out of the whole site. OAI-SearchBot, through User-agent: OAI-SearchBot (line 3) and Disallow: / (line 5): ChatGPT search leaves the site out of its answers. Claude-User, through User-agent: Claude-User (line 4) and Disallow: / (line 5): Claude does not fetch a page when someone asks about it.

How to fix it

The catalogue carries no snippet for a site rule: a cross-page rule builds its remedy from what the crawl found, so goflag attaches the fix to the finding itself rather than to the rule. Run it and read the finding.

Says who?

This rule is vendor-spec, and it cites the documents below. Nothing here is goflag’s opinion of what a good page looks like — if you disagree with a finding, this is what you are disagreeing with.