Skip to content
goflag
Documentation
Rule catalogue

indexing.site-blocked

robots.txt must not keep Googlebot out of a target declared in the index

errorsite-widevendor-spec

Why it matters

The outage this flag exists for. You said this site belongs in the index, and robots.txt keeps Googlebot out of all of it, so Google stops reading your pages while everything looks fine in a browser. It is a rule of its own rather than robots.blocks-site turned up, because a baseline captured on a staging host already holds that finding and would wave this one through. Only fires under --expect-indexing yes.

What the finding looks like

The message goflag prints, with example values substituted.

error indexing.site-blocked robots.txt keeps Googlebot out of the whole site, through Disallow: / (line 2), on a target declared in the index (--expect-indexing yes). Googlebot will not crawl any page on this origin.

How to fix it

The catalogue carries no snippet for a site rule: a cross-page rule builds its remedy from what the crawl found, so goflag attaches the fix to the finding itself rather than to the rule. Run it and read the finding.

Says who?

This rule is vendor-spec, and it cites the documents below. Nothing here is goflag’s opinion of what a good page looks like — if you disagree with a finding, this is what you are disagreeing with.