Skip to content
goflag
Documentation
Rule catalogue

indexing.blocked-url-indexable

On a target declared out of the index, robots.txt must not hide the noindex

warningsite-widevendor-spec

Why it matters

The usual staging setup, and it does less than it looks like. Google says robots.txt is "not a mechanism for keeping a web page out of Google": a blocked URL can still appear in results if something links to it, with no description, and a noindex on it is never read because the page is never fetched. So a block plus a noindex is two locks where the first one hides the second. The fix is opening the host so the noindex can be read, and that also lets in every crawler that honours the file. Keeping the block is a defensible trade, and a baseline is how you accept it. Only fires under --expect-indexing no.

What the finding looks like

The message goflag prints, with example values substituted.

warn indexing.blocked-url-indexable robots.txt keeps Googlebot out of 24 pages on a target declared out of the index: https://staging.example.com/fr, https://staging.example.com/en, and 22 more. All of them say noindex, which Googlebot never reads because it never fetches them. A blocked URL can still appear in Google's results, from links alone.

How to fix it

The catalogue carries no snippet for a site rule: a cross-page rule builds its remedy from what the crawl found, so goflag attaches the fix to the finding itself rather than to the rule. Run it and read the finding.

Says who?

This rule is vendor-spec, and it cites the documents below. Nothing here is goflag’s opinion of what a good page looks like — if you disagree with a finding, this is what you are disagreeing with.