Skip to content
goflag
Documentation
Rule catalogue

robots.blocks-site

robots.txt must not forbid crawling a site that asks to be indexed

errorsite-widevendor-spec

Why it matters

The most expensive misconfiguration a site can carry, and it is invisible from inside a browser. Severity drops to a warning when nothing contradicts the block: a staging environment that disallows everything and claims nothing else is doing exactly what it means to.

What the finding looks like

The message goflag prints, with example values substituted.

error robots.blocks-site robots.txt disallows the whole site for User-agent: *, but 42 crawled pages declare <meta name="robots" content="index">. Both cannot be true: robots.txt wins, so the pages are never fetched and the meta tag is never read.

How to fix it

The catalogue carries no snippet for a site rule: a cross-page rule builds its remedy from what the crawl found, so goflag attaches the fix to the finding itself rather than to the rule. Run it and read the finding.

Says who?

This rule is vendor-spec, and it cites the documents below. Nothing here is goflag’s opinion of what a good page looks like — if you disagree with a finding, this is what you are disagreeing with.