Skip to content
goflag
Documentation
Rule catalogue

robotstxt.unreachable

A robots.txt that errors is read as forbidding the whole site

errorsite-widespec-required

Why it matters

The failure mode nobody plans for: a 500 on one small text file is read as a site-wide ban. RFC 9309 is explicit — while robots.txt errors, a crawler must assume complete disallow. An outage here costs more than an outage on any page.

What the finding looks like

The message goflag prints, with example values substituted.

error robotstxt.unreachable robots.txt could not be read: the origin answered 503. RFC 9309 §2.3.1.4 tells a crawler to assume a complete disallow for as long as this lasts — an outage on this one file takes the whole site out of the index.

How to fix it

The catalogue carries no snippet for a site rule: a cross-page rule builds its remedy from what the crawl found, so goflag attaches the fix to the finding itself rather than to the rule. Run it and read the finding.

Says who?

This rule is spec-required, and it cites the document below. Nothing here is goflag’s opinion of what a good page looks like — if you disagree with a finding, this is what you are disagreeing with.