Skip to content
goflag
Documentation
Rule catalogue

robotstxt.invalid-line

Every line of robots.txt should parse as something

warningsite-widespec-required

Why it matters

A typo in robots.txt does not fail loudly, it fails as silence: the crawler drops the line and the rule you meant to write simply never existed. Disalow: looks right at a glance and protects nothing.

What the finding looks like

The message goflag prints, with example values substituted.

warn robotstxt.invalid-line robots.txt has 1 line that parses as nothing: line 2 (unknown directive disalow). A crawler drops them silently, so the rule you meant to write is simply absent.

How to fix it

The catalogue carries no snippet for a site rule: a cross-page rule builds its remedy from what the crawl found, so goflag attaches the fix to the finding itself rather than to the rule. Run it and read the finding.

Says who?

This rule is spec-required, and it cites the document below. Nothing here is goflag’s opinion of what a good page looks like — if you disagree with a finding, this is what you are disagreeing with.