Skip to content
goflag
Documentation
Rule catalogue

robotstxt.oversized

Rules past 500 KiB of robots.txt are not guaranteed to be read

errorsite-widespec-required

Why it matters

Parsers are only required to read the first 500 KiB. Past that, rules are not wrong, they are absent — and a file that long is usually generated, so nobody scrolls to the end to notice.

What the finding looks like

The message goflag prints, with example values substituted.

error robotstxt.oversized robots.txt is 812 KiB. A parser is only required to honour the first 500 KiB (RFC 9309 §2.4), so every rule past that point silently does not exist.

How to fix it

The catalogue carries no snippet for a site rule: a cross-page rule builds its remedy from what the crawl found, so goflag attaches the fix to the finding itself rather than to the rule. Run it and read the finding.

Says who?

This rule is spec-required, and it cites the document below. Nothing here is goflag’s opinion of what a good page looks like — if you disagree with a finding, this is what you are disagreeing with.