Skip to content
goflag
Documentation
Rule catalogue

robotstxt.unknown-directive

Non-standard robots.txt directives are read by some crawlers and ignored by others

infosite-wideguideline

Why it matters

Not a defect — Crawl-delay and Host are spelled correctly and some crawlers honour them. Worth knowing because Google does not, so a site relying on one for rate limiting is relying on nothing where it matters most.

What the finding looks like

The message goflag prints, with example values substituted.

info robotstxt.unknown-directive robots.txt uses crawl-delay, which RFC 9309 does not define. Some crawlers honour it and Google ignores it — so this is worth knowing, not necessarily worth changing.

How to fix it

The catalogue carries no snippet for a site rule: a cross-page rule builds its remedy from what the crawl found, so goflag attaches the fix to the finding itself rather than to the rule. Run it and read the finding.

Says who?

This rule is guideline, and it cites the documents below. Nothing here is goflag’s opinion of what a good page looks like — if you disagree with a finding, this is what you are disagreeing with.