Skip to content
goflag
Documentation
Rule catalogue

robotstxt.cross-origin

/robots.txt should not redirect to another origin

warningsite-widevendor-spec

Why it matters

Legal, and almost always accidental. Following the redirect is permitted, but the policy governing this site now lives on a host this site does not control — and the day that host answers differently, nobody here will know why the traffic changed.

What the finding looks like

The message goflag prints, with example values substituted.

warn robotstxt.cross-origin robots.txt redirects to https://cdn.example.net/robots.txt, on another origin. RFC 9309 §2.3.1.2 permits following it, so this works — but the policy for this site now lives somewhere this site does not control, and it is usually a proxy accident rather than a decision.

How to fix it

The catalogue carries no snippet for a site rule: a cross-page rule builds its remedy from what the crawl found, so goflag attaches the fix to the finding itself rather than to the rule. Run it and read the finding.

Says who?

This rule is vendor-spec, and it cites the document below. Nothing here is goflag’s opinion of what a good page looks like — if you disagree with a finding, this is what you are disagreeing with.