Skip to content
goflag
Documentation
Rule catalogue

sitemap.entry.blocked-by-robots

A sitemap must not list URLs that robots.txt forbids fetching

errorsite-widevendor-spec

Why it matters

The sitemap says index this and robots.txt says never fetch it. Both are the same site speaking, and robots.txt is the one that decides — so the entry is not merely ignored, it is an instruction the site contradicts a line later.

What the finding looks like

The message goflag prints, with example values substituted.

error sitemap.entry.blocked-by-robots 3 sitemap entries are disallowed by robots.txt: https://example.com/admin/a, https://example.com/admin/b. The sitemap says "index this" and robots.txt says "never fetch it" — both cannot hold, and robots.txt is the one that decides.

How to fix it

The catalogue carries no snippet for a site rule: a cross-page rule builds its remedy from what the crawl found, so goflag attaches the fix to the finding itself rather than to the rule. Run it and read the finding.

Says who?

This rule is vendor-spec, and it cites the documents below. Nothing here is goflag’s opinion of what a good page looks like — if you disagree with a finding, this is what you are disagreeing with.