robots.blocks-site

robots.txt must not forbid crawling a site that asks to be indexed

errorsite-wide

Why it matters

The most expensive misconfiguration a site can carry, and it is invisible from inside a browser. Severity drops to a warning when nothing contradicts the block: a staging environment that disallows everything and claims nothing else is doing exactly what it means to.

What the finding looks like

The message goflag prints, with example values substituted.

error robots.blocks-site robots.txt disallows the whole site for User-agent: *, but 42 crawled pages declare <meta name="robots" content="index">. Both cannot be true: robots.txt wins, so the pages are never fetched and the meta tag is never read.

Gate the disallow on the deployed environment

Written for the Next.js App Router. The finding is correct on any stack; only the remedy assumes one.

// app/robots.ts — the flag must be readable at build AND at runtime,
// or a production container silently serves the staging rules.
const isProduction = process.env.APP_ENV === "production";

export default function robots(): MetadataRoute.Robots {
  if (!isProduction) return { rules: { userAgent: "*", disallow: "/" } };
  return {
    rules: { userAgent: "*", allow: "/", disallow: ["/api/", "/_next/"] },
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

Says who?

This rule runs on the cross-page contract, which does not yet carry sources. It picks them up when the site-level rules move onto the same descriptor as the page rules.