slop audit
Blog post 40 / Crawlability

Robots.txt testing: a practical developer guide

Use robots.txt testing to verify the deployed file, test crawler rules against real URLs, catch accidental blocks, and keep checks in CI.

By Slop Audit · 11 min read

Robots.txt testing verifies the file that crawlers can actually fetch, the rule group each crawler will use, and the result for every important public URL. Test the deployed response first, then evaluate a URL matrix and keep the critical cases in CI so a release cannot quietly block the site.

A valid-looking source file is not enough. A framework fallback can return HTML from /robots.txt, a deployment can publish the file on the wrong host, or one broad Disallow rule can cover more paths than its author intended. Treat the public response and the path-matching result as separate checks.

Robots.txt testing checklist

Use this sequence before launch and after any routing, hosting, or crawler-policy change:

  1. Request /robots.txt from the exact production protocol and host.
  2. Record the final URL, status code, content type, and response body.
  3. Confirm that the file is plain text at the origin root, not an application fallback.
  4. List the crawler user agents whose behavior matters to the release.
  5. Build representative URLs that should be allowed and disallowed.
  6. Evaluate the matching group and the most specific rule for every case.
  7. Check broad rules such as Disallow: / before narrower paths.
  8. Verify each Sitemap URL independently.
  9. Keep crawl controls separate from noindex and access control.
  10. Run the same cases in CI and verify the deployed file in Search Console.

The output should say which crawler, URL, and rule produced the result. A generic “robots.txt passed” message is not useful when one user-agent group or one path is wrong.

Test the deployed robots.txt response

The Robots Exclusion Protocol puts the file at the top-level /robots.txt path. Its rules apply only to that protocol, host, and port. Google's robots.txt location guidance gives concrete examples: a file on www.example.com does not automatically control example.com, another subdomain, or a nonstandard port.

Start with the response, not the repository:

curl -sS -D /tmp/robots-headers.txt -o /tmp/robots-body.txt https://example.com/robots.txt
sed -n '1,20p' /tmp/robots-headers.txt
sed -n '1,80p' /tmp/robots-body.txt

Check these conditions:

  • The requested URL uses the production scheme and host.
  • The final response is intentional and does not cross to a staging or unrelated host.
  • A successful response contains crawler directives rather than the site's HTML shell.
  • The file is UTF-8 plain text and remains small enough for the crawlers you support.
  • A CDN, framework rewrite, or authentication layer does not intercept the path.

RFC 9309 defines the root location, UTF-8 encoding, and text/plain media type for the protocol. Google documents some crawler-specific tolerance, including parsing valid lines from imperfect files, but production tests should still catch an HTML fallback or malformed response before relying on that tolerance.

Understand what robots.txt testing can prove

A robots.txt result answers a crawl-access question for a crawler and URL. It does not prove that the URL will be indexed, ranked, kept private, or excluded from every automated client.

Use the right control for the job:

  • Use robots.txt to tell cooperating crawlers which paths they may request.
  • Use a robots meta tag or X-Robots-Tag when a supported search engine may crawl a resource but should not index it.
  • Use authentication and authorization for private content.
  • Use a sitemap to advertise canonical public URLs for discovery.

Google's `noindex` documentation warns that a crawler must be allowed to fetch a page before it can see a noindex directive. Blocking the page in robots.txt can therefore prevent that indexing rule from being read. If a public page is excluded unexpectedly, use the noindex tag troubleshooting guide to inspect its HTML, response headers, and deployment policy. The protocol itself also states that robots rules are not access authorization and can reveal the paths they name.

If the distinction is still unclear, read what robots.txt and sitemap.xml each do before writing a policy. That comparison covers the two files' roles; this guide focuses on testing one deployed robots policy against real URLs.

Build a crawler and URL test matrix

Do not test only the homepage. Create a small matrix from the routes your release must expose or protect.

Include cases such as:

  • /, a product page, documentation, and a representative article that must be crawlable;
  • /preview/, internal search results, or faceted routes you intentionally restrict;
  • a path near each boundary, such as /admin and /administrator;
  • encoded paths, uppercase variants, and query strings when your router treats them differently;
  • a URL covered by an Allow exception inside a broader disallowed path; and
  • every crawler-specific group you maintain, plus the wildcard group.

Write the expected result beside every case:

Googlebot   https://example.com/                 ALLOW
Googlebot   https://example.com/docs/start       ALLOW
Googlebot   https://example.com/preview/draft    DISALLOW
ExampleBot  https://example.com/private/report   DISALLOW

Choose test URLs from the deployed route inventory rather than inventing only convenient examples. This catches stale rules after a rename and broad prefix matches that were invisible in the original ticket.

Apply robots.txt rule matching correctly

A robots file is organized into user-agent groups followed by Allow and Disallow rules. For a matching group, RFC 9309 rule matching uses the most specific matching path. When equally specific Allow and Disallow rules conflict, Allow should win.

Consider this file:

User-agent: *
Disallow: /private/
Allow: /private/public-guide

Sitemap: https://example.com/sitemap.xml

/private/account is disallowed, while /private/public-guide is allowed because the longer matching rule is more specific. Do not assume the first matching line wins. Also test the complete URL path: prefix boundaries can surprise teams that intended to block one directory name rather than every path beginning with the same characters.

Crawler implementations can support extensions such as * and $, and behavior outside the standard can differ. When Googlebot behavior is the release requirement, use Google's documented rule-precedence examples or its open-source parser rather than a homegrown regular expression.

Test redirects and HTTP failures

Status codes change how crawlers interpret the policy. Test failure paths deliberately instead of checking only a 200 response.

RFC 9309 says crawlers should follow at least five redirects while fetching robots.txt. It distinguishes an unavailable response such as many 4xx statuses from an unreachable file caused by server or network errors. Google publishes its own robots.txt status-code behavior: most 4xx responses are treated as no crawl restrictions, while 5xx and network failures can pause crawling or cause a cached policy to be reused.

For the deployed host, verify:

  • HTTP redirects once to the intended HTTPS host.
  • Hostname normalization does not move the policy to an unrelated authority.
  • /robots.txt does not enter a redirect loop.
  • A missing file returns the response you intended rather than a soft HTML fallback.
  • An application or CDN failure cannot persistently return 5xx for this route.

Keep the observed status and redirect chain with the test result. “File missing” and “file temporarily unreachable” are different operational states.

Verify Sitemap directives separately

A Sitemap line is not an access rule. It advertises a sitemap location, and a valid-looking absolute URL can still redirect, return HTML, contain invalid XML, or list stale pages.

Sitemap: https://example.com/sitemap.xml

Request every declared sitemap and validate its response independently. Then compare the listed URLs with the canonical host and the robots rules. A public page should not be advertised in the sitemap while a matching crawler group disallows its path.

Use the XML sitemap validation checklist for response, parsing, canonical, and discovery checks. Use the canonical URL and redirects guide when the final URL, internal links, canonical tag, and sitemap entry disagree.

Add robots.txt testing to CI

Keep a fast contract test for the policy your deployment owns. The example below verifies the response and critical lines without pretending to implement the complete matching standard:

import assert from "node:assert/strict";

const origin = process.env.PUBLIC_ORIGIN;
assert(origin, "PUBLIC_ORIGIN is required");

const response = await fetch(new URL("/robots.txt", origin), {
  redirect: "follow",
});
const body = await response.text();

assert.equal(response.status, 200);
assert.match(response.headers.get("content-type") ?? "", /^text\/plain\b/i);
assert.doesNotMatch(body, /<!doctype|<html\b/i);
assert.match(body, /^User-agent:\s*\*/im);
assert.doesNotMatch(body, /^Disallow:\s*\/$/im);
assert.match(body, /^Sitemap:\s*https:\/\//im);

Add standards-aware URL matching when the file contains crawler-specific groups, wildcards, or exceptions. Google maintains an open-source robots.txt parser and matcher with a command-line tool that accepts a local policy, user agent, and URL. Use a maintained parser for the URL matrix; keep simple text assertions only for release invariants that are genuinely exact.

Run the deployed check after the preview or production URL exists. The website audit with GitHub Actions guide shows how to keep public-URL evidence separate from source checks, protect credentials, and report unavailable external checks honestly. Add the result to your website regression testing routine after the first reviewed baseline.

Verify Googlebot behavior in Search Console

Local and CI checks prove what your chosen parser saw. They do not prove what Google last fetched or whether Google can reach the host from its infrastructure.

Google's current robots.txt testing guidance recommends two paths: use the robots.txt report in Search Console for a file already published on a verified property, or use Google's open-source parser locally. After a production change:

  1. Open the robots.txt report for the correct property.
  2. Confirm that Google fetched the intended file and can process it.
  3. Inspect any reported fetch or syntax issue.
  4. Use URL Inspection for a small set of critical pages.
  5. Request an updated robots.txt crawl when an urgent fix must replace Google's cached copy.

Keep Search Console verification beside the automated matrix. One checks Google's observed state; the other gives your team repeatable release coverage.

Run robots.txt testing with Slop Audit

Run a Slop Audit on the deployed page to add a bounded public check to the release evidence. Slop Audit requests the origin-root robots.txt, distinguishes a valid text response from a missing or HTML fallback, checks for a global Disallow: /, compares the submitted page path with observed Disallow prefixes, and follows the first declared sitemap target to look for parseable XML.

The check uses the Slop Audit crawler identity and is intentionally narrower than a standards-complete Googlebot simulator. It does not model every user-agent merge, Allow exception, wildcard, encoded path, crawler extension, cached policy, or search-engine indexing decision. Use Search Console or the relevant crawler's parser for those questions.

The useful conversion path is evidence, not a generic pass badge: submit a public page that must remain crawlable, inspect the robots finding and source URL, fix the deployed policy, and rerun the same page. Pair that result with your URL matrix so both the launch route and deliberate exclusions remain reviewable.

Robots.txt testing release sign-off

  • The file is available at /robots.txt on the exact production protocol and host.
  • The response status, redirects, content type, encoding, and body are intentional.
  • The response is not an HTML application fallback.
  • Critical public URLs are allowed for the crawler groups that matter.
  • Deliberately restricted URLs are disallowed by the expected rule.
  • Specific Allow exceptions beat broader restrictions where intended.
  • No production group accidentally contains Disallow: /.
  • Every declared sitemap returns the expected XML and canonical URLs.
  • Private content uses access control rather than a public robots rule.
  • Index exclusions use a supported noindex mechanism on crawlable resources.
  • CI tests the deployed response and a standards-aware URL matrix.
  • Search Console shows that Google can fetch and process the current file.
  • A Slop Audit records the bounded production evidence for the release.

Robots.txt testing is complete when a reviewer can name the crawler, URL, matched rule, deployed response, and expected result. That evidence is much safer than checking that a file exists and assuming every crawler will interpret it the way the author intended.