Robots.txt vs sitemap.xml: what each file does

robots.txt and sitemap.xml are small files with different jobs. Confusing them can leave a site reachable but hard for crawlers to understand, or block pages you meant to publish.

The short version is simple:

• robots.txt tells cooperating crawlers which paths they may request.
• sitemap.xml lists canonical URLs that are useful for discovery.

Neither file guarantees indexing, ranking, or a specific search result. They work with the page, links, redirects, and the crawler's own decisions.

What robots.txt controls

Place the file at the origin root: https://example.com/robots.txt. A minimal public policy can look like this:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

The User-agent group identifies the crawler policy. Allow and Disallow apply path rules inside that group. The Sitemap line advertises where a crawler can find the sitemap; it should use an absolute URL on the same site.

Keep the policy short and clear. Do not disallow / on a site that should be crawled, and do not assume a robots rule removes a URL from search results. If a URL must stay private, use access control or a page-level response.

What sitemap.xml describes

A sitemap is an XML document containing stable, absolute URLs you want crawlers to discover. A small site can begin with one URL:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
  </url>
</urlset>

Add important public pages when they become canonical. Avoid previews, user-specific routes, duplicate query-string URLs, and URLs that redirect. Keep the list current. A short, accurate sitemap is better than every route in an application.

Google's sitemap documentation covers the XML format and submission options. Their robots.txt guidance explains how crawler rules are interpreted.

How the two files should agree

The two files should follow these rules:

1. robots.txt should not block a page that you list in the sitemap.
2. The sitemap should list the canonical HTTPS URL, not an old host or redirect source.
3. The page should have internal links and metadata that agree with that canonical URL.
4. Every listed URL should return a useful response when fetched publicly.

If you block a route, leave it out of the sitemap and document why. If a page is private, do not use a public crawl rule as its privacy control.

Common mistakes to catch before launch

• The sitemap contains http:// URLs while the site redirects to HTTPS.
• The sitemap points at a staging hostname or a previous domain.
• robots.txt blocks a parent path that contains important pages.
• The sitemap is linked with a relative URL or is not available at the origin root.
• Generated route lists include login, checkout, search, or duplicate parameter URLs.
• The files are present in source control but missing from the deployed public origin.

Test the deployed URLs directly, then check the response and headers. A file in your repository does not prove that the public site serves it.

A simple launch check

Open both files in a private browser window and compare them with the canonical links on important pages. Then use the website launch checklist to check crawlability and the user experience.

For a report of what was found and what could not be checked, run an audit. Use the output for review. It cannot guarantee what a crawler will decide.