XML sitemap validation: a pre-launch test for developers

XML sitemap validation should answer a practical release question: does the deployed sitemap list the public pages you want crawlers to find, in a file they can actually read? A generated file can look correct in source control while the production URL serves an HTML fallback, an old host, or a list of routes that redirect.

This guide shows how to test the live response, parse the XML, compare its URLs with the pages you intend to publish, and keep a small check in CI. If you first need to decide what belongs in a sitemap, read the robots.txt and sitemap.xml guide.

XML sitemap validation checklist

Use this order on the deployed host:

1. Request the sitemap URL without a logged-in session and check its status, content type, and body.
2. Parse the entire XML file, including its required <urlset>, <url>, and <loc> elements.
3. Confirm every <loc> is an absolute URL on the intended public host.
4. Remove duplicate, preview, private, redirected, missing, and noncanonical URLs.
5. Confirm the critical new or changed pages are present.
6. Check that robots.txt names the right sitemap and allows the representative page.
7. Submit or inspect the sitemap in Search Console after deployment.
8. Rerun the checks when routes, domains, or publishing rules change.

A passing XML parse is only the first step. It says the file has valid structure; it does not say the listed pages are reachable, indexable, or useful.

Request the production sitemap, not a build artifact

Start with the exact public URL, usually https://example.com/sitemap.xml. A CDN rule or single-page-app fallback can turn a missing sitemap into a 200 HTML response. Save the response so you can inspect what was delivered:

curl --fail --location --show-error \
  --dump-header sitemap.headers \
  --output sitemap.xml \
  https://example.com/sitemap.xml

Check the final status and Content-Type in sitemap.headers, then open the saved file. It should be XML rather than a login page or application shell. Record redirects, because a Sitemap: line in robots.txt should point to a stable final URL. Google's sitemap creation guidance recommends hosting a sitemap where Google can fetch it and using absolute URLs for its entries.

For a small site, one urlset file is enough. Google limits a single sitemap to 50,000 URLs or 50 MB uncompressed; larger sites can split files and use a sitemap index. An index uses <sitemapindex> and points to child sitemaps. Test those child files too. The example below checks a regular urlset, not an index.

Parse the XML sitemap and check its URL entries

The Sitemaps protocol requires a <urlset> root with the sitemap namespace and a <loc> under each <url>. Values must be escaped and the file must be UTF-8. For example, a query parameter separator in XML is &, not a bare &.

This small Python check parses the entire saved file and catches a few common release mistakes. Change the host and expected path for your site:

from urllib.parse import urlsplit
import xml.etree.ElementTree as ET

root = ET.parse("sitemap.xml").getroot()
ns = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
assert root.tag == f"{ns}urlset", "Expected a standard URL sitemap"

locations = [
    entry.findtext(f"{ns}loc", default="").strip()
    for entry in root.findall(f"{ns}url")
]
assert locations, "No URL entries found"
assert len(locations) == len(set(locations)), "Duplicate URL entries found"
assert "https://example.com/docs" in locations, "New docs page is missing"

for location in locations:
    url = urlsplit(location)
    assert url.scheme == "https" and url.netloc == "example.com", location
    assert not url.fragment, f"Fragment in sitemap URL: {location}"

print(f"Checked {len(locations)} sitemap URLs")

Run it from the directory containing sitemap.xml. This verifies syntax, host consistency, duplicates, and one release-critical route. It does not request each page or interpret a page's canonical tag. Add those checks for the routes that matter most. If your sitemap uses a sitemapindex, parse each referenced child file before checking page URLs.

Compare each URL with the page you publish

Treat the sitemap as a list of preferred public URLs. Google's canonicalization guidance calls sitemap inclusion a canonical signal, although weaker than redirects or a rel="canonical" annotation. Conflicting signals make a release harder to understand. The canonical URL and redirects guide covers how to align them.

Take a sample from each route type: home, article, documentation, product, and any newly launched landing page. For each <loc>, check:

• The request ends on the same preferred URL, without a redirect to another path or host.
• The final response is a usable public page, not a 404, login screen, or soft 404 application shell.
• The page's canonical URL agrees with the sitemap entry.
• The route is not a preview, search result, account page, duplicate parameter variant, or retired page.
• A page meant to be discovered is not blocked by the relevant robots.txt rule.

Use a focused sample in CI and a broader crawl when the site changes substantially. The sitemap should include URLs you want to appear in search results, not every URL your application can generate. Google says to use fully qualified URLs and generally include the canonical version of each page in a sitemap.

Do not update <lastmod> on every deploy just because the XML file was rebuilt. Google's sitemap guidance says it uses this value when it is consistently accurate and reflects a significant page change. The element is optional; an omitted date is more honest than a fabricated one.

Check discovery and indexing separately

Open https://example.com/robots.txt and look for an absolute Sitemap: URL. Confirm it fetches the file you just tested. The sitemap can also be submitted through Google Search Console, where the Sitemaps report shows access and processing errors. Google's documentation says submission is a hint, not a guarantee that Google will crawl or index every entry.

If a URL is missing from search results, test the page itself. A readable sitemap cannot repair a blocked route, a wrong canonical, an error response, or content that was never published. The website launch checklist covers those wider release checks.

For a repeatable release gate, keep the Python parser alongside your site build. Fetch the deployed sitemap after the preview or production deployment, then run the parser with that release's expected critical URLs. Keep the response headers and the first failing URL in the CI log. The website regression testing checklist shows how to compare a small, stable set of production signals between releases.

Check a public page with Slop Audit

After validating the full sitemap, run a Slop Audit on a representative public page. The report checks whether it can reach a sitemap from the first declared Sitemap: URL or the usual /sitemap.xml path, and whether the audited page appears in the bounded sitemap response. It also shows the page's final URL, canonical tag, and crawler-access evidence so you can investigate a mismatch.

Slop Audit checks one sitemap target and only part of its response. It does not parse every XML entry, follow a sitemap index through all child files, or prove that a search engine indexed the page. Keep the full-file parser and route sample above as your release check, then use the audit report to see how one live page fits into the rest of your launch evidence.