CleanedWeb

Glossary

Sitemap

A sitemap is an XML file that lists the URLs of a website, with optional details such as when each page last changed. Sites publish one so crawlers can find pages that links alone would not reveal.

Format and limits

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/products/desk-lamp</loc>
    <lastmod>2026-01-15</lastmod>
  </url>
</urlset>

Using one to collect data

A sitemap is a quick way to discover item pages such as products or job postings, to size a job before you start, and to pick out the pages that changed since your last visit through lastmod (see data freshness).

Treat it as a hint. It lists pages, not records. It can be incomplete or stale, some sites stamp every URL with the date of the last deploy, and many sites have no sitemap at all. Listing a URL is not permission to fetch it, so robots.txt and the site's terms still apply.

In CleanedWeb

CleanedWeb builds an API from a collection page, the public list page you give it, rather than from a sitemap. A sitemap can still help you choose that page, or see how many item pages a site has. This site publishes its own at /sitemap.xml.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.