Glossary
Sitemap
A sitemap is an XML file that lists the URLs of a website, with optional details such as when each page last changed. Sites publish one so crawlers can find pages that links alone would not reveal.
Format and limits
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/products/desk-lamp</loc>
<lastmod>2026-01-15</lastmod>
</url>
</urlset>
- One file holds at most 50,000 URLs and 50 MB uncompressed. Larger sites publish a sitemap index, a file that lists child sitemaps.
- Special characters in a URL must be escaped for XML, for example
&for an ampersand. - Crawlers find it through a
Sitemap:line in robots.txt, or at a common address such as/sitemap.xml. lastmodis useful only when it is accurate. Google says it ignorespriorityandchangefreq.
Using one to collect data
A sitemap is a quick way to discover item pages such as products or job postings, to size a job before you start, and to pick out the pages that changed since your last visit through lastmod (see data freshness).
Treat it as a hint. It lists pages, not records. It can be incomplete or stale, some sites stamp every URL with the date of the last deploy, and many sites have no sitemap at all. Listing a URL is not permission to fetch it, so robots.txt and the site's terms still apply.
In CleanedWeb
CleanedWeb builds an API from a collection page, the public list page you give it, rather than from a sitemap. A sitemap can still help you choose that page, or see how many item pages a site has. This site publishes its own at /sitemap.xml.
Tools and docs
- Sitemap checker Find a site's sitemap and check its URLs, child sitemaps, duplicates and lastmod dates.
- robots.txt tester Check whether the URLs a sitemap lists are open to your crawler.
- Choosing a source Pick the list page an API should read.
Related terms
-
robots.txt
The file where a site tells crawlers what they may fetch, and what it does not do.
-
Data freshness
How current your collected data is, how to choose a refresh interval and detect changes.
-
Pagination
How lists are split into pages and how to collect every record without gaps or repeats.
-
llms.txt
A proposed Markdown file that gives AI systems a short guide to a site's best pages.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.