CleanedWeb

Glossary

robots.txt

robots.txt is a plain-text file at the root of a website (/robots.txt) in which the owner tells crawlers which paths they may or may not fetch. The format is standardized as the Robots Exclusion Protocol, RFC 9309.

What the file says

User-agent: *
Disallow: /cart
Disallow: /search

User-agent: ExampleBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

What it is not

If the file is missing and the server answers with a 4xx status, RFC 9309 lets crawlers treat everything as allowed. If the server cannot be reached (5xx), crawlers should assume nothing may be fetched.

In a data pipeline

Read a site's robots.txt before you point a collector at it, and read it again from time to time, because it changes. Follow the group for your crawler's name, and combine it with a gentle request rate (see rate limiting). If the pages you need are disallowed, ask the owner for an API or a data export rather than looking for a way around the rule.

CleanedWeb reads public pages only. Which sites to build from is your decision, so check a site's robots.txt and terms with the tester below before you start a build.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.