Glossary
robots.txt
robots.txt is a plain-text file at the root of a website (/robots.txt) in which the owner tells crawlers which paths they may or may not fetch. The format is standardized as the Robots Exclusion Protocol, RFC 9309.
What the file says
User-agent: *
Disallow: /cart
Disallow: /search
User-agent: ExampleBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
- A group starts with one or more
User-agentlines and applies to the crawlers with those names.*matches any crawler without a group of its own. DisallowandAllowmatch the start of a URL path. When several rules match, the most specific one, the longest, wins.Sitemappoints to the site's sitemap.Crawl-delayis not part of RFC 9309. Some crawlers honour it and others, including Google, ignore it.
What it is not
- Not access control. It is a request that well-behaved crawlers follow. Anyone can read it, so never use it to hide something private; listing a path points people to it.
- Not the whole rulebook. A site's terms of use, copyright and privacy law apply whatever robots.txt says.
- Not about indexing. It controls fetching. A blocked page can still appear in search results if other sites link to it.
If the file is missing and the server answers with a 4xx status, RFC 9309 lets crawlers treat everything as allowed. If the server cannot be reached (5xx), crawlers should assume nothing may be fetched.
In a data pipeline
Read a site's robots.txt before you point a collector at it, and read it again from time to time, because it changes. Follow the group for your crawler's name, and combine it with a gentle request rate (see rate limiting). If the pages you need are disallowed, ask the owner for an API or a data export rather than looking for a way around the rule.
CleanedWeb reads public pages only. Which sites to build from is your decision, so check a site's robots.txt and terms with the tester below before you start a build.
Tools and docs
- robots.txt tester Pick a crawler and see which rule, on which line, allows or blocks each URL.
- Website data checker See a site's robots.txt rules next to its sitemap and structured data.
- Choosing a source Start from a public list page, not a whole site.
Related terms
-
Sitemap
The XML file that lists a site's URLs, and how to use it to find pages and spot changes.
-
llms.txt
A proposed Markdown file that gives AI systems a short guide to a site's best pages.
-
HTTP 403 Forbidden
The server understood the request and refuses it. Why it happens and how to respond.
-
Rate limiting
How servers cap request rates, and how to throttle and back off when collecting data.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.