Glossary
Web data glossary
Short, plain explanations of the terms and errors you meet when you collect data from websites and APIs. Each entry says what the thing is, why it matters for a data pipeline, and what to do about it, and points to the free tool or documentation page that helps.
Web data terms
The vocabulary of collecting data from websites and APIs: how lists are paged, what sites publish for machines, and how versions and freshness work.
-
Pagination
How lists are split into pages and how to collect every record without gaps or repeats.
-
Rate limiting
How servers cap request rates, and how to throttle and back off when collecting data.
-
Structured data
JSON-LD and schema.org markup: what sites publish for machines and where it falls short.
-
robots.txt
The file where a site tells crawlers what they may fetch, and what it does not do.
-
Sitemap
The XML file that lists a site's URLs, and how to use it to find pages and spot changes.
-
llms.txt
A proposed Markdown file that gives AI systems a short guide to a site's best pages.
-
JSON Schema
A way to describe the shape of JSON records and validate each response against it.
-
API versioning
How APIs label changes, why you pin a version, and how major versions, deprecation and sunset work.
-
Web scraping vs API
Scraping pages versus calling an API: when each fits and what scraping costs to maintain.
-
Data freshness
How current your collected data is, how to choose a refresh interval and detect changes.
HTTP errors when collecting web data
What each status code means for a scraper or data pipeline, why it happens, and how to respond without making it worse.
-
HTTP 403 Forbidden
The server understood the request and refuses it. Why it happens and how to respond.
-
HTTP 429 Too Many Requests
You sent requests too fast. How to read Retry-After, back off and slow down.
-
HTTP 503 Service Unavailable
The server cannot answer right now. How to retry with backoff and not read it as no data.
-
HTTP 404 Not Found
There is nothing at that URL. How to tell a removed item from a bug in your pipeline.
-
HTTP 401 Unauthorized
The request lacks valid credentials. How it differs from 403 and how to fix it.
-
Cloudflare challenge page
The interstitial a site shows instead of its content. How to recognise it and respond.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.