Glossary
Data freshness
Data freshness is how current the data you hold is compared with its source. It is the gap between when a value last changed on the website and when your pipeline last observed it.
Choose a refresh interval
Set it from what the data is used for, not from what is possible. A price used for repricing may need hourly observations, job postings often need only daily ones, and a reference table may need weekly ones. Every refresh costs requests, time and, on a metered API, money, so a shorter interval is not automatically better. Keep the interval gentle enough to respect the source's rate limits.
Detect change instead of re-reading everything
- Conditional requests. When a server sends
ETagorLast-Modified, ask again withIf-None-MatchorIf-Modified-Since. An unchanged page answers304 Not Modifiedwithout a body. - Sitemap
lastmod. A cheap hint about which pages to revisit (see sitemap), when the site keeps it accurate. - Record-level comparison. Match records by a stable identifier and compare the fields you care about. Hash those fields, not the page's HTML, because pages change on every request through timestamps and ads.
None of these is proof. Validators and dates can be wrong, and a page can change without updating them, so occasionally re-read in full.
Record when you observed it
Store an observation time with every record. If change over time matters, keep each observation rather than overwriting the latest value. Be careful with deletions: a record missing from a bounded or partial run may simply be outside that run's scope, so do not mark it removed on that evidence alone (see 404 Not Found).
In CleanedWeb
Each run reads the website as it is at that moment, and reading a saved result later never collects the site again. There is no historical data: to build a history, store each response with its observation time. CleanedWeb does not schedule your collection. You call the API from your own cron job or workflow runner and choose the frequency from how fresh the data must be.
A version pins the contract, not the website. CleanedWeb also checks the catalog APIs it publishes once a day to catch breakage; that is a health check, not a refresh of your data.
Tools and docs
- Schedule API runs Collect on a recurring schedule with a native schedule or your own scheduler.
- Store and deduplicate Keep each response with the evidence needed to compare runs.
- Sitemap checker Check whether a site's lastmod dates look usable as a change signal.
Related terms
-
Sitemap
The XML file that lists a site's URLs, and how to use it to find pages and spot changes.
-
Pagination
How lists are split into pages and how to collect every record without gaps or repeats.
-
Rate limiting
How servers cap request rates, and how to throttle and back off when collecting data.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.