CleanedWeb

Glossary

Data freshness

Data freshness is how current the data you hold is compared with its source. It is the gap between when a value last changed on the website and when your pipeline last observed it.

Choose a refresh interval

Set it from what the data is used for, not from what is possible. A price used for repricing may need hourly observations, job postings often need only daily ones, and a reference table may need weekly ones. Every refresh costs requests, time and, on a metered API, money, so a shorter interval is not automatically better. Keep the interval gentle enough to respect the source's rate limits.

Detect change instead of re-reading everything

None of these is proof. Validators and dates can be wrong, and a page can change without updating them, so occasionally re-read in full.

Record when you observed it

Store an observation time with every record. If change over time matters, keep each observation rather than overwriting the latest value. Be careful with deletions: a record missing from a bounded or partial run may simply be outside that run's scope, so do not mark it removed on that evidence alone (see 404 Not Found).

In CleanedWeb

Each run reads the website as it is at that moment, and reading a saved result later never collects the site again. There is no historical data: to build a history, store each response with its observation time. CleanedWeb does not schedule your collection. You call the API from your own cron job or workflow runner and choose the frequency from how fresh the data must be.

A version pins the contract, not the website. CleanedWeb also checks the catalog APIs it publishes once a day to catch breakage; that is a health check, not a refresh of your data.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.