CleanedWeb

Glossary

Pagination

Pagination is splitting a long list of results into pages, so a server returns a few records at a time instead of all of them at once. To collect a whole list, a client follows the pages until there are no more.

Common styles

What goes wrong when you collect data

A record added or removed between two page reads moves everything after it, so you can miss one record or read another twice. Deduplicate on a stable identifier from the source, never on position in the list.

Stop when a page comes back empty or has no next cursor, not after a fixed number of pages, and set an upper limit so a bug in your loop cannot read forever. Check what a site does past the last page: it may return an empty list, an error, or the last page again.

Every page is a request, so a long list is also a question of rate limiting. Pause between pages and honour any Retry-After delay.

In CleanedWeb

A CleanedWeb API follows the site's own pagination for you. A run collects until the source has no more records or a bound ends it: max_items, timeout_seconds, or the point where the workspace's Units or monthly usage budget can pay for no further page.

These are run options, not page numbers. Do not add page, offset or cursor inputs unless your API's contract lists them, and remember that repeating a run is a new collection, not a request for the next page. run.stop_reason says which bound ended the run, and a bounded run is not a promise to cover a whole website. To read a long run's records while it is going, follow next_cursor on GET /v1/runs/{run_id}/items.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.