Glossary
Pagination
Pagination is splitting a long list of results into pages, so a server returns a few records at a time instead of all of them at once. To collect a whole list, a client follows the pages until there are no more.
Common styles
- Page numbers.
?page=3asks for the third page. Simple, but if the list changes while you read it, records shift between pages. - Offset and limit.
?offset=100&limit=50skips 100 records and returns 50. It has the same drift problem, and very deep offsets can be slow on the server. - Cursors. Each response carries a token, such as
next_cursor, that marks where the next page starts. A cursor stays correct when records are added or removed, which is why APIs prefer it. - Next links and infinite scroll. On a website the next page is a link or a button, or more records load as you scroll, often from a background JSON request.
What goes wrong when you collect data
A record added or removed between two page reads moves everything after it, so you can miss one record or read another twice. Deduplicate on a stable identifier from the source, never on position in the list.
Stop when a page comes back empty or has no next cursor, not after a fixed number of pages, and set an upper limit so a bug in your loop cannot read forever. Check what a site does past the last page: it may return an empty list, an error, or the last page again.
Every page is a request, so a long list is also a question of rate limiting. Pause between pages and honour any Retry-After delay.
In CleanedWeb
A CleanedWeb API follows the site's own pagination for you. A run collects until the source has no more records or a bound ends it: max_items, timeout_seconds, or the point where the workspace's Units or monthly usage budget can pay for no further page.
These are run options, not page numbers. Do not add page, offset or cursor inputs unless your API's contract lists them, and remember that repeating a run is a new collection, not a request for the next page. run.stop_reason says which bound ended the run, and a bounded run is not a promise to cover a whole website. To read a long run's records while it is going, follow next_cursor on GET /v1/runs/{run_id}/items.
Tools and docs
- Pagination and output Collection bounds, stop reasons and how to read a partial run.
- Store and deduplicate Choose a stable key so repeated records do not double up.
- JSON to CSV converter Flatten the pages you collected into one spreadsheet-ready file.
Related terms
-
Rate limiting
How servers cap request rates, and how to throttle and back off when collecting data.
-
Data freshness
How current your collected data is, how to choose a refresh interval and detect changes.
-
HTTP 404 Not Found
There is nothing at that URL. How to tell a removed item from a bug in your pipeline.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.