CleanedWeb
Markdown

Core concepts

CleanedWeb turns a set of records on a website into an API that your application can call. You review the API before you use it, and you can pin the version your code expects. These are the terms you will meet along the way.

On this page

Sources and collections#

A source identifies the website data your API collects. Its collection describes the set of records you want: products in a category, jobs matching a search, or articles in a section. One website can contain many collections with different fields and access requirements.

Start with a specific public URL; inspection discovers its collections and fields. A category filter, location, or date range in the URL is part of the scope. Choosing a source does not establish that every page on the website is accessible or supported.

Records, fields, and schemas#

A record represents one item. Fields describe it: a product name, price, currency, or source URL. The output schema defines the fields and types your accepted API returns. It is specific to that API.

Keep these distinctions when consuming data:

Value Meaning for your application
A populated field A value was returned; validate its type and meaning.
A missing or null field The value is unavailable under the schema; do not manufacture a default fact.
A detail field Its availability may depend on optional detail-page collection.
An example record A sample for review; it does not establish current source content or full coverage.

Review fields and schemas before making a field mandatory downstream.

Builds and definition versions#

Inspection discovers possible collections and fields. A build validates a selected collection and produces the saved API. A preview, an unfinished build, and a validated API are distinct stages.

A definition version identifies immutable collection rules and their schema. Keep that version with your integration configuration. expected_version checks that the active definition still matches the one you reviewed. A version identifies the rules, not a frozen copy of the website: source content can change between runs.

See build and validate and versions and changes.

Requests, runs, and results#

A request asks the API to execute with specific inputs. A run is that execution; it has its own ID and continues after a request that answers 202. Its result contains records and an execution summary; customer deliveries also carry the run ID, which locates both the saved result and its charges.

An idempotency key identifies one intended execution. Keep it with the request before sending. Replaying a request with the same identity returns the same run rather than starting another; a new deliberate collection uses a new key. See idempotency and recovery.

Bounded and partial output#

A request can collect only a selected part of a website. A run ends at max_items, at timeout_seconds, or when the workspace balance or usage budget runs out; run.stop_reason reports why it ended. Detail enrichment uses the input the API publishes: enrich_records (true or false) on newer APIs, or detail_limit on APIs created before it.

A bounded result can pass its selected scope without collecting the entire catalog. A partial result signals that some expected work did not complete. Always evaluate summary.status, summary.within_scope_passed, run.stop_reason, and catalog_complete with the records.

Continue with the quickstart for a first request or the run reference for the shared contract.

Updated Get help
Search across the documentation · Esc to close