CleanedWeb
Markdown

Run an API

Execute an accepted source definition and receive structured records with an execution summary. Send requests to https://api.cleanedweb.com. Copy your source identifier and full definition version from its connection details in the console.

On this page
HTTP
POST /v1/sources/{source_id}/run

Every request creates a run. A run that ends within the request's wait answers with its records; a longer run answers 202 with a run object you read until it ends. This reference describes generated source APIs. To run an API from the public catalog, see run a catalog API. The saved request and OpenAPI contract for your API define its supported inputs and record schema.

Before you send a request#

Create a server credential with the run scope and access to the source. Keep it in your server's secret store. Add read to retrieve connection details and results to retrieve retained run results. See authentication.

Save one unique idempotency key with the source, definition version, input, and run options before sending the request. Reuse that exact identity when recovering the same request. Create a new key for each intentional new run.

Headers#

Header Value
Authorization Bearer <API_KEY>
Content-Type application/json
Idempotency-Key Required for customer workspace execution. A saved string of 1–200 characters using letters, numbers, ., _, :, or -.

The API origin is https://api.cleanedweb.com for every workspace. It is separate from the website being collected. Send credentials only to that HTTPS origin. Connection details give paths, and their curl examples read the origin from $CLEANEDWEB_API_BASE; set it to https://api.cleanedweb.com.

Request body#

Field Type Meaning
expected_version string Require the active definition to match this full 64-character lowercase SHA-256 version. Recommended for normal execution.
definition_version string Select an accepted immutable revision explicitly. Use instead of expected_version. It applies that revision's rules to current source content.
input object Inputs accepted by this source's generated contract.
endpoint string Optional; the supported value is search.
max_items integer ≥ 1 Optional. Stop after this many de-duplicated records.
timeout_seconds integer 60–86400 Optional, default 10800. The run's own time budget, counted from when it starts.
wait_seconds integer 0–86400 Optional, default 90 on /run. How long this request waits for the run to end before answering 202. It does not change the run.
fields array of strings Optional. Return only these published output fields; required and identity fields are always kept. Errors: run_fields_invalid (not a non-empty list of unique names), run_fields_unknown (a name the contract does not publish), run_fields_unsupported (this API's output does not support selection).
nulls boolean Optional, default false. true sends every published field, with absent values as null. /result and /items take the same view as ?nulls=true. Any other value returns run_nulls_invalid.

Do not combine expected_version and definition_version, including a null value. An inactive candidate is not an accepted revision. A version selects extraction rules; it does not restore an older copy of the website. A run option outside its range returns 422 with run_option_invalid.

Begin with a small max_items:

Shell
umask 077
: "${CLEANEDWEB_IDEMPOTENCY_KEY:?Set the saved key for this request}"

curl --fail-with-body --max-time 120 --write-out '%{http_code}\n' \
  "${CLEANEDWEB_API_BASE}/v1/sources/${CLEANEDWEB_SOURCE_ID}/run" \
  -H "Authorization: Bearer ${CLEANEDWEB_API_KEY}" \
  -H "Idempotency-Key: ${CLEANEDWEB_IDEMPOTENCY_KEY}" \
  -H 'Content-Type: application/json' \
  --data "{\"expected_version\":\"${CLEANEDWEB_DEFINITION_VERSION}\",\"input\":{},\"max_items\":25}" \
  --output response.json

Set the connection variables as described in the quickstart. Store the request key before executing this command. Use a fresh private directory or a unique output filename for each intended run, so an earlier response is not overwritten. Keep the response file private. The 120-second client limit leaves room for the default 90-second wait.

If the printed status is 202, response.json holds the run object, not records: the run continues on the server. After Retry-After seconds, read GET /v1/runs/{run_id}/result (links.result in the body) until it answers 200 or an error. Do not resend the POST to check progress.

A run ends when the source has no more records, at max_items, at timeout_seconds, or when the workspace's Units or monthly usage budget run out. A run never ends on a page count. When it ends for another reason, stop_reason says so: error when a page read failed after earlier pages (the records already saved stay yours), end_not_confirmed when it read only what its definition had proven and nothing showed the source has no more, and scope_end when it ended at the definition's reviewed scope. Search APIs can require a query; use their exact allowed values or constraints. Unknown fields and values outside the accepted contract are rejected.

Detail enrichment#

APIs with a verified detail plan publish one detail input; use the one in your API's generated schema and request example:

  • enrich_records (boolean) on newer APIs. true reads each returned record's detail page and merges its fields. The schema publishes the default; new APIs default to true. Detail reads end with the run's time, max_items and money; an API that also caps them publishes the cap as x-detail-ceiling, otherwise limits.max_details is null. These APIs reject detail_limit.
  • detail_limit (integer) on APIs created before enrich_records. 0 disables detail reads; when omitted, the API's default applies. These APIs reject enrich_records.

APIs without a detail plan publish neither input.

Responses#

Outcome Status Body
Run ended within wait_seconds 200 Records, summary, and the fields below.
Run still queued or running 202 The run object, with headers Location: /v1/runs/{run_id} and Retry-After: 30.
Run failed 4xx or 5xx The run's error: its HTTP status, detail, and any Retry-After. See errors and retries.

A failed run is final for its idempotency key: the same key returns the same error for 7 days and the source is not started again. To run again, send a new key. The error's detail carries run_id, error_code and replayed, and the response has a Location: /v1/runs/{run_id} header. error_code is run_failed for the failure itself and failed_run_replayed when the error answers a replay of the key; either one marks the run as failed, and replayed says which. An error without error_code (a timeout, a lost connection, a gateway error) does not show whether the run failed; keep the key and read GET /v1/runs/{run_id}/result (or resend the same request) before you decide. See idempotency.

A new run waits in the queue (status: queued) while the workspace already has a run in progress in the same lane; it is not rejected for concurrency. Runs started with /runs share one lane; /run, /stream and MCP calls share another, so a long background run does not queue them.

Response fields#

Field Meaning
records Array of records conforming to this source's output schema, de-duplicated by identity within the run.
summary.source_id Source that executed.
summary.definition_version Full version used for execution. Must match the requested version.
summary.status completed, bounded, or partial.
summary.within_scope_passed Whether execution passed within its requested scope.
summary.catalog_complete The generated contract reports false; a run does not establish full-catalog coverage.
summary.records Number of records in the run's dataset.
summary.limits Collection and detail limits applied to the run.
summary.details_selected Records selected for detail enrichment.
summary.details_complete Selected details that completed.
summary.details_not_selected Records outside the selected detail budget.
summary.record_enrichment On APIs with enrich_records: requested, records_enriched, records_not_enriched, detail_reads, detail_reads_incomplete, detail_ceiling and reason.
metadata.definition_version Execution version; must agree with the summary and requested version.
artifacts Additional execution artifacts. Preserve relevant diagnostics alongside the result.
traceId Trace identifier when available; it may be null.
run The run object as it ended.
delivery Present for a customer workspace run: runId, chargeReceiptId (the same identifier as runId), and recordCount.

The generated contract can include additional summary and metadata fields. Keep the complete response so you can review missing fields, collection stop reasons, and other source-specific evidence.

The run object#

Field Meaning
run_id 32 lowercase hexadecimal characters.
surface, source_id, definition_version How the run was started and what it executes.
status queued, running, finished, failed, or aborted.
stop_reason source_exhausted, scope_end, series_unfilled, end_not_confirmed, max_items, timeout, budget_exhausted, aborted, blocked, error, or null while running.
created_at, started_at, finished_at Timestamps; null until reached.
limits max_items and timeout_seconds applied to the run.
progress pages, items (committed records), and duplicates (repeated records dropped).
units_charged Units charged so far; a run is charged per committed page.
abort_requested Whether an abort was requested.
error On failure: code, status, detail, and optional retry_after.
poll_after_seconds Present while the run is queued or running.
links self, items, result, and abort paths.

A run stopped by max_items, timeout, budget_exhausted, or an abort reports summary.status as bounded when its scope passed, otherwise partial.

Read and control a run#

Request Answer
GET /v1/runs The caller's runs, newest first: {runs, next_cursor}. Filters source_id and status; cursor (the opaque next_cursor string of the previous page), and limit (default 50). A bad query returns run_list_query_invalid.
GET /v1/runs/{run_id} The run object.
GET /v1/runs/{run_id}/result 200 with the final body once the run ended, 202 with the run object while it runs, or the run's error.
GET /v1/runs/{run_id}/items Records committed so far, readable while the run runs: run_id, items, count, next_cursor, and run_status.
POST /v1/runs/{run_id}/abort 202 with the run object while it stops; 200 if it had already ended. Records committed so far stay readable.
GET /v1/sources/{source_id}/runs/{run_id} The run's recorded details: run, results, billing and recovery. Requires results.
GET /v1/sources/{source_id}/runs/{run_id}/result Same as /v1/runs/{run_id}/result for a run of that source. Requires results.

The GET routes accept a key with run or results scope; abort requires run. A 404 means the run was not found or is inaccessible. Reading a run never collects the website again.

/items takes cursor (a string; the first page is 0), limit (default 100), and format (json or jsonl). Pass the returned next_cursor, a string, as cursor to continue; it is null once the run has ended and every record was read. With jsonl, each line is one record. Both formats carry X-Run-Status and, while more can follow, X-Next-Cursor headers. A bad query returns run_items_query_invalid.

Start without waiting#

HTTP
POST /v1/sources/{source_id}/runs

Takes the same headers and body as /run, with wait_seconds defaulting to 0. It answers with the run object and a Location header: usually 202, or 200 when the run has already ended, including a failed run. A replayed key adds Idempotent-Replayed: true. Read records from /items and the final body, or the failed run's error, from /result.

Stream records#

HTTP
POST /v1/sources/{source_id}/stream

Takes the same headers and body as /run and answers newline-delimited JSON. Every line has an increasing sequence.

Line Meaning
{"type":"progress","stage":"running","run":{...}} First line, with the run object.
{"type":"records","records":[...]} Records as pages are committed.
{"type":"progress","stage":"heartbeat"} After 10 seconds without records.
{"type":"done", ...} Final line: summary, metadata, artifacts, traceId, run, delivery when present, and elapsedMs.
{"type":"error","statusCode":..., ...} Final line for a failed run: its status, error detail, and elapsedMs.

The stream has no fixed time limit. Closing it leaves the run running; read it with GET /v1/runs/{run_id}.

Accept or review the result#

Before downstream processing, require both version fields to match your requested version, summary.within_scope_passed to be true, and summary.status to be completed or bounded. Validate the records against your saved schema and required fields. Compare summary.records with the records array.

A bounded result can meet the request while leaving later pages or detail records uncollected; run.stop_reason says why the run ended. Preserve a partial response for review; do not replace it with an empty dataset. HTTP 200 alone does not establish a usable result. See pagination and output.

For a lost response, read the run if you have its identifier; otherwise follow idempotency and recovery. For rejected requests, see errors and retries.

Updated Get help
Search across the documentation · Esc to close