Run an API
Execute an accepted source definition and receive structured records with an execution summary. Send requests to https://api.cleanedweb.com. Copy your source identifier and full definition version from its connection details in the console.
On this page
POST /v1/sources/{source_id}/run
Every request creates a run. A run that ends within the request's wait answers with its records; a longer run answers 202 with a run object you read until it ends. This reference describes generated source APIs. To run an API from the public catalog, see run a catalog API. The saved request and OpenAPI contract for your API define its supported inputs and record schema.
Before you send a request#
Create a server credential with the run scope and access to the source. Keep it in your server's secret store. Add read to retrieve connection details and results to retrieve retained run results. See authentication.
Save one unique idempotency key with the source, definition version, input, and run options before sending the request. Reuse that exact identity when recovering the same request. Create a new key for each intentional new run.
Headers#
| Header | Value |
|---|---|
Authorization |
Bearer <API_KEY> |
Content-Type |
application/json |
Idempotency-Key |
Required for customer workspace execution. A saved string of 1–200 characters using letters, numbers, ., _, :, or -. |
The API origin is https://api.cleanedweb.com for every workspace. It is separate from the website being collected. Send credentials only to that HTTPS origin. Connection details give paths, and their curl examples read the origin from $CLEANEDWEB_API_BASE; set it to https://api.cleanedweb.com.
Request body#
| Field | Type | Meaning |
|---|---|---|
expected_version |
string | Require the active definition to match this full 64-character lowercase SHA-256 version. Recommended for normal execution. |
definition_version |
string | Select an accepted immutable revision explicitly. Use instead of expected_version. It applies that revision's rules to current source content. |
input |
object | Inputs accepted by this source's generated contract. |
endpoint |
string | Optional; the supported value is search. |
max_items |
integer ≥ 1 | Optional. Stop after this many de-duplicated records. |
timeout_seconds |
integer 60–86400 | Optional, default 10800. The run's own time budget, counted from when it starts. |
wait_seconds |
integer 0–86400 | Optional, default 90 on /run. How long this request waits for the run to end before answering 202. It does not change the run. |
fields |
array of strings | Optional. Return only these published output fields; required and identity fields are always kept. Errors: run_fields_invalid (not a non-empty list of unique names), run_fields_unknown (a name the contract does not publish), run_fields_unsupported (this API's output does not support selection). |
nulls |
boolean | Optional, default false. true sends every published field, with absent values as null. /result and /items take the same view as ?nulls=true. Any other value returns run_nulls_invalid. |
Do not combine expected_version and definition_version, including a null value. An inactive candidate is not an accepted revision. A version selects extraction rules; it does not restore an older copy of the website. A run option outside its range returns 422 with run_option_invalid.
Begin with a small max_items:
umask 077
: "${CLEANEDWEB_IDEMPOTENCY_KEY:?Set the saved key for this request}"
curl --fail-with-body --max-time 120 --write-out '%{http_code}\n' \
"${CLEANEDWEB_API_BASE}/v1/sources/${CLEANEDWEB_SOURCE_ID}/run" \
-H "Authorization: Bearer ${CLEANEDWEB_API_KEY}" \
-H "Idempotency-Key: ${CLEANEDWEB_IDEMPOTENCY_KEY}" \
-H 'Content-Type: application/json' \
--data "{\"expected_version\":\"${CLEANEDWEB_DEFINITION_VERSION}\",\"input\":{},\"max_items\":25}" \
--output response.json
Set the connection variables as described in the quickstart. Store the request key before executing this command. Use a fresh private directory or a unique output filename for each intended run, so an earlier response is not overwritten. Keep the response file private. The 120-second client limit leaves room for the default 90-second wait.
If the printed status is 202, response.json holds the run object, not records: the run continues on the server. After Retry-After seconds, read GET /v1/runs/{run_id}/result (links.result in the body) until it answers 200 or an error. Do not resend the POST to check progress.
A run ends when the source has no more records, at max_items, at timeout_seconds, or when the workspace's Units or monthly usage budget run out. A run never ends on a page count. When it ends for another reason, stop_reason says so: error when a page read failed after earlier pages (the records already saved stay yours), end_not_confirmed when it read only what its definition had proven and nothing showed the source has no more, and scope_end when it ended at the definition's reviewed scope. Search APIs can require a query; use their exact allowed values or constraints. Unknown fields and values outside the accepted contract are rejected.
Detail enrichment#
APIs with a verified detail plan publish one detail input; use the one in your API's generated schema and request example:
enrich_records(boolean) on newer APIs.truereads each returned record's detail page and merges its fields. The schema publishes the default; new APIs default totrue. Detail reads end with the run's time,max_itemsand money; an API that also caps them publishes the cap asx-detail-ceiling, otherwiselimits.max_detailsis null. These APIs rejectdetail_limit.detail_limit(integer) on APIs created beforeenrich_records.0disables detail reads; when omitted, the API's default applies. These APIs rejectenrich_records.
APIs without a detail plan publish neither input.
Responses#
| Outcome | Status | Body |
|---|---|---|
Run ended within wait_seconds |
200 |
Records, summary, and the fields below. |
| Run still queued or running | 202 |
The run object, with headers Location: /v1/runs/{run_id} and Retry-After: 30. |
| Run failed | 4xx or 5xx |
The run's error: its HTTP status, detail, and any Retry-After. See errors and retries. |
A failed run is final for its idempotency key: the same key returns the same error for 7 days and the source is not started again. To run again, send a new key. The error's detail carries run_id, error_code and replayed, and the response has a Location: /v1/runs/{run_id} header. error_code is run_failed for the failure itself and failed_run_replayed when the error answers a replay of the key; either one marks the run as failed, and replayed says which. An error without error_code (a timeout, a lost connection, a gateway error) does not show whether the run failed; keep the key and read GET /v1/runs/{run_id}/result (or resend the same request) before you decide. See idempotency.
A new run waits in the queue (status: queued) while the workspace already has a run in progress in the same lane; it is not rejected for concurrency. Runs started with /runs share one lane; /run, /stream and MCP calls share another, so a long background run does not queue them.
Response fields#
| Field | Meaning |
|---|---|
records |
Array of records conforming to this source's output schema, de-duplicated by identity within the run. |
summary.source_id |
Source that executed. |
summary.definition_version |
Full version used for execution. Must match the requested version. |
summary.status |
completed, bounded, or partial. |
summary.within_scope_passed |
Whether execution passed within its requested scope. |
summary.catalog_complete |
The generated contract reports false; a run does not establish full-catalog coverage. |
summary.records |
Number of records in the run's dataset. |
summary.limits |
Collection and detail limits applied to the run. |
summary.details_selected |
Records selected for detail enrichment. |
summary.details_complete |
Selected details that completed. |
summary.details_not_selected |
Records outside the selected detail budget. |
summary.record_enrichment |
On APIs with enrich_records: requested, records_enriched, records_not_enriched, detail_reads, detail_reads_incomplete, detail_ceiling and reason. |
metadata.definition_version |
Execution version; must agree with the summary and requested version. |
artifacts |
Additional execution artifacts. Preserve relevant diagnostics alongside the result. |
traceId |
Trace identifier when available; it may be null. |
run |
The run object as it ended. |
delivery |
Present for a customer workspace run: runId, chargeReceiptId (the same identifier as runId), and recordCount. |
The generated contract can include additional summary and metadata fields. Keep the complete response so you can review missing fields, collection stop reasons, and other source-specific evidence.
The run object#
| Field | Meaning |
|---|---|
run_id |
32 lowercase hexadecimal characters. |
surface, source_id, definition_version |
How the run was started and what it executes. |
status |
queued, running, finished, failed, or aborted. |
stop_reason |
source_exhausted, scope_end, series_unfilled, end_not_confirmed, max_items, timeout, budget_exhausted, aborted, blocked, error, or null while running. |
created_at, started_at, finished_at |
Timestamps; null until reached. |
limits |
max_items and timeout_seconds applied to the run. |
progress |
pages, items (committed records), and duplicates (repeated records dropped). |
units_charged |
Units charged so far; a run is charged per committed page. |
abort_requested |
Whether an abort was requested. |
error |
On failure: code, status, detail, and optional retry_after. |
poll_after_seconds |
Present while the run is queued or running. |
links |
self, items, result, and abort paths. |
A run stopped by max_items, timeout, budget_exhausted, or an abort reports summary.status as bounded when its scope passed, otherwise partial.
Read and control a run#
| Request | Answer |
|---|---|
GET /v1/runs |
The caller's runs, newest first: {runs, next_cursor}. Filters source_id and status; cursor (the opaque next_cursor string of the previous page), and limit (default 50). A bad query returns run_list_query_invalid. |
GET /v1/runs/{run_id} |
The run object. |
GET /v1/runs/{run_id}/result |
200 with the final body once the run ended, 202 with the run object while it runs, or the run's error. |
GET /v1/runs/{run_id}/items |
Records committed so far, readable while the run runs: run_id, items, count, next_cursor, and run_status. |
POST /v1/runs/{run_id}/abort |
202 with the run object while it stops; 200 if it had already ended. Records committed so far stay readable. |
GET /v1/sources/{source_id}/runs/{run_id} |
The run's recorded details: run, results, billing and recovery. Requires results. |
GET /v1/sources/{source_id}/runs/{run_id}/result |
Same as /v1/runs/{run_id}/result for a run of that source. Requires results. |
The GET routes accept a key with run or results scope; abort requires run. A 404 means the run was not found or is inaccessible. Reading a run never collects the website again.
/items takes cursor (a string; the first page is 0), limit (default 100), and format (json or jsonl). Pass the returned next_cursor, a string, as cursor to continue; it is null once the run has ended and every record was read. With jsonl, each line is one record. Both formats carry X-Run-Status and, while more can follow, X-Next-Cursor headers. A bad query returns run_items_query_invalid.
Start without waiting#
POST /v1/sources/{source_id}/runs
Takes the same headers and body as /run, with wait_seconds defaulting to 0. It answers with the run object and a Location header: usually 202, or 200 when the run has already ended, including a failed run. A replayed key adds Idempotent-Replayed: true. Read records from /items and the final body, or the failed run's error, from /result.
Stream records#
POST /v1/sources/{source_id}/stream
Takes the same headers and body as /run and answers newline-delimited JSON. Every line has an increasing sequence.
| Line | Meaning |
|---|---|
{"type":"progress","stage":"running","run":{...}} |
First line, with the run object. |
{"type":"records","records":[...]} |
Records as pages are committed. |
{"type":"progress","stage":"heartbeat"} |
After 10 seconds without records. |
{"type":"done", ...} |
Final line: summary, metadata, artifacts, traceId, run, delivery when present, and elapsedMs. |
{"type":"error","statusCode":..., ...} |
Final line for a failed run: its status, error detail, and elapsedMs. |
The stream has no fixed time limit. Closing it leaves the run running; read it with GET /v1/runs/{run_id}.
Accept or review the result#
Before downstream processing, require both version fields to match your requested version, summary.within_scope_passed to be true, and summary.status to be completed or bounded. Validate the records against your saved schema and required fields. Compare summary.records with the records array.
A bounded result can meet the request while leaving later pages or detail records uncollected; run.stop_reason says why the run ended. Preserve a partial response for review; do not replace it with an empty dataset. HTTP 200 alone does not establish a usable result. See pagination and output.
For a lost response, read the run if you have its identifier; otherwise follow idempotency and recovery. For rejected requests, see errors and retries.