CleanedWeb
Markdown

Idempotency and recovery

Save an idempotency key with each intended API execution. If a response is lost, the same key and request return the original run instead of starting another one.

On this page

POST /v1/sources/{source_id}/run, /runs, and /stream all create runs. Customer workspace calls to /run and /stream require the Idempotency-Key header; send it on /runs too so a lost response can be recovered. These rules describe those execution endpoints; builds, key creation, checkout, and other operations have their own contracts.

Save the request before sending#

Use a UUID or another unique string containing 1–200 letters, numbers, periods, underscores, colons, or hyphens. Save it in your job record together with:

  • The workspace and source identifier.
  • The full definition version.
  • The execution endpoint.
  • The exact input values and run options (max_items, timeout_seconds).

For the documentation examples, load this saved value into CLEANEDWEB_IDEMPOTENCY_KEY. Keep it stable across an interrupted attempt. Generating a new key every time a retry function runs creates new work.

HTTP
Idempotency-Key: 72ad1b16-20ad-4ef4-a578-d8b893601d7e

This key is illustrative. Generate your own for a new execution. The connection details supplied by the console also include an example key. Fetching those details again creates a new example key, so retain the one you actually used.

What a key identifies#

Within an authenticated workspace, a key binds the source, resolved definition version, input, selected output fields, max_items, and timeout_seconds. Object property order does not create a new identity, and neither wait_seconds nor nulls changes it. Changing a limit, adding an input, or selecting another version changes the request.

The table shows /run. With the same key, /runs answers with the same run object and /stream streams the same run.

Situation Behavior
New key and valid request Starts a new run.
Same key and same request, run still queued or running Returns 202 with the same run object.
Same key and same request, run ended Returns the run's final body. The source is not collected or charged again.
Same key and same request, run failed Returns the run's saved error for 7 days. The source is not started again; use a new key to run again.
Same key with a changed request Returns 409 with delivery_identity_conflict.
Missing or invalid key Returns 422 with idempotency_key_required.

An expected_version check still applies before replay. If the active definition has changed, a request can return a version conflict. Preserve the original identity and use recorded run details or support to resolve it; do not silently update the version on the saved request.

Recover a lost response#

  1. Keep the original key, source, version, endpoint, inputs, and run options. A timeout does not tell you whether the run completed.
  2. If you have the run identifier (the 202 body, its Location header, or delivery.runId), read GET /v1/runs/{run_id}/result. It answers 202 while the run continues, then its final body or error.
  3. If the run identifier is unavailable, resend the same request with the original key. It returns that same run, in whichever state it has reached.
  4. While the answer is 202, wait Retry-After (or poll_after_seconds) between reads. Use POST /v1/runs/{run_id}/abort to stop a run you no longer need.
  5. Validate the recovered response's version, records, and summary as you would the initial response.

Use a bounded recovery policy and stop when the state remains uncertain. Do not generate replacement keys in a loop. A new key starts the source again and creates new usage.

After a failed run#

A run that ended in failure answers with its error, including 5xx statuses, and that error is replayed to every request with the same key for 7 days. Retrying the same key cannot change it.

Answer Meaning Next request
An error whose detail has error_code run_failed or failed_run_replayed, with run_id Final: the run failed (failed_run_replayed when the key was replayed) New key
Without error_code (older deployments): the key's run answers with an error status and an error object (GET /v1/runs/{run_id}/result, or the same request resent) Final: the run failed New key
409 delivery_identity_conflict Final: the key belongs to a different request New key
Timeout, lost connection, or no response Unknown: the run may have started Same key
5xx from a gateway or storage fault, 401, or a text-only error with no run behind it Unknown or not started Same key, with bounded backoff
202, or a read that shows the run still running Not failed Same key; read /result

A failed run marks its error so you need not read the run to classify it. The saved detail is unchanged, with run_id, error_code and replayed (boolean) added, and a Location: /v1/runs/{run_id} header names the run. error_code is failed_run_replayed when the error answers a replay of the key (resending the request or reading /result) and run_failed otherwise. Treat either value as final and use a new key for the next run. An error without error_code, such as a timeout, a gateway fault, or a deployment that predates the marker, does not show the outcome: keep the key and read GET /v1/runs/{run_id}/result (or resend the same request); an error status with an error object there is final.

When in doubt, read the run before starting another: a new key starts the source again and creates new usage.

Keep delivery evidence#

Store delivery.runId and delivery.recordCount with the request identity and output. delivery.chargeReceiptId is the same identifier as runId: a run's Units are recorded under the run, in run.units_charged. It is separate from a credit-purchase receipt or invoice.

For review, provide the source identifier, definition version, run and trace identifiers, error code, and request timing. Never share API credentials. See errors and retries and Run an API.

Updated Get help
Search across the documentation · Esc to close