# Run a catalog API

Run a [published catalog API](/docs/catalog-apis/) from your server, read its records, and recover safely when a response is lost. A run uses the credits of your workspace.

```http
POST /v1/catalog/public/apis/{api_id}/endpoints/{endpoint_id}/run
```

Use the `url` from the listing's `contract.json` on the catalog site, or the `path` from the live contract (send it to `https://broker.cleanedweb.com`); both already have your API and endpoint identifiers. The server runs the endpoint's published version. You cannot choose the version, the source, or the owner.

## Before you send a request

- Read the contract and decide your `input`. Send every input its `input_schema.required` lists; an empty list means `{}` is valid.
- Create an API key with the `run` scope. Keep it in your server's secret store. See [authentication](/docs/authentication/).
- Make sure the workspace has Units. A run is charged by committed records; see [billing and failure behavior](#errors-and-billing-codes).
- Generate an idempotency key and **save it before you send**. Reuse it only to recover this same request.

Set these variables. The origin is the run origin, without a path. Runs and results are served by `https://broker.cleanedweb.com`, or by the origin your workspace supplies. The read routes under `/api/catalog/` are on `https://app.cleanedweb.com`, which answers `/v1/` with a redirect to the home page.

| Variable | Value |
| --- | --- |
| `CLEANEDWEB_RUN_BASE` | `https://broker.cleanedweb.com` |
| `CLEANEDWEB_API_KEY` | A key with the `run` scope. |
| `CLEANEDWEB_CATALOG_API_ID` | The API's `api_id`, such as `generated_21ac18f87ad65090f8480c76`. |
| `CLEANEDWEB_CATALOG_ENDPOINT_ID` | The endpoint's `endpoint_id`, such as `generated_21ac18f87ad65090f8480c76_search`. |
| `CLEANEDWEB_IDEMPOTENCY_KEY` | The saved key for this one request. |

## Headers

| Header | Value |
| --- | --- |
| `Authorization` | `Bearer <API_KEY>` |
| `Content-Type` | `application/json` |
| `Idempotency-Key` | Required. 1–200 characters: letters, numbers, `.`, `_`, `:`, `-`. |

## Body

| Field | Type | Meaning |
| --- | --- | --- |
| `input` | object | The inputs from the contract. Unknown fields are rejected. |
| `max_items` | integer ≥ 1 | Optional. Stop after this many de-duplicated records. |
| `timeout_seconds` | integer 60–86400 | Optional, default `10800`. The run's own time budget, counted from when it starts running. |
| `wait_seconds` | integer 0–86400 | Optional, default `90`. How long this request waits for the run to end before answering `202`. It does not change the run. |
| `fields`, `nulls` | | Optional output selection and null filling, as in [run an API](/docs/run-api/#request-body). |

A run ends when the source has no more records, or by `max_items`, `timeout_seconds`, or money (see [stop reasons](#stop-reasons)). There is no page or row cap. Where the contract lists `max_pages` or `detail_limit` in `input_schema`, they are optional input limits like any other input; omit them unless you need them. A run option outside its range returns `422` with `run_option_invalid`.

`wait_seconds` is not part of the request identity; changing it does not create a new run. Changing `input`, `max_items` or `timeout_seconds` does.

## Send the request

Start with a small `max_items` while you check the shape of the records. This request asks for 25 records from the Gymshark Products API's "All Socks" collection:

```bash
umask 077
: "${CLEANEDWEB_IDEMPOTENCY_KEY:?Set the saved key for this request}"

curl --fail-with-body --max-time 120 --write-out '%{http_code}\n' \
  "${CLEANEDWEB_RUN_BASE}/v1/catalog/public/apis/${CLEANEDWEB_CATALOG_API_ID}/endpoints/${CLEANEDWEB_CATALOG_ENDPOINT_ID}/run" \
  -H "Authorization: Bearer ${CLEANEDWEB_API_KEY}" \
  -H "Idempotency-Key: ${CLEANEDWEB_IDEMPOTENCY_KEY}" \
  -H 'Content-Type: application/json' \
  --data '{"input":{"collection":"All Socks"},"max_items":25,"timeout_seconds":3600,"wait_seconds":90}' \
  --output response.json
```

Use a fresh private directory or a unique output filename for each run. Keep `response.json` private. The 120-second client limit leaves room for the 90-second wait. A client timeout does not stop the run. A `3xx` status means the request went to the wrong origin and nothing ran; `curl --fail-with-body` does not treat it as an error, so check the printed status.

| Outcome | Status | Body |
| --- | --- | --- |
| Run ended within `wait_seconds` | `200` | Records, `summary`, `metadata`, `run` and `delivery`. See [run an API](/docs/run-api/#response-fields). |
| Run queued or still running | `202` | The run object. Headers `Location: /v1/runs/{run_id}` and `Retry-After`. |
| Run failed | `4xx` or `5xx` | The run's error and its [failed-run marker](#failed-runs). |

A new run waits in the queue (`status: queued`) while the workspace already has a `/run`, `/stream` or MCP call in progress. It is not rejected. Background runs started with `/v1/sources/{source_id}/runs` queue separately.

## Wait for a longer run

On `202` the run continues on the server. Do not send the POST again. After `Retry-After` seconds, read the result with the **same API key and the same `Idempotency-Key`**:

```http
GET /v1/catalog/public/apis/{api_id}/endpoints/{endpoint_id}/result
```

```bash
curl --fail-with-body --max-time 120 --write-out '%{http_code}\n' \
  "${CLEANEDWEB_RUN_BASE}/v1/catalog/public/apis/${CLEANEDWEB_CATALOG_API_ID}/endpoints/${CLEANEDWEB_CATALOG_ENDPOINT_ID}/result" \
  -H "Authorization: Bearer ${CLEANEDWEB_API_KEY}" \
  -H "Idempotency-Key: ${CLEANEDWEB_IDEMPOTENCY_KEY}" \
  --output result.json
```

It answers `202` with the run object while the run goes on, `200` with the final body when it ends, or the run's error. Reading a result never collects the website again. It works after the API is unpublished or its version changes, because the delivery keeps its original endpoint and version. Another workspace, a revoked key, or a different endpoint cannot read it; a key unknown for the endpoint returns `404`, and a key reused for a different endpoint or version returns `409 delivery_identity_conflict`.

With the `run_id` from the `202`, you can also use the run routes:

| Request | Answer |
| --- | --- |
| `GET /v1/runs/{run_id}` | The run object: `status`, `stop_reason`, `progress`, `units_charged`. |
| `GET /v1/runs/{run_id}/items?cursor=0&limit=100` | Records committed so far, readable while the run runs: `items`, `count`, `next_cursor`, `run_status`. |
| `POST /v1/runs/{run_id}/abort` | `202` with the run object while it stops; `200` if it had already ended. Records committed so far stay readable. |

Pass the returned `next_cursor` to `/items` to continue. It is null once the run has ended and every record was read. `format=jsonl` returns one record per line and the next cursor in the `X-Next-Cursor` header. Aborting needs the `run` scope. See [run an API](/docs/run-api/#read-and-control-a-run) for the full run object.

## JavaScript example

This Node.js 20 script sends one run, waits with the same key, and refuses to accept an incomplete result. Keep the key on the server. It never resends the POST.

```js
const base = process.env.CLEANEDWEB_RUN_BASE;          // https://broker.cleanedweb.com
const root = `${base}/v1/catalog/public/apis/${process.env.CLEANEDWEB_CATALOG_API_ID}` +
  `/endpoints/${process.env.CLEANEDWEB_CATALOG_ENDPOINT_ID}`;
const headers = {
  Authorization: `Bearer ${process.env.CLEANEDWEB_API_KEY}`,
  'Idempotency-Key': process.env.CLEANEDWEB_IDEMPOTENCY_KEY, // saved before sending
};
const sleep = (seconds) => new Promise((resolve) => setTimeout(resolve, seconds * 1000));

async function read(response) {
  const body = await response.json().catch(() => null);
  if (response.ok && !body) throw new Error(`HTTP ${response.status} without a JSON body`);
  if (response.status === 202) return { pending: true, body, response };
  if (response.ok) return { body };
  const detail = body?.detail;
  const marker = detail?.error_code;           // run_failed | failed_run_replayed
  if (marker) throw new Error(`run ${detail.run_id} failed (${marker}): ${response.status}`);
  const error = new Error(`HTTP ${response.status} ${detail?.code ?? ''}`.trim());
  error.unsure = response.status >= 500 || response.status === 408; // a gateway error or timeout proves nothing
  throw error;
}

async function run() {
  let outcome = await read(await fetch(`${root}/run`, {
    method: 'POST',
    headers: { ...headers, 'Content-Type': 'application/json' },
    body: JSON.stringify({
      input: { collection: 'All Socks' },
      max_items: 25,
      timeout_seconds: 3600,
      wait_seconds: 90,
    }),
    signal: AbortSignal.timeout(120_000),
    redirect: 'error',                         // a redirect means the wrong origin
  }));
  while (outcome.pending) {
    const wait = Number(outcome.response.headers.get('retry-after')) || outcome.body?.poll_after_seconds || 30;
    await sleep(wait);
    outcome = await read(await fetch(`${root}/result`, { headers, redirect: 'error', signal: AbortSignal.timeout(120_000) }));
  }
  return outcome.body;
}

const result = await run();
const { summary, run: info } = result;
if (!['completed', 'bounded'].includes(summary.status) || !summary.within_scope_passed) {
  throw new Error(`review needed: ${summary.status}, stop_reason ${info.stop_reason}`);
}
console.log(`${result.records.length} records, stop_reason ${info.stop_reason}, run ${result.delivery.runId}`);
```

If `run()` throws with `unsure`, or `fetch` itself throws (a timeout or lost connection), the run may have started. Keep the key and call `/result` with it before you decide; do not create a new key. If the error has `error_code`, the run failed and a new attempt needs a new key.

To read records while a run is going, call `GET ${base}/v1/runs/${runId}/items?cursor=${cursor}` with the `run_id` from the `202` body and follow `next_cursor`.

## Stop reasons

`run.stop_reason` says why a run ended:

| `stop_reason` | `status` | Meaning |
| --- | --- | --- |
| `source_exhausted` | `finished` | The source had no more records. |
| `max_items` | `finished` | Your `max_items` was reached exactly. |
| `timeout` | `finished` | `timeout_seconds` passed. Records already saved stay readable and charged. |
| `budget_exhausted` | `finished` | The workspace's Units or monthly usage budget ran out. |
| `aborted` | `aborted` | You aborted the run. |
| `page_limit` | `finished` | The run reached its page limit; the source may hold more. |
| `blocked` | `failed` | The source refused access. |
| `error` | `failed` | A definition or engine failure, with an error code. |

Saved records stay readable and charged in every state. A run stopped by `max_items`, `timeout`, `budget_exhausted` or an abort reports `summary.status` as `bounded` when its scope passed, otherwise `partial`. `bounded` and `partial` mean the source can hold more records than you received. A `completed` run still does not assert full-site coverage: when the listing's coverage arrays are empty, coverage is unspecified, and `summary.catalog_complete` is `false`.

## Failed runs

A failed run is final for its idempotency key. The same key returns the same error for 7 days (replay) and the source is not started again. To run again, send a new key.

A failed-run error carries a marker in `detail`, plus a header:

| Where | Field | Meaning |
| --- | --- | --- |
| `detail.run_id` | string | The failed run. |
| `detail.error_code` | `run_failed` or `failed_run_replayed` | `run_failed` for the failure itself; `failed_run_replayed` when the answer replays the key (including `/result`). Either marks the run as failed. |
| `detail.replayed` | boolean | Which of the two this is. |
| `Location` | `/v1/runs/{run_id}` | Where to read the failed run. |

An error without `error_code`, such as a timeout, a lost connection or a gateway error, does not show whether the run failed. Keep the key, read `/result`, and decide from that. Records saved before a failure stay readable at `/v1/runs/{run_id}/items`.

## Errors and billing codes

Billing is one Unit per saved record, charged as pages are committed. Missing or unreadable billing configuration fails closed: the request is refused, never run for free.

| Status | Code | Meaning | What to do |
| --- | --- | --- | --- |
| `401` | | No credential, or not signed in. | Supply a valid key. |
| `403` | | The credential is not tied to a workspace account, or lacks permission. | Use a workspace key with the `run` scope. |
| `404` | | API, endpoint or delivery not found. | Re-read the catalog; the API may be unpublished. |
| `402` | `insufficient_units` | The balance cannot fund the run. The error shows `required` and `available`. | Add credit in [Billing](/docs/billing/), then send a **new** key if the run failed. |
| `409` | `usage_budget_exceeded` | The workspace's monthly usage budget cannot cover the request. | Ask the workspace Owner to review Billing. |
| `409` | `delivery_identity_conflict` | The key was reused for another endpoint, version or request. | Use a new key for a new request. |
| `422` | `run_option_invalid` | A run option is outside its range. | Fix the option; do not retry unchanged. |
| `422` | | An input the contract does not accept. | Correct `input` against the contract. |
| `503` | `builder_billing_unavailable` | Billing could not be confirmed, so the run was refused (fail-closed). | Retry later with the same key and request. |
| `503` | `runs_unavailable` | The run store could not answer. | Retry later with the same key. |
| `503` | | Execution or catalog storage unavailable. | Retry later with the same key. |

Treat an unknown code as a failure and keep the response. See [errors and retries](/docs/errors/) for the general rules and [idempotency](/docs/idempotency/#after-a-failed-run) for recovery. A `503` is never an empty result set.

A `budget_exhausted` stop is not an error: the run returns what the credit paid for. To check a charge, compare `run.units_charged`, `delivery.recordCount` and the Billing record for the `run_id`.

## Accept the result

Before downstream processing, check `summary.status` is `completed` or `bounded`, `summary.within_scope_passed` is `true`, and `summary.definition_version` matches the contract's `definition_version`. Validate records against `output_schema` and compare `summary.records` with the array. HTTP 200 alone does not establish a usable result. A saved sample on the catalog page shows what records look like; it does not predict your result.

## Next

- [Published catalog APIs](/docs/catalog-apis/): find an API and read its contract.
- [Run a source](/docs/run-api/): the full response and run object.
- [Idempotency and recovery](/docs/idempotency/)