Run a catalog API

Run a published catalog API from your server, read its records, and recover safely when a response is lost. A run uses the credits of your workspace.

POST /v1/catalog/public/apis/{api_id}/endpoints/{endpoint_id}/run

Use the url from the listing's contract.json on the catalog site, or the path from the live contract (send it to https://broker.cleanedweb.com); both already have your API and endpoint identifiers. The server runs the endpoint's published version. You cannot choose the version, the source, or the owner.

Before you send a request#

• Read the contract and decide your input. Send every input its input_schema.required lists; an empty list means {} is valid.
• Create an API key with the run scope. Keep it in your server's secret store. See authentication.
• Make sure the workspace has Units. A run is charged by committed records; see billing and failure behavior.
• Generate an idempotency key and save it before you send. Reuse it only to recover this same request.

Set these variables. The origin is the run origin, without a path. Runs and results are served by https://broker.cleanedweb.com, or by the origin your workspace supplies. The read routes under /api/catalog/ are on https://app.cleanedweb.com, which answers /v1/ with a redirect to the home page.

Headers#

Body#

A run ends when the source has no more records, or by max_items, timeout_seconds, or money (see stop reasons). There is no page or row cap. Where the contract lists max_pages or detail_limit in input_schema, they are optional input limits like any other input; omit them unless you need them. A run option outside its range returns 422 with run_option_invalid.

wait_seconds is not part of the request identity; changing it does not create a new run. Changing input, max_items or timeout_seconds does.

Send the request#

Start with a small max_items while you check the shape of the records. This request asks for 25 records from the Gymshark Products API's "All Socks" collection:

umask 077
: "${CLEANEDWEB_IDEMPOTENCY_KEY:?Set the saved key for this request}"

curl --fail-with-body --max-time 120 --write-out '%{http_code}\n' \
  "${CLEANEDWEB_RUN_BASE}/v1/catalog/public/apis/${CLEANEDWEB_CATALOG_API_ID}/endpoints/${CLEANEDWEB_CATALOG_ENDPOINT_ID}/run" \
  -H "Authorization: Bearer ${CLEANEDWEB_API_KEY}" \
  -H "Idempotency-Key: ${CLEANEDWEB_IDEMPOTENCY_KEY}" \
  -H 'Content-Type: application/json' \
  --data '{"input":{"collection":"All Socks"},"max_items":25,"timeout_seconds":3600,"wait_seconds":90}' \
  --output response.json

Use a fresh private directory or a unique output filename for each run. Keep response.json private. The 120-second client limit leaves room for the 90-second wait. A client timeout does not stop the run. A 3xx status means the request went to the wrong origin and nothing ran; curl --fail-with-body does not treat it as an error, so check the printed status.

A new run waits in the queue (status: queued) while the workspace already has a /run, /stream or MCP call in progress. It is not rejected. Background runs started with /v1/sources/{source_id}/runs queue separately.

Wait for a longer run#

On 202 the run continues on the server. Do not send the POST again. After Retry-After seconds, read the result with the same API key and the same Idempotency-Key:

GET /v1/catalog/public/apis/{api_id}/endpoints/{endpoint_id}/result

curl --fail-with-body --max-time 120 --write-out '%{http_code}\n' \
  "${CLEANEDWEB_RUN_BASE}/v1/catalog/public/apis/${CLEANEDWEB_CATALOG_API_ID}/endpoints/${CLEANEDWEB_CATALOG_ENDPOINT_ID}/result" \
  -H "Authorization: Bearer ${CLEANEDWEB_API_KEY}" \
  -H "Idempotency-Key: ${CLEANEDWEB_IDEMPOTENCY_KEY}" \
  --output result.json

It answers 202 with the run object while the run goes on, 200 with the final body when it ends, or the run's error. Reading a result never collects the website again. It works after the API is unpublished or its version changes, because the delivery keeps its original endpoint and version. Another workspace, a revoked key, or a different endpoint cannot read it; a key unknown for the endpoint returns 404, and a key reused for a different endpoint or version returns 409 delivery_identity_conflict.

With the run_id from the 202, you can also use the run routes:

Pass the returned next_cursor to /items to continue. It is null once the run has ended and every record was read. format=jsonl returns one record per line and the next cursor in the X-Next-Cursor header. Aborting needs the run scope. See run an API for the full run object.

JavaScript example#

This Node.js 20 script sends one run, waits with the same key, and refuses to accept an incomplete result. Keep the key on the server. It never resends the POST.

const base = process.env.CLEANEDWEB_RUN_BASE;          // https://broker.cleanedweb.com
const root = `${base}/v1/catalog/public/apis/${process.env.CLEANEDWEB_CATALOG_API_ID}` +
  `/endpoints/${process.env.CLEANEDWEB_CATALOG_ENDPOINT_ID}`;
const headers = {
  Authorization: `Bearer ${process.env.CLEANEDWEB_API_KEY}`,
  'Idempotency-Key': process.env.CLEANEDWEB_IDEMPOTENCY_KEY, // saved before sending
};
const sleep = (seconds) => new Promise((resolve) => setTimeout(resolve, seconds * 1000));

async function read(response) {
  const body = await response.json().catch(() => null);
  if (response.ok && !body) throw new Error(`HTTP ${response.status} without a JSON body`);
  if (response.status === 202) return { pending: true, body, response };
  if (response.ok) return { body };
  const detail = body?.detail;
  const marker = detail?.error_code;           // run_failed | failed_run_replayed
  if (marker) throw new Error(`run ${detail.run_id} failed (${marker}): ${response.status}`);
  const error = new Error(`HTTP ${response.status} ${detail?.code ?? ''}`.trim());
  error.unsure = response.status >= 500 || response.status === 408; // a gateway error or timeout proves nothing
  throw error;
}

async function run() {
  let outcome = await read(await fetch(`${root}/run`, {
    method: 'POST',
    headers: { ...headers, 'Content-Type': 'application/json' },
    body: JSON.stringify({
      input: { collection: 'All Socks' },
      max_items: 25,
      timeout_seconds: 3600,
      wait_seconds: 90,
    }),
    signal: AbortSignal.timeout(120_000),
    redirect: 'error',                         // a redirect means the wrong origin
  }));
  while (outcome.pending) {
    const wait = Number(outcome.response.headers.get('retry-after')) || outcome.body?.poll_after_seconds || 30;
    await sleep(wait);
    outcome = await read(await fetch(`${root}/result`, { headers, redirect: 'error', signal: AbortSignal.timeout(120_000) }));
  }
  return outcome.body;
}

const result = await run();
const { summary, run: info } = result;
if (!['completed', 'bounded'].includes(summary.status) || !summary.within_scope_passed) {
  throw new Error(`review needed: ${summary.status}, stop_reason ${info.stop_reason}`);
}
console.log(`${result.records.length} records, stop_reason ${info.stop_reason}, run ${result.delivery.runId}`);

If run() throws with unsure, or fetch itself throws (a timeout or lost connection), the run may have started. Keep the key and call /result with it before you decide; do not create a new key. If the error has error_code, the run failed and a new attempt needs a new key.

To read records while a run is going, call GET ${base}/v1/runs/${runId}/items?cursor=${cursor} with the run_id from the 202 body and follow next_cursor.

Stop reasons#

run.stop_reason says why a run ended:

Saved records stay readable and charged in every state. A run stopped by max_items, timeout, budget_exhausted or an abort reports summary.status as bounded when its scope passed, otherwise partial. bounded and partial mean the source can hold more records than you received. A completed run still does not assert full-site coverage: when the listing's coverage arrays are empty, coverage is unspecified, and summary.catalog_complete is false.

Failed runs#

A failed run is final for its idempotency key. The same key returns the same error for 7 days (replay) and the source is not started again. To run again, send a new key.

A failed-run error carries a marker in detail, plus a header:

An error without error_code, such as a timeout, a lost connection or a gateway error, does not show whether the run failed. Keep the key, read /result, and decide from that. Records saved before a failure stay readable at /v1/runs/{run_id}/items.

Errors and billing codes#

Billing is one Unit per saved record, charged as pages are committed. Missing or unreadable billing configuration fails closed: the request is refused, never run for free.

Treat an unknown code as a failure and keep the response. See errors and retries for the general rules and idempotency for recovery. A 503 is never an empty result set.

A budget_exhausted stop is not an error: the run returns what the credit paid for. To check a charge, compare run.units_charged, delivery.recordCount and the Billing record for the run_id.

Accept the result#

Before downstream processing, check summary.status is completed or bounded, summary.within_scope_passed is true, and summary.definition_version matches the contract's definition_version. Validate records against output_schema and compare summary.records with the array. HTTP 200 alone does not establish a usable result. A saved sample on the catalog page shows what records look like; it does not predict your result.

Next#

• Published catalog APIs: find an API and read its contract.
• Run a source: the full response and run object.
• Idempotency and recovery