CleanedWeb
Markdown

JavaScript

Call your saved API from a Node.js server or worker using the built-in fetch. This guide makes one bounded request, saves its response, and checks the execution summary before accepting records.

On this page

Before you start#

Use Node.js 20 or later and a validated API. For a search API or another API with required inputs, add them to input as in the request supplied by your workspace. This guide needs no SDK; a typed client, cleanedweb-js, is in pre-release and not yet on npm.

Keep the API key on the server. A browser application should call your own backend; do not bundle the key into browser code or expose it through public environment variables.

Configure the request#

Load these variables through your server's secret and configuration management:

Variable Value
CLEANEDWEB_API_BASE https://api.cleanedweb.com, the API origin.
CLEANEDWEB_SOURCE_ID Your saved source identifier.
CLEANEDWEB_DEFINITION_VERSION The reviewed, lowercase 64-character definition version.
CLEANEDWEB_API_KEY A credential with access to this source and the run scope.
CLEANEDWEB_IDEMPOTENCY_KEY A saved identifier for this one intended execution.

Generate and persist the idempotency key with your request before sending. Keep the same key and body while investigating an uncertain outcome. Assign a new key for a new intentional collection. See idempotency and recovery.

The example sends this body with your saved version substituted for the placeholder:

JSON · example
{
  "expected_version": "YOUR_SAVED_DEFINITION_VERSION",
  "input": {},
  "max_items": 25
}

This stops after 25 records; it does not establish complete coverage. Your API's schema publishes the enrich_records default, and new APIs read detail pages by default; add "enrich_records": false to input to skip them, or "detail_limit": 0 on APIs created before enrich_records (see detail enrichment).

Run the example#

Download website-api.mjs, review it, and run it after setting your configuration. Use a fresh private directory or a unique output filename for each intended run so a later invocation cannot overwrite earlier evidence.

Shell
umask 077
node website-api.mjs > response.json

The script sends one POST /v1/sources/{source_id}/run with bearer authentication, the saved idempotency key, and a 120-second client timeout. If the API answers 202, the run is still going: the script prints its progress to stderr and reads GET /v1/runs/{run_id}/result, waiting Retry-After (or poll_after_seconds) between reads, until it returns the result or an error. It rejects redirects, never resends the POST, and does not retry failures. A timeout stops the client's wait; it does not establish that server execution stopped.

Check the outcome#

Exit code Meaning Next action
0 A completed or bounded response passed its scope and matched both returned definition versions. Validate record fields against your schema before storage.
2 The result needs review: partial, failed scope, missing summary, or a version mismatch. Preserve the result and investigate before downstream use.
1 Configuration, network, HTTP, or response error. Inspect the saved request and workspace before another execution.

Successful JSON responses are preserved in response.json, including their summary. HTTP failures save only the HTTP status; raw error bodies are excluded because they can echo credentials. Connection errors may leave an empty file, so never interpret an empty output file as an empty collection.

Keep delivery.runId and traceId when returned. Use the run reference to read a run's result again with an appropriately scoped key. A passing bounded result can still have catalog_complete: false.

Continue with storage and deduplication, scheduled collection, and the production checklist.

Updated Get help
Search across the documentation · Esc to close