CleanedWeb
Markdown

Python

Make a bounded request with Python's standard library. The downloadable example preserves the response and checks its summary before your pipeline accepts the records.

On this page

Before you start#

Use Python 3.9 or later and a saved, validated API. If your API has required inputs, such as a search query, add them to input as in the generated request body.

No package installation is required. For an end-to-end guide that starts with a website URL, use the website to REST API tutorial.

Set connection details#

Load the following values from your server's configuration and secret storage:

Variable Value
CLEANEDWEB_API_BASE https://api.cleanedweb.com, the API origin, without an extra path.
CLEANEDWEB_SOURCE_ID The source identifier in your saved API.
CLEANEDWEB_DEFINITION_VERSION The reviewed 64-character definition version.
CLEANEDWEB_API_KEY A credential allowed to run this source.
CLEANEDWEB_IDEMPOTENCY_KEY A persisted identifier for this intended execution.

Store the key and exact request before sending. A new deliberate collection needs a new key; an uncertain existing request needs investigation using its original identity. Do not generate a replacement key automatically after a timeout. See idempotency and recovery.

Run one bounded collection#

Download website-api.py and inspect it before running. Use a fresh private directory or unique result filename for each run to avoid overwriting a previous result.

Shell
umask 077
python3 website-api.py > response.json

The script sends a single POST with bearer authentication and the saved idempotency key. It requests max_items: 25, refuses redirects, and uses a 120-second network timeout. If the API answers 202, the run is still going: the script prints its progress to stderr and reads GET /v1/runs/{run_id}/result, waiting Retry-After (or poll_after_seconds) between reads, until it returns the result or an error. It never resends the POST and does not retry failures. Your API's schema publishes the enrich_records default, and new APIs read detail pages by default; add "enrich_records": false to the input to skip them, or "detail_limit": 0 on APIs created before enrich_records.

A client timeout does not cancel or confirm server execution. Inspect the workspace and recover the run before deciding what to do next.

Review the saved response#

The script writes the response before checking whether it is usable. Preserve the file privately: records can contain data that should not appear in application logs or support messages.

Exit code Interpretation
0 The run passed its selected scope, has a completed or bounded status, and matches the requested definition version.
2 The response needs review before downstream use.
1 Configuration, request, HTTP, or response processing failed.

An HTTP error saves its status and selected, redacted diagnostic fields and headers. Non-JSON or oversized error bodies are marked rather than copied. A connection error may leave no response body. Always inspect the process exit status as well as the file.

A successful bounded run does not establish full-catalog coverage. Check summary.catalog_complete, collection limits, missing fields, and the types your application requires. Use delivery.runId and traceId, when returned, to connect the response to its workspace history.

Connect your pipeline#

Validate and map records against the saved output schema. Store the execution summary and definition identity alongside the records. Keep missing values distinct from empty strings, zero, or an inferred deletion.

Read pagination and output before raising limits, storage and deduplication before updating a dataset, and errors and retries before adding recovery logic.

Updated Get help
Search across the documentation · Esc to close