Python
Make a bounded request with Python's standard library. The downloadable example preserves the response and checks its summary before your pipeline accepts the records.
On this page
Before you start#
Use Python 3.9 or later and a saved, validated API. If your API has required inputs, such as a search query, add them to input as in the generated request body.
No package installation is required. For an end-to-end guide that starts with a website URL, use the website to REST API tutorial.
Set connection details#
Load the following values from your server's configuration and secret storage:
| Variable | Value |
|---|---|
CLEANEDWEB_API_BASE |
https://api.cleanedweb.com, the API origin, without an extra path. |
CLEANEDWEB_SOURCE_ID |
The source identifier in your saved API. |
CLEANEDWEB_DEFINITION_VERSION |
The reviewed 64-character definition version. |
CLEANEDWEB_API_KEY |
A credential allowed to run this source. |
CLEANEDWEB_IDEMPOTENCY_KEY |
A persisted identifier for this intended execution. |
Store the key and exact request before sending. A new deliberate collection needs a new key; an uncertain existing request needs investigation using its original identity. Do not generate a replacement key automatically after a timeout. See idempotency and recovery.
Run one bounded collection#
Download website-api.py and inspect it before running. Use a fresh private directory or unique result filename for each run to avoid overwriting a previous result.
umask 077
python3 website-api.py > response.json
The script sends a single POST with bearer authentication and the saved idempotency key. It requests max_items: 25, refuses redirects, and uses a 120-second network timeout. If the API answers 202, the run is still going: the script prints its progress to stderr and reads GET /v1/runs/{run_id}/result, waiting Retry-After (or poll_after_seconds) between reads, until it returns the result or an error. It never resends the POST and does not retry failures. Your API's schema publishes the enrich_records default, and new APIs read detail pages by default; add "enrich_records": false to the input to skip them, or "detail_limit": 0 on APIs created before enrich_records.
A client timeout does not cancel or confirm server execution. Inspect the workspace and recover the run before deciding what to do next.
Review the saved response#
The script writes the response before checking whether it is usable. Preserve the file privately: records can contain data that should not appear in application logs or support messages.
| Exit code | Interpretation |
|---|---|
0 |
The run passed its selected scope, has a completed or bounded status, and matches the requested definition version. |
2 |
The response needs review before downstream use. |
1 |
Configuration, request, HTTP, or response processing failed. |
An HTTP error saves its status and selected, redacted diagnostic fields and headers. Non-JSON or oversized error bodies are marked rather than copied. A connection error may leave no response body. Always inspect the process exit status as well as the file.
A successful bounded run does not establish full-catalog coverage. Check summary.catalog_complete, collection limits, missing fields, and the types your application requires. Use delivery.runId and traceId, when returned, to connect the response to its workspace history.
Connect your pipeline#
Validate and map records against the saved output schema. Store the execution summary and definition identity alongside the records. Keep missing values distinct from empty strings, zero, or an inferred deletion.
Read pagination and output before raising limits, storage and deduplication before updating a dataset, and errors and retries before adding recovery logic.