Python
Make a bounded request with Python's standard library. The downloadable example preserves the response and checks its summary before your pipeline accepts the records.
On this page
Before you start#
Use Python 3.9 or later and a saved, validated API. This example is for a collection contract that accepts max_pages and detail_limit; use the generated request body if your source has different inputs.
No package installation is required. For an end-to-end guide that starts with a website URL, use the website to REST API tutorial.
Set connection details#
Load the following values from your server's configuration and secret storage:
| Variable | Value |
|---|---|
CLEANEDWEB_API_BASE |
The supplied HTTPS API origin, without an extra path. |
CLEANEDWEB_SOURCE_ID |
The source identifier in your saved API. |
CLEANEDWEB_DEFINITION_VERSION |
The reviewed 64-character definition version. |
CLEANEDWEB_API_KEY |
A credential allowed to run this source. |
CLEANEDWEB_IDEMPOTENCY_KEY |
A persisted identifier for this intended execution. |
Store the key and exact request before sending. A new deliberate collection needs a new key; an uncertain existing request needs investigation using its original identity. Do not generate a replacement key automatically after a timeout. See idempotency and recovery.
Run one bounded collection#
Download website-api.py and inspect it before running. Use a fresh private directory or unique result filename for each run to avoid overwriting a previous result.
umask 077
python3 website-api.py > response.json
The script sends a single POST with bearer authentication and the saved idempotency key. It requests max_pages: 1 and detail_limit: 0, refuses redirects, and uses a 60-second network timeout. It does not retry automatically.
A client timeout does not cancel or confirm server execution. Inspect the workspace and retained result before deciding what to do next.
Review the saved response#
The script writes the response before checking whether it is usable. Preserve the file privately: records can contain data that should not appear in application logs or support messages.
| Exit code | Interpretation |
|---|---|
0 |
The run passed its selected scope, has a completed or bounded status, and matches the requested definition version. |
2 |
The response needs review before downstream use. |
1 |
Configuration, request, HTTP, or response processing failed. |
An HTTP error saves its status and selected, redacted diagnostic fields and headers. Non-JSON or oversized error bodies are marked rather than copied. A connection error may leave no response body. Always inspect the process exit status as well as the file.
A successful bounded run does not establish full-catalog coverage. Check summary.catalog_complete, collection limits, missing fields, and the types your application requires. Use delivery.runId and traceId, when returned, to connect the response to its workspace history.
Connect your pipeline#
Validate and map records against the saved output schema. Store the execution summary and definition identity alongside the records. Keep missing values distinct from empty strings, zero, or an inferred deletion.
Read pagination and output before raising limits, storage and deduplication before updating a dataset, and errors and retries before adding recovery logic.