CleanedWeb
Markdown

Turn a website into a REST API

This walkthrough takes you from a public collection URL to a bounded API request in cURL or Python. You will review the source, validate fields, and check whether returned records satisfy the scope you selected.

On this page

The example builds an API from GitHub's topic page for web scraping, the same page the catalog's GitHub Topics Repositories API reads. Your own build has its own identifiers and fields; use the connection details from its saved API page.

1. Choose one collection#

Open the API generator with the page that lists your records, not the site's homepage. https://github.com/topics/web-scraping lists only the repositories tagged with that topic; https://github.com holds everything else too.

State the records you need: “Name, description, topics, stars and link for each repository in this topic.” Check that inspection finds one row per repository and the fields you asked for. Navigation links, promoted boxes, login pages and challenge pages are not records.

See choosing a source if inspection finds a different collection or cannot access the page.

2. Review fields and validate the API#

Select the collection and review its observed fields, types, and missing values. On GitHub, owner, stars and language are read from each repository's own page, so a record whose page was not read lacks them.

Build and validate the selected API, then review its saved schema and sample records. A preview is not a working endpoint. Use fields and schemas to decide whether the output fits your application.

3. Copy the connection details#

Use the source identifier, definition version, credential, and request example from the saved API detail page. The API origin is https://api.cleanedweb.com. If machine access is unavailable, complete validation in the workspace before attempting these requests.

Set these variables in your server environment or secret manager; keep the credential out of browser code, source control, and shared logs:

Environment variable Value from your workspace
CLEANEDWEB_API_BASE https://api.cleanedweb.com, the API origin, without a path
CLEANEDWEB_SOURCE_ID Saved source identifier
CLEANEDWEB_DEFINITION_VERSION Saved 64-character lowercase definition version
CLEANEDWEB_API_KEY Scoped API credential
CLEANEDWEB_IDEMPOTENCY_KEY Unique key saved with this intended request

Save the request key alongside the source, version, endpoint, and inputs before sending. Use a UUID or 1–200 letters, numbers, ., _, :, or -. Keep that value across a retry of the same request. A new intentional execution uses a new key. Read idempotency and recovery before automating requests.

The example contract below is the same one used in the API quickstart. Follow your generated request example if its route or inputs differ.

4. Make one bounded request with cURL#

This POST executes the source and may incur usage. It stops after 25 records (max_items). Your API's schema publishes the enrich_records default, and new APIs read detail pages by default; send "enrich_records":false to skip them, or "detail_limit":0 on APIs created before enrich_records (see detail enrichment). Search APIs can require a constrained query instead; follow their generated request. Do not automatically repeat a failed or timed-out request with a new key.

Shell
umask 077
: "${CLEANEDWEB_IDEMPOTENCY_KEY:?Set the saved key for this request}"

curl --fail-with-body --max-time 120 --write-out '%{http_code}\n' \
  "${CLEANEDWEB_API_BASE}/v1/sources/${CLEANEDWEB_SOURCE_ID}/run" \
  -H "Authorization: Bearer ${CLEANEDWEB_API_KEY}" \
  -H "Idempotency-Key: ${CLEANEDWEB_IDEMPOTENCY_KEY}" \
  -H 'Content-Type: application/json' \
  --data "{\"expected_version\":\"${CLEANEDWEB_DEFINITION_VERSION}\",\"input\":{},\"max_items\":25}" \
  --output curl-response.json

Keep curl-response.json private if the source contains sensitive data. Check the HTTP outcome and the response summary before using the records. A printed 202 means the run is still going: the file holds its run object, and GET /v1/runs/{run_id}/result returns the records once it ends (see responses). A timeout leaves the execution state uncertain; it does not establish that nothing ran.

5. Use website data in Python#

Download the Python client example. It uses Python 3.9 or later with no third-party packages. It sends the same body as the cURL example; add the inputs your API publishes if you need them. Set the same five environment variables, including the saved request key, then run:

Shell
umask 077
python3 website-api.py > python-response.json

Use a separate private directory for each intended run. If you already ran the cURL example with this key and the same body, the Python request returns that same run rather than collecting again.

The client sends one request with the saved idempotency key, limits, version check, and a 120-second timeout. If the API answers 202, it prints the run's progress to stderr and reads GET /v1/runs/{run_id}/result, waiting the returned Retry-After between reads, until the run ends; it never resends the POST. It refuses redirects, preserves the entire JSON response, and exits with code 2 when the summary needs review or either returned definition version differs from the requested version. Exit code 1 indicates a request, configuration, or response error. On HTTP errors (including 401, 429, and 504), python-response.json contains the HTTP status, allowlisted diagnostic body fields, and available content type, retry timing, and request/trace headers. Credentials are redacted; unrelated fields and headers are omitted. Non-JSON or oversized error bodies are marked without copying their contents. Keep this file private and share only the needed diagnostics with support. Inspect the saved result and workspace before deciding what to do next.

6. Read records together with the summary#

A bounded result, shortened, looks like this:

JSON · example
{
  "records": [{ "title": "creeper", "stargazer_count": "12", "url": "https://github.com/z7r1k3/creeper" }],
  "summary": {
    "status": "bounded",
    "within_scope_passed": true,
    "catalog_complete": false
  }
}

The request passed within its selected scope. It did not collect the entire catalog. This shortened example omits version and delivery fields. In the full response, require summary.definition_version and metadata.definition_version to equal the version you requested. Check the schema, limits, missing fields, and summary before downstream use. A nonempty array or HTTP 200 is insufficient evidence of completeness.

If the summary reports partial, fails its scope check, or is absent, preserve the output for review. Do not silently replace that outcome with an empty dataset. See errors and retries.

7. Expand only after checking the first result#

Raise max_items and timeout_seconds only when you understand the intended usage. Repeating a request does not mean “fetch the next page.” Read pagination and output before expanding.

Keep credentials scoped, preserve missing values, and retain delivery.runId (delivery.chargeReceiptId is the same identifier) and traceId when returned. If the saved definition changes, review the new schema instead of removing expected_version. The source-change guide explains how to handle changed fields and partial runs. Use the Run API reference when building your integration.

Updated Get help
Search across the documentation · Esc to close