CleanedWeb
Markdown

Move from Parse.bot

Replace a Parse.bot endpoint call with a CleanedWeb API that returns the records of the same site. Move one API at a time. Both tools start from a website and give you a typed API, so most of your code keeps its shape; the larger change is how you pay.

On this page

For the product comparison and current prices, see CleanedWeb vs. Parse.bot.

What moves and what stays#

Move an API when your code calls it on a schedule and reads a list of records from the result. Keep Parse.bot for endpoints that do something other than return records, like a booking or another action, and for any API you already use as a tool in an agent.

You use Parse.bot for In CleanedWeb
A generated GET endpoint that returns a list A collection API. One run returns every matching record, de-duplicated across pages.
An endpoint with parameters such as a page number or a search term input, defined by the API's published contract. Paging is part of the API.
An API from the Parse.bot marketplace An API from the catalog, or one you build for the site.
A POST endpoint that performs an action Keep Parse.bot.
Health checks and repair when the site changes Included. CleanedWeb checks its APIs on a schedule; a broken catalog API is rebuilt and swapped in once it qualifies. See source changes.

Concepts side by side#

Parse.bot CleanedWeb
An API, identified by scraper_id A source, identified by source_id
An endpoint, identified by endpoint_name The source's collection; a catalog API has endpoints with an endpoint_id
POST /dispatch with a url and a task The API generator with a URL, then a reviewed preview of the records
A revision of an API A definition version. Pin it with expected_version.
The endpoint's input_params input. Unknown fields are rejected.
The endpoint's return_schema The API's published output schema
X-API-Key header Authorization: Bearer with a key that has the run scope
Credits per call; the cost varies by API Units for the records returned; see units and billing

That last row is the main difference. Parse.bot charges credits for each call, and what a call costs varies by API; each response reports your remaining credits in an X-Credits-Remaining header. With CleanedWeb you pay for the records a run returns, and one run can return many. Compare the same request on both before you switch, and look at the total, not the per-call price.

1. Write down the endpoint you call today#

Pick one Parse.bot endpoint your code calls. Save:

  • the endpoint_name and the parameters you send, and which of them change between calls;
  • the fields your code reads from the response;
  • one recent response, as the baseline you will compare against.

The parameters you change become the API's inputs. The fields your code reads are the fields you keep in the build.

2. Find or build the API#

Search the API catalog for the site first. A published API can be run straight away; see run a catalog API. A catalog API runs its published version, and you cannot choose another.

If the site isn't there, open the API generator with the page your Parse.bot API reads. Preview the records, select the collection, and keep the fields from step 1. Name them the way your code expects. Building is included in CleanedWeb. When the build is accepted, copy the source ID, API origin and definition version from its connection details. The quickstart shows where each one is.

3. Replace the call#

A Parse.bot endpoint call is a request to the API's execution URL with your key in X-API-Key. POST endpoints take their parameters as a JSON body and GET endpoints take them as query parameters. The response is the endpoint's own JSON, with no wrapper:

Python
import os, requests

response = requests.post(
    f"https://api.parse.bot/scraper/{os.environ['PARSE_SCRAPER_ID']}/search_products",
    headers={"X-API-Key": os.environ["PARSE_API_KEY"]},
    json={"query": "desk", "page": 1},
    timeout=60,
)
response.raise_for_status()
products = response.json()  # shaped by the endpoint's return_schema

The search_products name and its parameters are placeholders. Yours come from your API's spec. The same job in CleanedWeb sends only the input. The site, the fields and how to page through results are already part of the API:

Python
import os, requests

response = requests.post(
    f"{os.environ['CLEANEDWEB_API_BASE']}/v1/sources/{os.environ['CLEANEDWEB_SOURCE_ID']}/run",
    headers={
        "Authorization": f"Bearer {os.environ['CLEANEDWEB_API_KEY']}",
        "Idempotency-Key": saved_key,  # your own saved key, stored before sending; see below
    },
    json={
        "expected_version": os.environ["CLEANEDWEB_DEFINITION_VERSION"],
        "input": {"query": "desk"},
        "max_items": 100,
    },
    timeout=120,
)
response.raise_for_status()
body = response.json()
if response.status_code == 202:
    body = wait_for_result(body["run_id"])  # your helper: poll GET /v1/runs/{run_id}/result
products = body["records"]

Things that work differently from Parse.bot:

  • No page loop. If your code passes a page number to Parse.bot and loops, remove the loop. A CleanedWeb run follows the site's pagination until the site runs out, max_items is reached, or timeout_seconds ends the run. Remove your page counter.
  • Idempotency key. Generate one per intended run and store it before sending. If the connection drops, resend with the same key instead of starting a second run. See idempotency.
  • Long runs answer 202. A run that takes longer than wait_seconds (90 seconds by default) keeps going on the server. Read GET /v1/runs/{run_id}/result after Retry-After seconds, or read records as they arrive from /items. Don't resend the POST to check progress. The Python guide includes a complete script that does this.
  • Records are in records. The response also has a summary and a run object. stop_reason tells you why the run ended, such as source_exhausted, max_items or timeout; the full list is in run an API.

4. Map errors and versions#

Parse.bot answers a blocked site, a failed site or invalid input with an error status such as 503, 502 or 422. A CleanedWeb run reports its outcome in the run: check run.stop_reason and summary.status before you use the records, not only the HTTP status. A request with an input the contract doesn't allow is rejected with 422, as with stale_input on Parse.bot.

A Parse.bot revision changes what an API does after you or its agent revise it. In CleanedWeb, a changed definition is a new version. Your call names the version it expects in expected_version, so it runs the rules you reviewed and not a newer set by surprise. When a new version arrives, read the new version from the connection details, compare a run, and update the saved value.

If you use a Parse.bot API as an MCP tool in an agent, leave that connection in place while you move the scheduled calls.

5. Compare, then switch#

Run both tools against the same scope and match records by URL or another stable identifier. Check the fields your code reads, including formats such as prices, dates and currencies. When the records match what you need, switch that call and remove the Parse.bot one for that site. Keep the old endpoint available until a few scheduled runs have matched.

Then do the next endpoint. The migration guide has a fuller checklist for comparing output and rolling out, and the production checklist covers credentials, monitoring and recovery.

Updated Get help
Search across the documentation · Esc to close