CleanedWeb
Markdown

Move from ScrapingBee

Replace a ScrapingBee request that extracts records from a page with a CleanedWeb API that returns the same records on every run. Move one site at a time and keep ScrapingBee for the work it does well, like fetching any page through rotating proxies, taking screenshots and running browser steps.

On this page

For the product comparison and current prices, see CleanedWeb vs. ScrapingBee.

What moves and what stays#

Move a job when your code calls ScrapingBee against the same site again and again, extracts fields with rules you maintain, and loops over pages. Keep ScrapingBee for pages you fetch once, pages you read yourself, and anything that needs a screenshot.

You use ScrapingBee for In CleanedWeb
extract_rules with CSS or XPath selectors on a listing or search page One API for that collection. Its fields are reviewed once, at build time, and are the same on every run.
ai_query or ai_extract_rules on each page A collection API. You choose the fields during the build instead of describing them on every call.
Your own loop over ?page=2, ?page=3 and so on A run that follows the site's pagination and de-duplicates records across pages.
Selectors that break when the site changes CleanedWeb checks its APIs on a schedule and rebuilds broken catalog APIs itself. See source changes.
Fetching arbitrary pages, screenshots, js_scenario browser steps Keep ScrapingBee.

Concepts side by side#

ScrapingBee CleanedWeb
GET /api/v1/ with url on every request A source with a reviewed definition version; your request sends only input
extract_rules, a JSON object of selectors you write The API's published output schema, from its OpenAPI contract
Response body: the page, or the values your rules extracted records: every matching record, de-duplicated across pages
One request per page One run across all pages; max_items and timeout_seconds end it
render_js, premium_proxy, stealth_proxy Not options. The API decides how to read the site, and the choice is part of the build.
Credits per request, multiplied by options Units for the records returned; see units and billing

1. Write down the job you run today#

Pick one ScrapingBee job your code runs on a schedule. Save:

  • the URL pattern, and the parts you change between calls (a search term, a city, a page number);
  • the extract_rules or ai_extract_rules JSON, and the fields your code actually reads from the result;
  • the loop that follows pagination, and the condition that stops it;
  • one recent set of results, as the baseline you will compare against.

The query values you change become the API's inputs. The page number is not an input: the run follows pagination itself. The fields your code reads are the fields you keep in the build.

2. Build the API#

Open the API generator with the first page of the job. Preview the records, select the collection, and keep the fields from step 1. Name them the way your code expects so the mapping later stays small. If the site is already in the API catalog, you can use the published API and skip the build.

When the build is accepted, copy the source ID, API origin and definition version from its connection details. The quickstart shows where each one is.

3. Replace the call#

A typical ScrapingBee job with extraction rules and a page loop:

Python
import os
from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key=os.environ["SCRAPINGBEE_API_KEY"])
rules = {
    "products": {
        "selector": ".product-card",
        "type": "list",
        "output": {
            "title": ".product-title",
            "price": ".price",
            "url": {"selector": "a", "output": "@href"},
        },
    }
}

items, page = [], 1
while True:
    response = client.html_api(
        f"https://shop.test/search?q=desk&page={page}",
        method="GET",
        params={"extract_rules": rules, "render_js": True},
    )
    response.raise_for_status()
    batch = response.json()["products"]
    if not batch:
        break
    items += batch
    page += 1

The same job in CleanedWeb sends only the input. The site, the fields and how to page through results are already part of the API:

Python
import os, requests

response = requests.post(
    f"{os.environ['CLEANEDWEB_API_BASE']}/v1/sources/{os.environ['CLEANEDWEB_SOURCE_ID']}/run",
    headers={
        "Authorization": f"Bearer {os.environ['CLEANEDWEB_API_KEY']}",
        "Idempotency-Key": saved_key,  # saved before sending; see below
    },
    json={
        "expected_version": os.environ["CLEANEDWEB_DEFINITION_VERSION"],
        "input": {"query": "desk"},
        "max_items": 500,
    },
    timeout=120,
)
response.raise_for_status()
body = response.json()
if response.status_code == 202:
    body = wait_for_result(body["run_id"])  # read GET /v1/runs/{run_id}/result
items = body["records"]

Things that work differently from ScrapingBee:

  • No selectors in your code. The rules live in the API's definition, were reviewed when it was built, and are checked by CleanedWeb on a schedule. Your code reads records.
  • No page loop. One run covers every page until the site runs out, max_items is reached, or timeout_seconds ends the run. stop_reason says which; the values are listed in run an API.
  • Idempotency key. Generate one per intended run and store it before sending. If the connection drops, resend with the same key instead of starting a second run. See idempotency.
  • Long runs answer 202. A run that takes longer than wait_seconds (90 seconds by default) keeps going on the server. Read GET /v1/runs/{run_id}/result after Retry-After seconds, or read records as they arrive from /items. Don't resend the POST to check progress. The Python guide includes a complete script that does this.

4. Move the credit settings#

With ScrapingBee you tune each request: render_js costs 5 credits by default, premium_proxy costs 25 with JavaScript rendering on (10 without), stealth_proxy costs 75, and the AI parameters add 5. You pay per request, so a page loop multiplies those costs by the number of pages.

With CleanedWeb you don't set any of these. A run is charged for the records it returns, and max_items caps how many that is. Details are in units and billing.

ScrapingBee setting In CleanedWeb
render_js, wait, wait_for Handled by the API's definition
premium_proxy, stealth_proxy, country_code Not set per request
ai_extract_rules Replaced by the fields you reviewed at build time
Your own retry and page loop One run, with an idempotency key for recovery

5. Compare, then switch#

Run both against the same scope and match records by URL or another stable identifier. Check the fields your code reads, including formats such as prices, dates and currencies, and check that the record count matches the pages your loop reached. When the records match what you need, switch that part of your code to the new API and remove the ScrapingBee call for that site.

Keep your ScrapingBee key for pages you still fetch yourself. Then do the next site. The migration guide has a fuller checklist for comparing output and rolling out, and the production checklist covers credentials, monitoring and recovery.

Updated Get help
Search across the documentation · Esc to close