# Move from ScrapingBee

Replace a ScrapingBee request that extracts records from a page with a CleanedWeb API that returns the same records on every run. Move one site at a time and keep ScrapingBee for the work it does well, like fetching any page through rotating proxies, taking screenshots and running browser steps.

For the product comparison and current prices, see [CleanedWeb vs. ScrapingBee](/compare/scrapingbee/).

## What moves and what stays

Move a job when your code calls ScrapingBee against the same site again and again, extracts fields with rules you maintain, and loops over pages. Keep ScrapingBee for pages you fetch once, pages you read yourself, and anything that needs a screenshot.

| You use ScrapingBee for | In CleanedWeb |
| --- | --- |
| `extract_rules` with CSS or XPath selectors on a listing or search page | One API for that collection. Its fields are reviewed once, at build time, and are the same on every run. |
| `ai_query` or `ai_extract_rules` on each page | A collection API. You choose the fields during the build instead of describing them on every call. |
| Your own loop over `?page=2`, `?page=3` and so on | A run that follows the site's pagination and de-duplicates records across pages. |
| Selectors that break when the site changes | CleanedWeb checks its APIs on a schedule and rebuilds broken catalog APIs itself. See [source changes](/docs/source-changes/). |
| Fetching arbitrary pages, screenshots, `js_scenario` browser steps | Keep ScrapingBee. |

## Concepts side by side

| ScrapingBee | CleanedWeb |
| --- | --- |
| `GET /api/v1/` with `url` on every request | A source with a reviewed definition version; your request sends only `input` |
| `extract_rules`, a JSON object of selectors you write | The API's published output schema, from its OpenAPI contract |
| Response body: the page, or the values your rules extracted | `records`: every matching record, de-duplicated across pages |
| One request per page | One run across all pages; `max_items` and `timeout_seconds` end it |
| `render_js`, `premium_proxy`, `stealth_proxy` | Not options. The API decides how to read the site, and the choice is part of the build. |
| Credits per request, multiplied by options | Units for the records returned; see [units and billing](/docs/billing/) |

## 1. Write down the job you run today

Pick one ScrapingBee job your code runs on a schedule. Save:

- the URL pattern, and the parts you change between calls (a search term, a city, a page number);
- the `extract_rules` or `ai_extract_rules` JSON, and the fields your code actually reads from the result;
- the loop that follows pagination, and the condition that stops it;
- one recent set of results, as the baseline you will compare against.

The query values you change become the API's inputs. The page number is not an input: the run follows pagination itself. The fields your code reads are the fields you keep in the build.

## 2. Build the API

Open the [API generator](/start/) with the first page of the job. Preview the records, select the collection, and keep the fields from step 1. Name them the way your code expects so the mapping later stays small. If the site is already in the [API catalog](/catalog/), you can use the published API and skip the build.

When the build is accepted, copy the source ID, API origin and definition version from its connection details. The [quickstart](/docs/quickstart/) shows where each one is.

## 3. Replace the call

A typical ScrapingBee job with extraction rules and a page loop:

```python
import os
from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key=os.environ["SCRAPINGBEE_API_KEY"])
rules = {
    "products": {
        "selector": ".product-card",
        "type": "list",
        "output": {
            "title": ".product-title",
            "price": ".price",
            "url": {"selector": "a", "output": "@href"},
        },
    }
}

items, page = [], 1
while True:
    response = client.html_api(
        f"https://shop.test/search?q=desk&page={page}",
        method="GET",
        params={"extract_rules": rules, "render_js": True},
    )
    response.raise_for_status()
    batch = response.json()["products"]
    if not batch:
        break
    items += batch
    page += 1
```

The same job in CleanedWeb sends only the input. The site, the fields and how to page through results are already part of the API:

```python
import os, requests

response = requests.post(
    f"{os.environ['CLEANEDWEB_API_BASE']}/v1/sources/{os.environ['CLEANEDWEB_SOURCE_ID']}/run",
    headers={
        "Authorization": f"Bearer {os.environ['CLEANEDWEB_API_KEY']}",
        "Idempotency-Key": saved_key,  # saved before sending; see below
    },
    json={
        "expected_version": os.environ["CLEANEDWEB_DEFINITION_VERSION"],
        "input": {"query": "desk"},
        "max_items": 500,
    },
    timeout=120,
)
response.raise_for_status()
body = response.json()
if response.status_code == 202:
    body = wait_for_result(body["run_id"])  # read GET /v1/runs/{run_id}/result
items = body["records"]
```

Things that work differently from ScrapingBee:

- **No selectors in your code.** The rules live in the API's definition, were reviewed when it was built, and are checked by CleanedWeb on a schedule. Your code reads `records`.
- **No page loop.** One run covers every page until the site runs out, `max_items` is reached, or `timeout_seconds` ends the run. `stop_reason` says which; the values are listed in [run an API](/docs/run-api/#the-run-object).
- **Idempotency key.** Generate one per intended run and store it before sending. If the connection drops, resend with the same key instead of starting a second run. See [idempotency](/docs/idempotency/).
- **Long runs answer `202`.** A run that takes longer than `wait_seconds` (90 seconds by default) keeps going on the server. Read `GET /v1/runs/{run_id}/result` after `Retry-After` seconds, or read records as they arrive from `/items`. Don't resend the POST to check progress. The [Python guide](/docs/python/) includes a complete script that does this.

## 4. Move the credit settings

With ScrapingBee you tune each request: `render_js` costs 5 credits by default, `premium_proxy` costs 25 with JavaScript rendering on (10 without), `stealth_proxy` costs 75, and the AI parameters add 5. You pay per request, so a page loop multiplies those costs by the number of pages.

With CleanedWeb you don't set any of these. A run is charged for the records it returns, and `max_items` caps how many that is. Details are in [units and billing](/docs/billing/).

| ScrapingBee setting | In CleanedWeb |
| --- | --- |
| `render_js`, `wait`, `wait_for` | Handled by the API's definition |
| `premium_proxy`, `stealth_proxy`, `country_code` | Not set per request |
| `ai_extract_rules` | Replaced by the fields you reviewed at build time |
| Your own retry and page loop | One run, with an idempotency key for recovery |

## 5. Compare, then switch

Run both against the same scope and match records by URL or another stable identifier. Check the fields your code reads, including formats such as prices, dates and currencies, and check that the record count matches the pages your loop reached. When the records match what you need, switch that part of your code to the new API and remove the ScrapingBee call for that site.

Keep your ScrapingBee key for pages you still fetch yourself. Then do the next site. The [migration guide](/docs/migration-guide/) has a fuller checklist for comparing output and rolling out, and the [production checklist](/docs/production-checklist/) covers credentials, monitoring and recovery.
