# Move from Bright Data

Replace a Bright Data Web Scraper API collection with a CleanedWeb API that returns records from the same site. Move one scraper at a time. Keep Bright Data for the work it does well: proxy networks, Web Unlocker and Scraping Browser for your own crawlers, and bulk datasets you buy ready-made.

For the product comparison and current prices, see [CleanedWeb vs. Bright Data](/compare/bright-data/).

## What moves and what stays

Move a job when your code triggers the same scraper again and again and reads the fields out of each snapshot. Keep Bright Data where you need its network or its volume.

| You use Bright Data for | In CleanedWeb |
| --- | --- |
| A Web Scraper API scraper (`dataset_id`) for one site, triggered on a schedule | A published API from the [catalog](/catalog/), or an API built for your site in the [API generator](/start/). |
| `limit_per_input` on a discovery collection | `max_items` and `timeout_seconds` on the run. |
| Downloading a snapshot | Reading `/items` or `/result` of the run. |
| Web Unlocker or Scraping Browser inside your own crawler | Keep Bright Data. CleanedWeb is not a proxy product. |
| Datasets marketplace, a large dataset bought in one piece | Keep Bright Data. |

## Concepts side by side

| Bright Data | CleanedWeb |
| --- | --- |
| A scraper from the library, chosen by `dataset_id` | A published API from the catalog, or a source you built |
| `input`: an array of objects, with fields set by the scraper | `input`: one object, defined by the API's published contract |
| `POST /datasets/v3/trigger` returns a `snapshot_id` | `POST /v1/sources/{source_id}/runs` returns a run object with `run_id` |
| `POST /datasets/v3/scrape`, which answers `202` with a `snapshot_id` after about a minute | `POST /v1/sources/{source_id}/run`, which waits up to `wait_seconds` and then answers `202` |
| `GET /datasets/v3/progress/{snapshot_id}` with `status` | `GET /v1/runs/{run_id}` with `status` |
| `GET /datasets/v3/snapshot/{snapshot_id}` with `format` | `GET /v1/runs/{run_id}/items`, paged by `next_cursor`, or `/result` |
| `limit_per_input`, `limit_multiple_results` | `max_items`, `timeout_seconds` |
| `endpoint` (a webhook URL) and `notify` for completion alerts | Your own scheduler; see [schedule collection](/docs/scheduled-runs/) |
| Per record delivered | Units for the records returned; see [units and billing](/docs/billing/) |

## 1. Write down the collection you run today

Pick one scraper your code triggers. Save:

- the `dataset_id`, and the `input` objects you send, noting which values change between runs (a URL, a keyword, a category);
- any `limit_per_input` or `limit_multiple_results` you set;
- the snapshot fields your code reads, and the format you download (`json`, `ndjson` or `csv`);
- one recent snapshot, as the baseline you will compare against.

The values that change between runs become the API's inputs. The fields your code reads are the fields you keep in the build.

## 2. Find or build the API

Search the [API catalog](/catalog/) for the site first. If it is there, read its contract and skip the build; see [run a catalog API](/docs/run-catalog-api/).

If not, open the [API generator](/start/) with the page your scraper reads. Preview the records, select the collection, and keep the fields from step 1. Name them the way your code expects. When the build is accepted, copy the source ID, API origin and definition version from its connection details. The [quickstart](/docs/quickstart/) shows where each one is.

## 3. Replace the call

A typical Bright Data collection: trigger it, wait until it is ready, download the snapshot.

```python
import os, time, requests

HEADERS = {"Authorization": f"Bearer {os.environ['BRIGHTDATA_API_KEY']}"}
BASE = "https://api.brightdata.com/datasets/v3"

trigger = requests.post(
    f"{BASE}/trigger",
    headers=HEADERS,
    params={"dataset_id": os.environ["BRIGHTDATA_DATASET_ID"]},
    json={"input": [{"url": "https://shop.test/category/desks"}]},
    timeout=60,
).json()
snapshot_id = trigger["snapshot_id"]

while requests.get(
    f"{BASE}/progress/{snapshot_id}", headers=HEADERS, timeout=60
).json()["status"] in ("starting", "running"):
    time.sleep(10)

items = requests.get(
    f"{BASE}/snapshot/{snapshot_id}",
    headers=HEADERS, params={"format": "json"}, timeout=300,
).json()
```

The same job in CleanedWeb, started in the background and read as it runs:

```python
import os, time, uuid, requests

BASE = os.environ["CLEANEDWEB_API_BASE"]
HEADERS = {"Authorization": f"Bearer {os.environ['CLEANEDWEB_API_KEY']}"}
saved_key = str(uuid.uuid4())  # store this in your job record before sending

response = requests.post(
    f"{BASE}/v1/sources/{os.environ['CLEANEDWEB_SOURCE_ID']}/runs",
    headers={**HEADERS, "Idempotency-Key": saved_key},
    json={
        "expected_version": os.environ["CLEANEDWEB_DEFINITION_VERSION"],
        "input": {"category": "desks"},
        "max_items": 500,
    },
    timeout=60,
)
response.raise_for_status()  # 202 means the run started; 200 means it already ended
run = response.json()

items, cursor = [], 0
while cursor is not None:
    page = requests.get(
        f"{BASE}/v1/runs/{run['run_id']}/items",
        headers=HEADERS, params={"cursor": cursor}, timeout=60,
    ).json()
    items += page["items"]
    cursor = page["next_cursor"]
    if page["run_status"] in ("queued", "running") and not page["items"]:
        time.sleep(30)
```

Things that work differently from Bright Data:

- **No separate progress call.** `/items` returns records as they are committed and carries `run_status`. `next_cursor` becomes null once the run has ended and every record was read. For a short run, call `/run` instead of `/runs`: it waits up to `wait_seconds` (90 by default) and answers with the records. The [Python guide](/docs/python/) has a complete script.
- **Idempotency key.** Generate one per intended run and store it before sending. If the connection drops, resend with the same key instead of starting a second run. See [idempotency](/docs/idempotency/).
- **Inputs are checked.** Unknown input fields and values outside the contract are rejected with an error.
- **Why a run ended** is in `stop_reason`: `source_exhausted`, `max_items`, `timeout`, `budget_exhausted` and others in [run an API](/docs/run-api/#the-run-object).
- **Detail pages.** Where the API has them (newer APIs publish `enrich_records`; older ones use `detail_limit`), `enrich_records: true` merges each record's detail page into the record.

## 4. Map limits, snapshots and webhooks

| Bright Data | CleanedWeb |
| --- | --- |
| `limit_per_input` caps records for each input object | `max_items` caps the records of the run. Send one run per input if you need a cap per input. |
| One trigger with many inputs gives one snapshot | One run per `input` object. Start several runs, each with its own idempotency key. |
| `batch_size` and `part` to download a large snapshot in pieces | Page `/items` with `cursor` and `limit`, or read it as `jsonl` with `format=jsonl`. |
| Snapshots are kept for 30 days after the job completes | Read a run with `GET /v1/runs/{run_id}`; keep records you need in your own storage. See [storing results](/docs/storing-results/). |
| Completion webhook (`endpoint`) | Start downstream work when the run's `/result` answers `200`. |

## 5. Compare, then switch

Run the Bright Data scraper and the new API on the same scope. Match records by URL or another stable identifier and compare the fields your code reads, including formats such as prices, dates and currencies. When they match, switch that call, stop the scraper's schedule, and move the next one.

If part of your work also uses Web Unlocker or Scraping Browser for pages you fetch yourself, leave that on Bright Data. You can move the collections one at a time and keep the rest.

The [migration guide](/docs/migration-guide/) has a fuller checklist for comparing output and rolling out, and the [production checklist](/docs/production-checklist/) covers credentials, monitoring and recovery.
