CleanedWeb
Markdown

Move from Bright Data

Replace a Bright Data Web Scraper API collection with a CleanedWeb API that returns records from the same site. Move one scraper at a time. Keep Bright Data for the work it does well: proxy networks, Web Unlocker and Scraping Browser for your own crawlers, and bulk datasets you buy ready-made.

On this page

For the product comparison and current prices, see CleanedWeb vs. Bright Data.

What moves and what stays#

Move a job when your code triggers the same scraper again and again and reads the fields out of each snapshot. Keep Bright Data where you need its network or its volume.

You use Bright Data for In CleanedWeb
A Web Scraper API scraper (dataset_id) for one site, triggered on a schedule A published API from the catalog, or an API built for your site in the API generator.
limit_per_input on a discovery collection max_items and timeout_seconds on the run.
Downloading a snapshot Reading /items or /result of the run.
Web Unlocker or Scraping Browser inside your own crawler Keep Bright Data. CleanedWeb is not a proxy product.
Datasets marketplace, a large dataset bought in one piece Keep Bright Data.

Concepts side by side#

Bright Data CleanedWeb
A scraper from the library, chosen by dataset_id A published API from the catalog, or a source you built
input: an array of objects, with fields set by the scraper input: one object, defined by the API's published contract
POST /datasets/v3/trigger returns a snapshot_id POST /v1/sources/{source_id}/runs returns a run object with run_id
POST /datasets/v3/scrape, which answers 202 with a snapshot_id after about a minute POST /v1/sources/{source_id}/run, which waits up to wait_seconds and then answers 202
GET /datasets/v3/progress/{snapshot_id} with status GET /v1/runs/{run_id} with status
GET /datasets/v3/snapshot/{snapshot_id} with format GET /v1/runs/{run_id}/items, paged by next_cursor, or /result
limit_per_input, limit_multiple_results max_items, timeout_seconds
endpoint (a webhook URL) and notify for completion alerts Your own scheduler; see schedule collection
Per record delivered Units for the records returned; see units and billing

1. Write down the collection you run today#

Pick one scraper your code triggers. Save:

  • the dataset_id, and the input objects you send, noting which values change between runs (a URL, a keyword, a category);
  • any limit_per_input or limit_multiple_results you set;
  • the snapshot fields your code reads, and the format you download (json, ndjson or csv);
  • one recent snapshot, as the baseline you will compare against.

The values that change between runs become the API's inputs. The fields your code reads are the fields you keep in the build.

2. Find or build the API#

Search the API catalog for the site first. If it is there, read its contract and skip the build; see run a catalog API.

If not, open the API generator with the page your scraper reads. Preview the records, select the collection, and keep the fields from step 1. Name them the way your code expects. When the build is accepted, copy the source ID, API origin and definition version from its connection details. The quickstart shows where each one is.

3. Replace the call#

A typical Bright Data collection: trigger it, wait until it is ready, download the snapshot.

Python
import os, time, requests

HEADERS = {"Authorization": f"Bearer {os.environ['BRIGHTDATA_API_KEY']}"}
BASE = "https://api.brightdata.com/datasets/v3"

trigger = requests.post(
    f"{BASE}/trigger",
    headers=HEADERS,
    params={"dataset_id": os.environ["BRIGHTDATA_DATASET_ID"]},
    json={"input": [{"url": "https://shop.test/category/desks"}]},
    timeout=60,
).json()
snapshot_id = trigger["snapshot_id"]

while requests.get(
    f"{BASE}/progress/{snapshot_id}", headers=HEADERS, timeout=60
).json()["status"] in ("starting", "running"):
    time.sleep(10)

items = requests.get(
    f"{BASE}/snapshot/{snapshot_id}",
    headers=HEADERS, params={"format": "json"}, timeout=300,
).json()

The same job in CleanedWeb, started in the background and read as it runs:

Python
import os, time, uuid, requests

BASE = os.environ["CLEANEDWEB_API_BASE"]
HEADERS = {"Authorization": f"Bearer {os.environ['CLEANEDWEB_API_KEY']}"}
saved_key = str(uuid.uuid4())  # store this in your job record before sending

response = requests.post(
    f"{BASE}/v1/sources/{os.environ['CLEANEDWEB_SOURCE_ID']}/runs",
    headers={**HEADERS, "Idempotency-Key": saved_key},
    json={
        "expected_version": os.environ["CLEANEDWEB_DEFINITION_VERSION"],
        "input": {"category": "desks"},
        "max_items": 500,
    },
    timeout=60,
)
response.raise_for_status()  # 202 means the run started; 200 means it already ended
run = response.json()

items, cursor = [], 0
while cursor is not None:
    page = requests.get(
        f"{BASE}/v1/runs/{run['run_id']}/items",
        headers=HEADERS, params={"cursor": cursor}, timeout=60,
    ).json()
    items += page["items"]
    cursor = page["next_cursor"]
    if page["run_status"] in ("queued", "running") and not page["items"]:
        time.sleep(30)

Things that work differently from Bright Data:

  • No separate progress call. /items returns records as they are committed and carries run_status. next_cursor becomes null once the run has ended and every record was read. For a short run, call /run instead of /runs: it waits up to wait_seconds (90 by default) and answers with the records. The Python guide has a complete script.
  • Idempotency key. Generate one per intended run and store it before sending. If the connection drops, resend with the same key instead of starting a second run. See idempotency.
  • Inputs are checked. Unknown input fields and values outside the contract are rejected with an error.
  • Why a run ended is in stop_reason: source_exhausted, max_items, timeout, budget_exhausted and others in run an API.
  • Detail pages. Where the API has them (newer APIs publish enrich_records; older ones use detail_limit), enrich_records: true merges each record's detail page into the record.

4. Map limits, snapshots and webhooks#

Bright Data CleanedWeb
limit_per_input caps records for each input object max_items caps the records of the run. Send one run per input if you need a cap per input.
One trigger with many inputs gives one snapshot One run per input object. Start several runs, each with its own idempotency key.
batch_size and part to download a large snapshot in pieces Page /items with cursor and limit, or read it as jsonl with format=jsonl.
Snapshots are kept for 30 days after the job completes Read a run with GET /v1/runs/{run_id}; keep records you need in your own storage. See storing results.
Completion webhook (endpoint) Start downstream work when the run's /result answers 200.

5. Compare, then switch#

Run the Bright Data scraper and the new API on the same scope. Match records by URL or another stable identifier and compare the fields your code reads, including formats such as prices, dates and currencies. When they match, switch that call, stop the scraper's schedule, and move the next one.

If part of your work also uses Web Unlocker or Scraping Browser for pages you fetch yourself, leave that on Bright Data. You can move the collections one at a time and keep the rest.

The migration guide has a fuller checklist for comparing output and rolling out, and the production checklist covers credentials, monitoring and recovery.

Updated Get help
Search across the documentation · Esc to close