Move from Scrapy or Playwright scrapers
Replace one of your own Scrapy spiders or Playwright scripts with a CleanedWeb API that returns records from the same site. Move one spider at a time, and keep the old one running until the new output matches. Keep your own code for sites that need a login, pages behind a form flow, or logic the API can't express.
On this page
For the product comparison and current prices, see CleanedWeb vs. Custom scrapers.
What moves and what stays#
Move a scraper when it reads the same public site on a schedule and your code uses the fields it returns. Keep it when you need full control over the browser or have to work behind a login; CleanedWeb reads public pages only.
| Your scraper has | In CleanedWeb |
|---|---|
| A spider or script per site, and an item class | A source, and the API's published output schema |
| CSS or XPath selectors, or Playwright locators | Fields you review once, at build time |
| Pagination: following "next", or clicking through | Pagination the API follows until the run ends: the site has no more records, max_items, timeout_seconds or your budget |
| A second request for each detail page | enrich_records: true on an API with a detail plan (detail_limit on older APIs) |
| Proxies, retries and blocked-request handling | You don't run or configure them; a run a site blocks ends with stop_reason: blocked |
| Cron, Airflow or a job queue | Keep it; it starts a run instead of a spider (scheduled runs) |
| An item pipeline that drops duplicates and writes to storage | Records are de-duplicated within a run; you still write to your storage (storing results) |
| Someone watching for broken selectors | CleanedWeb checks its APIs on a schedule and rebuilds broken catalog APIs; a contract change to your own API waits for your review (source changes) |
1. Write down what the scraper does#
For one spider or script, save:
- the start URLs, and any values that change between runs (a search term, a category, a city);
- the fields it returns, with the transformations it applies (price parsing, date formats, currency codes);
- one recent output file, as the baseline you will compare against.
The values that change become the API's inputs. The fields your code reads are the fields you keep in the build. Note any transformation too: keep it in a small mapping layer on your side, after the API call.
2. Build the API#
Open the API generator with the page your spider starts from. Preview the records, select the collection, and check that the reviewed fields cover the ones from step 1. Compare the saved schema's field names with what your storage expects and handle any difference in your mapping layer. If the site is already in the API catalog, you can use the published API and skip the build. See run a catalog API.
When the build is accepted, copy the source ID, API origin and definition version from its connection details. The quickstart shows where each one is.
3. Replace the spider#
A typical Scrapy spider: it reads each listing, follows the detail page, and follows the next link.
import scrapy
class Product(scrapy.Item):
sku = scrapy.Field()
name = scrapy.Field()
price = scrapy.Field()
description = scrapy.Field()
class ProductsSpider(scrapy.Spider):
name = "products"
start_urls = ["https://shop.test/search?q=desk"]
def parse(self, response):
for card in response.css("div.product"):
href = card.css("a::attr(href)").get()
partial = {
"sku": card.attrib.get("data-sku"),
"name": card.css("h2::text").get(),
"price": card.css("span.price::text").get(),
}
yield response.follow(href, self.parse_detail, cb_kwargs={"partial": partial})
next_page = response.css("a.next::attr(href)").get()
if next_page is not None:
yield response.follow(next_page, callback=self.parse)
def parse_detail(self, response, partial):
item = Product(**partial)
item["description"] = response.css("div.description::text").get()
yield item
The listing part of the same job as a Playwright script, with the proxy you maintain (it doesn't open detail pages):
from playwright.sync_api import sync_playwright
items = []
with sync_playwright() as p:
browser = p.chromium.launch(proxy={"server": PROXY_URL, "username": PROXY_USER, "password": PROXY_PASS})
page = browser.new_page()
page.goto("https://shop.test/search?q=desk")
while True:
for card in page.locator("div.product").all():
items.append({
"name": card.locator("h2").inner_text(),
"price": card.locator("span.price").inner_text(),
"url": card.locator("a").first.get_attribute("href"),
})
next_link = page.locator("a.next")
if next_link.count() == 0:
break
next_link.click()
page.wait_for_load_state()
browser.close()
The same job in CleanedWeb sends only the input. The site, the fields, the detail pages and the pagination are already part of the API:
import os, requests
response = requests.post(
f"{os.environ['CLEANEDWEB_API_BASE']}/v1/sources/{os.environ['CLEANEDWEB_SOURCE_ID']}/run",
headers={
"Authorization": f"Bearer {os.environ['CLEANEDWEB_API_KEY']}",
"Idempotency-Key": saved_key, # saved before sending; see below
},
json={
"expected_version": os.environ["CLEANEDWEB_DEFINITION_VERSION"],
"input": {"query": "desk", "enrich_records": True}, # input names come from your API's schema
"max_items": 500,
},
timeout=120,
)
response.raise_for_status()
body = response.json()
if response.status_code == 202:
body = wait_for_result(body["run_id"]) # read GET /v1/runs/{run_id}/result
items = body["records"]
Use the detail input your API publishes: enrich_records on newer APIs, detail_limit on older ones. An API without a detail plan has neither. See run an API.
Things that are new compared with your own scraper:
- Idempotency key. Generate one per intended run and store it before sending. If the connection drops, resend with the same key instead of starting a second run. See idempotency.
- Long runs answer
202. A run that takes longer thanwait_seconds(90 seconds by default) keeps going on the server. ReadGET /v1/runs/{run_id}/resultafterRetry-Afterseconds, or read records as they arrive from/items. Don't resend the POST to check progress. The Python guide has a complete script that does this. - Records, not pages. There is no page loop in your code. A run ends when the site has no more records, at
max_items, attimeout_seconds, or when your Units or usage budget run out, andstop_reasonsays which. Check it before treating a result as complete (pagination and output).
4. Move the rest of the job#
Most of what surrounds a spider becomes smaller:
- Proxies, retries and blocks. Drop the proxy pool, the retry settings and the delay and concurrency tuning (in Scrapy, settings such as
DOWNLOAD_DELAY). A failed run returns an error you handle once, and the same idempotency key returns the same error; see errors and retries. - Schedules. Keep cron, Airflow or your queue worker. It starts the run with a fresh idempotency key and reads the result. Schedule collection covers overlapping runs and uncertain outcomes. CleanedWeb has no scheduler of its own.
- Item pipelines. The API drops repeated records within one run. Across runs you still need your own key, so keep the pipeline's write step and its cross-run checks. Store API results covers stable keys and run envelopes.
- Reading records while a run is going. Start the run with
POST /v1/sources/{source_id}/runsand readGET /v1/runs/{run_id}/items?cursor=…, followingnext_cursor, much as a Scrapy item feed or Playwright loop produced rows as it went. - Breakage. You no longer fix selectors. When the site changes, a failed run tells you, and CleanedWeb's scheduled checks catch it too. A broken catalog API is rebuilt and swapped in once it qualifies; a contract change to an API you built waits for your review. Pin
expected_versionso your code finds out when the definition changes; see source changes.
5. Retire the infrastructure one spider at a time#
Don't turn everything off at once.
- Run the spider and the API on the same scope, on the same day.
- Match records by URL, SKU or another stable identifier. Check the fields your code reads, including formats such as prices, dates and currencies.
- Keep the spider scheduled as a fallback while the API runs beside it. If the API's output is wrong or a run fails, your pipeline can still take the spider's output.
- When the API has matched for as long as you need to trust it, switch that call, then disable the spider's schedule.
- Remove the servers, proxy plan and alerts only when no remaining spider uses them.
Then do the next site. The migration guide has a fuller checklist for comparing output and rolling out, and the production checklist covers credentials, monitoring and recovery.