# Move from Zyte API

Replace the crawler you built around Zyte API with a CleanedWeb API that returns the records directly. Move one site at a time and keep Zyte for the work it does well: fetching arbitrary URLs through its unblocking and browser rendering, and extracting from sites you have not built.

For the product comparison and current prices, see [CleanedWeb vs. Zyte API](/compare/zyte/).

## What moves and what stays

Move a site when your code fetches its listing pages, follows its pagination, extracts each page and then assembles the results. Zyte API is a fetch-and-extract call. The crawl around it is yours: which URLs to request, how to page, how to de-duplicate, when to stop and what to do when the site changes.

| You use Zyte API for | In CleanedWeb |
| --- | --- |
| `product`, `productList`, `article` or `jobPosting` extraction on pages of one site | One API for that collection. Its fields are reviewed once, at build time, and are the same on every run. |
| Your own loop over listing pages and `nextPage` links | The run follows the site's pagination until the site runs out, `max_items` is reached, or `timeout_seconds` ends it. |
| A second pass that extracts each detail page | `enrich_records: true`, where the API has detail pages, merges each record's detail page into the record. |
| Scrapy spiders using `scrapy-zyte-api` | A call to the API from your scheduler. The spider for that site goes away. |
| Fetching arbitrary URLs, browser rendering, custom actions, sites you have not built | Keep Zyte API. |

## Concepts side by side

| Zyte API | CleanedWeb |
| --- | --- |
| `POST https://api.zyte.com/v1/extract` with a `url`, per page | `POST /v1/sources/{source_id}/run` with `input`, per collection |
| Basic auth, API key as the username | `Authorization: Bearer` with a server credential that has the `run` scope |
| One data type per request: `product`, `productList`, `article`, `jobPosting` and others | The API's published output schema, from its OpenAPI contract |
| Fields set by the vendor's schema for that data type | Fields you reviewed for this source during the build |
| `extractFrom` to choose HTTP body or browser HTML | Not a setting you make; how the site is read is part of the API |
| A response per URL | `records`: every matching record, de-duplicated across pages |
| Billed per successful response | Units for the records returned; see [units and billing](/docs/billing/) |

Zyte's automatic extraction returns the same schema for every site of a data type, which is what makes it work on sites nobody has configured. A CleanedWeb API is built for one site, so its fields can include whatever that site shows, such as a seller, a size chart or a custom attribute, and you decide at build time which ones to keep. If your code relies on a Zyte field that the site does not publish in a stable place, expect it to be null or absent in both tools.

## 1. Write down the job you run today

Pick one site your crawler covers. Save:

- the listing or search URLs it starts from, and anything you change between runs (a search term, a category, a city);
- the data type you request and the fields your code reads from it, including any you compute afterwards;
- how your crawler pages and when it stops;
- one recent output, as the baseline you will compare against.

The values you change between runs become the API's inputs. The fields your code reads are the fields you keep in the build.

## 2. Build or find the API

Open the [API generator](/start/) with the same starting URL. Preview the records, select the collection, and keep the fields from step 1. Name them the way your code expects so the mapping later stays small. If the site is already in the [API catalog](/catalog/), use the published API and skip the build; see [run a catalog API](/docs/run-catalog-api/).

When the build is accepted, copy the source ID, API origin and definition version from its connection details. The [quickstart](/docs/quickstart/) shows where each one is.

## 3. Replace the calls

A typical Zyte API crawl: read a listing page, take its next-page link, then extract each product from its own URL.

```python
import os, requests

ZYTE_API_KEY = os.environ["ZYTE_API_KEY"]
ZYTE = ("https://api.zyte.com/v1/extract", (ZYTE_API_KEY, ""))
url, products = "https://shop.test/desks", []
while url:
    page = requests.post(
        ZYTE[0], auth=ZYTE[1],
        json={"url": url, "productNavigation": True},
    ).json()["productNavigation"]
    for item in page.get("items", []):
        detail = requests.post(
            ZYTE[0], auth=ZYTE[1],
            json={"url": item["url"], "product": True},
        ).json()["product"]
        products.append(detail)
    url = (page.get("nextPage") or {}).get("url")
```

The same job in CleanedWeb is one request. The site, the fields and the paging are already part of the API:

```python
import os, requests

response = requests.post(
    f"{os.environ['CLEANEDWEB_API_BASE']}/v1/sources/{os.environ['CLEANEDWEB_SOURCE_ID']}/run",
    headers={
        "Authorization": f"Bearer {os.environ['CLEANEDWEB_API_KEY']}",
        "Idempotency-Key": saved_key,  # saved before sending; see below
    },
    json={
        "expected_version": os.environ["CLEANEDWEB_DEFINITION_VERSION"],
        "input": {"category": "desks"},
        "max_items": 500,
    },
    timeout=120,
)
response.raise_for_status()
body = response.json()
if response.status_code == 202:
    body = wait_for_result(body["run_id"])  # read GET /v1/runs/{run_id}/result
products = body["records"]
```

Two things are new compared with Zyte API:

- **Idempotency key.** Generate one per intended run and store it before sending. If the connection drops, resend with the same key instead of starting a second run. See [idempotency](/docs/idempotency/).
- **Long runs answer `202`.** A run that takes longer than `wait_seconds` (90 seconds by default) keeps going on the server. Read `GET /v1/runs/{run_id}/result` after `Retry-After` seconds, or read records as they arrive from `/items`. Don't resend the POST to check progress. The [Python guide](/docs/python/) includes a complete script that does this.

## 4. Remove the crawl code

Most of what you wrote around Zyte has a counterpart in the run:

| Your crawler code | CleanedWeb run |
| --- | --- |
| Loop on `nextPage` until it is missing | The run follows the site's pagination itself |
| A second request per item for detail fields | `enrich_records: true` merges each record's detail page |
| De-duplicating items seen on more than one page | Records are de-duplicated by identity within the run |
| A page or item counter that stops the loop | `max_items` and `timeout_seconds`; `stop_reason` says why the run ended |
| Retrying `429` and `503` responses from Zyte | Run errors and retries are described in [errors and retries](/docs/errors/); a repeated key returns the same run |
| Fixing selectors or switching `extractFrom` when a site changes | CleanedWeb maintains the API; a changed definition arrives as a new version you review before you update the one you pin. See [source changes](/docs/source-changes/) and [versions](/docs/versioning/) |

If you drive Zyte from Scrapy with `scrapy-zyte-api`, the spider for a migrated site can be deleted, along with its entries in your Scrapy project. Keep the Scrapy project and the `scrapy-zyte-api` settings for sites you have not moved. A job that calls the API on a schedule replaces the spider's cron entry; [schedule collection](/docs/scheduled-runs/) covers how to avoid overlapping runs.

Zyte bills each successful response, so a crawl of 40 listing pages and 800 detail pages is 840 billed responses. CleanedWeb bills the records the run returns. Compare the cost of the whole job, not one call.

## 5. Compare, then switch

Run both on the same scope and match records by URL or another stable identifier. Check the fields your code reads, including formats such as prices, currencies and dates. Zyte returns `price` and `currency` as separate fields; confirm the API you built does the same, or adjust your mapping once.

When the records match what you need, switch that part of your code to the new API, turn off the crawler's schedule for that site, and remove its Zyte calls. Then do the next site.

The [migration guide](/docs/migration-guide/) has a fuller checklist for comparing output and rolling out, and the [production checklist](/docs/production-checklist/) covers credentials, monitoring and recovery.
