# Move from Firecrawl

Replace a Firecrawl scrape, crawl or extract job with a CleanedWeb API that returns the same records on every run. Move one site at a time and keep Firecrawl for the work it does well, like web search and reading arbitrary pages as Markdown.

For the product comparison and current prices, see [CleanedWeb vs. Firecrawl](/compare/firecrawl/).

## What moves and what stays

Move a job when your code runs it against the same site again and again and reads fields out of the result. Keep Firecrawl for one-off pages, search, and agents reading content they have never seen before.

| You use Firecrawl for | In CleanedWeb |
| --- | --- |
| `/scrape` with a JSON schema on a listing or search page | One API for that collection. Its fields are reviewed once, at build time, and are the same on every run. |
| `/crawl` across a site, then parsing each page | A collection API with pagination and, where the site has them, detail pages merged into each record (`enrich_records`). |
| `/extract` over many URLs with a prompt | A collection API. You choose the fields during the build instead of describing them in a prompt on every call. |
| Monitoring a page for changes | Scheduled runs of the API. CleanedWeb checks its APIs on a schedule and rebuilds broken catalog APIs itself; see [source changes](/docs/source-changes/). |
| Search, Markdown for an LLM, browser actions | Keep Firecrawl. |

## Concepts side by side

| Firecrawl | CleanedWeb |
| --- | --- |
| A URL plus a schema or prompt on every request | A source with a reviewed definition version; your request sends only `input` |
| `formats: [{"type": "json", "schema": ...}]` | The API's published output schema, from its OpenAPI contract |
| `data.json` on one page | `records`: every matching record, de-duplicated across pages |
| Crawl job `id` and its status endpoint | `run_id`, `GET /v1/runs/{run_id}` and `/items` |
| `limit` on a crawl (pages) | `max_items` (records) and `timeout_seconds` |
| Credits per page | Units for the records returned; see [units and billing](/docs/billing/) |

## 1. Write down the job you run today

Pick one Firecrawl call your code makes on a schedule. Save:

- the URL or URLs it reads, and any query parameters you change between calls (a search term, a city, a category);
- the schema or prompt, and the fields your code actually reads from the result;
- one recent response, as the baseline you will compare against.

The query parameters you change become the API's inputs. The fields your code reads are the fields you keep in the build.

## 2. Build the API

Open the [API generator](/start/) with the same URL. Preview the records, select the collection, and keep the fields from step 1. Name them the way your code expects so the mapping later stays small. If the site is already in the [API catalog](/catalog/), you can use the published API and skip the build.

When the build is accepted, copy the source ID, API origin and definition version from its connection details. The [quickstart](/docs/quickstart/) shows where each one is.

## 3. Replace the call

A typical Firecrawl scrape with a schema:

```python
from firecrawl import Firecrawl

app = Firecrawl(api_key=FIRECRAWL_API_KEY)
result = app.scrape(
    "https://shop.test/search?q=desk",
    formats=[{"type": "json", "schema": Product.model_json_schema()}],
)
items = result.json["products"]
```

The same job in CleanedWeb sends only the input. The site, the fields and how to page through results are already part of the API:

```python
import os, requests

response = requests.post(
    f"{os.environ['CLEANEDWEB_API_BASE']}/v1/sources/{os.environ['CLEANEDWEB_SOURCE_ID']}/run",
    headers={
        "Authorization": f"Bearer {os.environ['CLEANEDWEB_API_KEY']}",
        "Idempotency-Key": saved_key,  # saved before sending; see below
    },
    json={
        "expected_version": os.environ["CLEANEDWEB_DEFINITION_VERSION"],
        "input": {"query": "desk"},
        "max_items": 100,
    },
    timeout=120,
)
response.raise_for_status()
body = response.json()
if response.status_code == 202:
    body = wait_for_result(body["run_id"])  # read GET /v1/runs/{run_id}/result
items = body["records"]
```

Two things are new compared with Firecrawl:

- **Idempotency key.** Generate one per intended run and store it before sending. If the connection drops, resend with the same key instead of starting a second run. See [idempotency](/docs/idempotency/).
- **Long runs answer `202`.** A run that takes longer than `wait_seconds` (90 seconds by default) keeps going on the server. Read `GET /v1/runs/{run_id}/result` after `Retry-After` seconds, or stream records as they arrive from `/items`. Don't resend the POST to check progress. The [Python guide](/docs/python/) includes a complete script that does this.

## 4. Replace a crawl

A Firecrawl crawl gives you pages and leaves turning them into records to you. A CleanedWeb collection API returns the records directly and follows the site's own pagination until the site runs out, `max_items` is reached, or `timeout_seconds` ends the run.

| Firecrawl crawl | CleanedWeb run |
| --- | --- |
| `POST /v2/crawl` returns a job `id` | `POST /v1/sources/{source_id}/runs` returns a run object with `run_id` |
| Read the job status and its `next` page of results | Read `GET /v1/runs/{run_id}/items?cursor=…`, following `next_cursor` |
| Parse each page's content into records | Records arrive with the API's schema |
| Fetch detail pages in a second pass | `enrich_records: true` merges each record's detail page |

`stop_reason` on the run tells you why it ended: `source_exhausted` when the site had nothing more, `max_items`, `timeout`, and others listed in [run an API](/docs/run-api/#the-run-object).

## 5. Compare, then switch

Run both tools against the same scope and match records by URL or another stable identifier. Check the fields your code reads, including formats such as prices, dates and currencies. When the records match what you need, switch that part of your code to the new API and remove the Firecrawl call for that site.

Then do the next site. The [migration guide](/docs/migration-guide/) has a fuller checklist for comparing output and rolling out, and the [production checklist](/docs/production-checklist/) covers credentials, monitoring and recovery.
