CleanedWeb
Markdown

Move from Firecrawl

Replace a Firecrawl scrape, crawl or extract job with a CleanedWeb API that returns the same records on every run. Move one site at a time and keep Firecrawl for the work it does well, like web search and reading arbitrary pages as Markdown.

On this page

For the product comparison and current prices, see CleanedWeb vs. Firecrawl.

What moves and what stays#

Move a job when your code runs it against the same site again and again and reads fields out of the result. Keep Firecrawl for one-off pages, search, and agents reading content they have never seen before.

You use Firecrawl for In CleanedWeb
/scrape with a JSON schema on a listing or search page One API for that collection. Its fields are reviewed once, at build time, and are the same on every run.
/crawl across a site, then parsing each page A collection API with pagination and, where the site has them, detail pages merged into each record (enrich_records).
/extract over many URLs with a prompt A collection API. You choose the fields during the build instead of describing them in a prompt on every call.
Monitoring a page for changes Scheduled runs of the API. CleanedWeb checks its APIs on a schedule and rebuilds broken catalog APIs itself; see source changes.
Search, Markdown for an LLM, browser actions Keep Firecrawl.

Concepts side by side#

Firecrawl CleanedWeb
A URL plus a schema or prompt on every request A source with a reviewed definition version; your request sends only input
formats: [{"type": "json", "schema": ...}] The API's published output schema, from its OpenAPI contract
data.json on one page records: every matching record, de-duplicated across pages
Crawl job id and its status endpoint run_id, GET /v1/runs/{run_id} and /items
limit on a crawl (pages) max_items (records) and timeout_seconds
Credits per page Units for the records returned; see units and billing

1. Write down the job you run today#

Pick one Firecrawl call your code makes on a schedule. Save:

  • the URL or URLs it reads, and any query parameters you change between calls (a search term, a city, a category);
  • the schema or prompt, and the fields your code actually reads from the result;
  • one recent response, as the baseline you will compare against.

The query parameters you change become the API's inputs. The fields your code reads are the fields you keep in the build.

2. Build the API#

Open the API generator with the same URL. Preview the records, select the collection, and keep the fields from step 1. Name them the way your code expects so the mapping later stays small. If the site is already in the API catalog, you can use the published API and skip the build.

When the build is accepted, copy the source ID, API origin and definition version from its connection details. The quickstart shows where each one is.

3. Replace the call#

A typical Firecrawl scrape with a schema:

Python
from firecrawl import Firecrawl

app = Firecrawl(api_key=FIRECRAWL_API_KEY)
result = app.scrape(
    "https://shop.test/search?q=desk",
    formats=[{"type": "json", "schema": Product.model_json_schema()}],
)
items = result.json["products"]

The same job in CleanedWeb sends only the input. The site, the fields and how to page through results are already part of the API:

Python
import os, requests

response = requests.post(
    f"{os.environ['CLEANEDWEB_API_BASE']}/v1/sources/{os.environ['CLEANEDWEB_SOURCE_ID']}/run",
    headers={
        "Authorization": f"Bearer {os.environ['CLEANEDWEB_API_KEY']}",
        "Idempotency-Key": saved_key,  # saved before sending; see below
    },
    json={
        "expected_version": os.environ["CLEANEDWEB_DEFINITION_VERSION"],
        "input": {"query": "desk"},
        "max_items": 100,
    },
    timeout=120,
)
response.raise_for_status()
body = response.json()
if response.status_code == 202:
    body = wait_for_result(body["run_id"])  # read GET /v1/runs/{run_id}/result
items = body["records"]

Two things are new compared with Firecrawl:

  • Idempotency key. Generate one per intended run and store it before sending. If the connection drops, resend with the same key instead of starting a second run. See idempotency.
  • Long runs answer 202. A run that takes longer than wait_seconds (90 seconds by default) keeps going on the server. Read GET /v1/runs/{run_id}/result after Retry-After seconds, or stream records as they arrive from /items. Don't resend the POST to check progress. The Python guide includes a complete script that does this.

4. Replace a crawl#

A Firecrawl crawl gives you pages and leaves turning them into records to you. A CleanedWeb collection API returns the records directly and follows the site's own pagination until the site runs out, max_items is reached, or timeout_seconds ends the run.

Firecrawl crawl CleanedWeb run
POST /v2/crawl returns a job id POST /v1/sources/{source_id}/runs returns a run object with run_id
Read the job status and its next page of results Read GET /v1/runs/{run_id}/items?cursor=…, following next_cursor
Parse each page's content into records Records arrive with the API's schema
Fetch detail pages in a second pass enrich_records: true merges each record's detail page

stop_reason on the run tells you why it ended: source_exhausted when the site had nothing more, max_items, timeout, and others listed in run an API.

5. Compare, then switch#

Run both tools against the same scope and match records by URL or another stable identifier. Check the fields your code reads, including formats such as prices, dates and currencies. When the records match what you need, switch that part of your code to the new API and remove the Firecrawl call for that site.

Then do the next site. The migration guide has a fuller checklist for comparing output and rolling out, and the production checklist covers credentials, monitoring and recovery.

Updated Get help
Search across the documentation · Esc to close