Move from Apify
Replace an Apify Actor run with a CleanedWeb API that returns records from the same site. Move one Actor at a time; the run model is close to Apify's, so most of your code keeps its shape.
On this page
For the product comparison and current prices, see CleanedWeb vs. Apify.
Concepts side by side#
| Apify | CleanedWeb |
|---|---|
| Actor from the Store | A published API from the catalog |
| Your own Actor, or a generic scraper Actor you configure | An API built for your site in the API generator |
| Actor input (JSON) | input, defined by the API's published contract |
| Actor build or version | Definition version; pin it with expected_version |
Run, defaultDatasetId |
Run, run_id |
GET /v2/datasets/{id}/items |
GET /v1/runs/{run_id}/items, paged by next_cursor |
run-sync-get-dataset-items |
POST /v1/sources/{source_id}/run, which waits up to wait_seconds |
maxItems, run timeout |
max_items, timeout_seconds |
| Abort a run | POST /v1/runs/{run_id}/abort |
| Schedules and webhooks | Your own scheduler; see schedule collection |
| Platform usage and compute units | Units for the records returned; see units and billing |
The biggest difference is maintenance. When the site changes, an Actor waits for its author to fix it. CleanedWeb checks its APIs on a schedule. A broken catalog API is rebuilt and swapped in behind the same URL once it qualifies; an API you built never changes its contract until you review the change. See source changes.
1. Write down the Actor run you rely on#
For one Actor your code calls today, save:
- the Actor input you send, and which of its values change between runs;
- the dataset fields your code reads;
- one recent dataset, as the baseline you will compare against.
2. Find or build the API#
Search the API catalog for the site first. A published API works like a Store Actor: read its contract, send its input and run it. See run a catalog API.
If the site isn't there, open the API generator with the page your Actor starts from. Preview the records, select the collection, and keep the fields from step 1. Copy the source ID, API origin and definition version from the connection details once the build is accepted.
3. Replace the call#
A typical Apify client call:
from apify_client import ApifyClient
client = ApifyClient(APIFY_TOKEN)
run = client.actor("someone/site-scraper").call(
run_input={"search": "desk", "maxItems": 100},
)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
The same job in CleanedWeb, started in the background and read as it runs:
import os, time, requests
BASE = os.environ["CLEANEDWEB_API_BASE"]
HEADERS = {"Authorization": f"Bearer {os.environ['CLEANEDWEB_API_KEY']}"}
run = requests.post(
f"{BASE}/v1/sources/{os.environ['CLEANEDWEB_SOURCE_ID']}/runs",
headers={**HEADERS, "Idempotency-Key": saved_key}, # saved before sending
json={
"expected_version": os.environ["CLEANEDWEB_DEFINITION_VERSION"],
"input": {"query": "desk"},
"max_items": 100,
},
timeout=60,
).json()
items, cursor = [], 0
while cursor is not None:
page = requests.get(
f"{BASE}/v1/runs/{run['run_id']}/items",
headers=HEADERS, params={"cursor": cursor}, timeout=60,
).json()
items += page["items"]
cursor = page["next_cursor"]
if page["run_status"] in ("queued", "running") and not page["items"]:
time.sleep(30)
next_cursor becomes null once the run has ended and every record has been read. For a short run, call /run instead of /runs: it waits up to wait_seconds (90 by default) and answers with the records, like run-sync-get-dataset-items.
Things that work differently from Apify:
- Idempotency key. Every run request carries one. Save it before sending and reuse it to recover a request whose response was lost; a new key starts a new run. See idempotency.
- Inputs are checked. Unknown input fields and values outside the contract are rejected with an error instead of being ignored.
- Why a run ended is in
stop_reason:source_exhausted,max_items,timeout,budget_exhaustedand others in run an API. - Detail pages. Where the API has them,
enrich_records: truereads each record's detail page and merges its fields into the record.
4. Move schedules and webhooks#
Apify runs schedules for you. With CleanedWeb, your scheduler (cron, a queue worker, your orchestrator) starts the run with a fresh idempotency key and reads its result. Schedule collection covers how to avoid overlapping runs and what to do when an outcome is uncertain. Where you used an Apify webhook to start downstream work, start it when the run's /result answers 200.
5. Compare, then switch#
Run the Actor and the new API on the same scope, match items by URL or another stable identifier, and compare the fields your code reads. When they match, switch that call and turn off the Actor's schedule. Then move the next Actor.
The migration guide has a fuller checklist for comparing output, and the production checklist covers credentials, monitoring and recovery.