Schedule API runs
Run a catalog API or one of your own APIs on a recurring schedule. CleanedWeb can run it for you with a native schedule at /v1/schedules, or you can call it from a scheduler you operate, such as a server cron job, workflow runner, or task queue. Over MCP, the same native schedules are the schedule_run, list_schedules, pause_schedule, resume_schedule and delete_schedule tools; see MCP.
On this page
Complete the quickstart and verify your storage path before enabling a schedule.
Create a native schedule#
POST https://api.cleanedweb.com/v1/schedules
A schedule starts an ordinary run at each due slot: every hour, day or week, in UTC, at most once an hour. Each run is charged like any other, and you read it with the run routes. Creating one needs an API key with the run scope. The key that creates the schedule stands behind every run it starts, so it must keep that scope; revoking it, letting it expire or removing run pauses the schedule. A workspace holds at most 50 schedules, paused ones included.
This request runs the GitHub Topics Repositories API every day at 06:30 UTC:
curl --fail-with-body --max-time 30 \
"https://api.cleanedweb.com/v1/schedules" \
-H "Authorization: Bearer ${CLEANEDWEB_API_KEY}" \
-H "Idempotency-Key: daily-github-topics-1" \
-H 'Content-Type: application/json' \
--data '{"api_id":"generated_047a8ebb25911ccf62f4a212","endpoint_id":"generated_047a8ebb25911ccf62f4a212_search","input":{"topics":"web-scraping"},"max_items":100,"every":"day","at_utc":"06:30","name":"Daily GitHub topics"}'
| Field | Meaning |
|---|---|
api_id and endpoint_id, or source_id |
What to run: a catalog API endpoint, or one of your own sources. Give one target, not both. |
version |
Optional, catalog APIs only. The major to run; default the listing's current one. See Catalog API versions. |
input |
The API's inputs. Give every required input now; each slot runs with them. |
max_items, timeout_seconds, fields |
Optional, as on /run. They bound each run. |
every |
hour, day or week. |
at_utc |
Optional HH:MM in UTC, default 00:00. For hour, only the minute counts. |
weekday |
Optional, for week: 0 Monday to 6 Sunday, default Monday. |
run_now |
Optional. true also runs once at the next tick, not only from the next slot. |
name |
Optional label, up to 200 characters. |
Send an idempotency key, in the Idempotency-Key header or as idempotency_key in the body (if both are present they must match). A new schedule answers 201. The same key and arguments answer the same schedule with 200 and Idempotent-Replayed: true and never create a second one, so a lost answer is safe to retry. The same key with different arguments answers 409 with schedule_idempotency_conflict.
Manage native schedules#
| Request | Scope | Answer |
|---|---|---|
GET /v1/schedules |
read |
{schedules, limit}: every schedule of the workspace, paused ones included. |
GET /v1/schedules/{schedule_id} |
read |
The schedule. |
POST /v1/schedules/{schedule_id}/pause |
run |
The paused schedule. No slot starts after this answer. |
POST /v1/schedules/{schedule_id}/resume |
run |
The resumed schedule. It continues from now; slots missed while paused are counted in skipped_slots, not run. Only a paused schedule can be resumed (409 schedule_not_paused). |
DELETE /v1/schedules/{schedule_id} |
run |
{schedule_id, deleted}. A deleted schedule is gone for good. |
A run that was already being started when you paused or deleted the schedule still runs and is charged; the answer names it as inflight_run_id.
A schedule reports state (active or paused), cadence, next_due_at, last_run_id, last_status, consecutive_failures and skipped_slots. Read each run's records and summary with GET /v1/runs/{last_run_id}/result, using a key with the run or results scope, and check its summary before you publish records downstream.
When a native schedule skips or pauses#
A schedule never overlaps its own runs. While its previous run is still queued or running, a due slot is skipped (last_skip_reason is previous_run_active) and the next slot is tried. After three skips in a row, attention includes previous_run_stuck.
A schedule pauses itself and says why in pause_reason:
pause_reason |
Cause |
|---|---|
paused_by_customer |
You paused it. |
key_revoked, key_expired, key_scope_changed |
The key behind it can no longer run. |
insufficient_units, budget_exhausted |
The workspace could not fund a run. |
consecutive_failures |
Three runs in a row failed. A finished run resets the count. |
version_changed, api_unavailable, target_rejected |
The API changed or can no longer be run as scheduled. |
Fix the cause, then resume it. Resuming checks the target again.
Use your own scheduler#
Operate your own scheduler when you need a time zone other than UTC, a cadence a native schedule does not offer, or a review step before every run. The rest of this guide describes that customer-managed integration. Decide how fresh the data must be, how much collection each run may perform, and who investigates an incomplete or uncertain outcome.
Save a bounded job configuration#
Keep the API origin, source identifier, reviewed definition version, and accepted inputs in your job configuration. Load the credential from your server's secret manager at execution time. Check its scope and expiry, and rotate it before the schedule loses access.
Define operational limits in your scheduler:
| Setting | What to decide |
|---|---|
| Frequency | The observation interval your application actually needs |
| Time zone | An explicit zone, including behavior around daylight saving changes |
| Collection budget | max_items and detail reads for one execution |
| Maximum runtime | timeout_seconds for the run, and when the worker stops waiting and records an uncertain outcome |
| Concurrency | How overlapping attempts for the same source are prevented |
| Recovery owner | Who reviews unresolved execution or ingestion failures |
Use pagination and output to set the collection budget. max_items and timeout_seconds limit work within an execution; repeated requests do not mean “continue from the next page.” Avoid accumulating missed intervals into an unbounded catch-up batch.
Execute one attempt at a time#
Use a durable job record and a lock or queue partition for each source. Begin conservatively with one active execution per source, and account for any other consumers sharing it. Do not assume a universal concurrency allowance; follow the limits available for the API and workspace.
Before dispatch, save an application attempt ID, scheduled time, definition version, intended input, and an Idempotency-Key. Customer-workspace runs require this header. Persist one key for each intended execution; use a new key for the next scheduled observation. Issue the request once, save the response and its run ID (delivery.runId, or run_id from a 202 answer), then check its summary before publishing records downstream. A 202 means the run is still going; read its result by run ID rather than sending the POST again.
Prevent the next scheduled interval from starting overlapping work when the current attempt is still running or unresolved. If a worker exits or its lock expires, reconcile the prior execution before dispatching its replacement.
Separate retries from reconciliation#
Disable blind POST retries in the HTTP client, job runner, and infrastructure. A client timeout or lost response does not prove that the source execution stopped or that no usage occurred.
Record the outcome as unknown and inspect available workspace run evidence. Retain the same idempotency key and identical request when reconciling that execution; a replay returns the existing run instead of starting another. Do not generate a replacement key to get around an unresolved attempt. Read idempotency for replay behavior and errors and retries before deciding whether to resubmit. Honor returned retry timing for rejected requests.
When collection succeeded but ingestion failed, replay the saved response into your storage path. See storing results for deduplication and publication checks.
Review the schedule in operation#
Monitor the last accepted observation, partial results, authentication failures, version conflicts, and unresolved attempts. An active scheduler or recent heartbeat does not establish that usable records were delivered.
Pause affected jobs when a definition change requires review. Validate the replacement contract and a bounded response before resuming the normal interval. Use monitoring, definition versions, and the production checklist to keep that process repeatable.