Source changes
Websites evolve. A collection can change its layout, URLs, fields, available records, or access requirements. Review the returned records and execution summary before sending changed output downstream.
On this page
What CleanedWeb does when a site changes#
CleanedWeb checks the catalog APIs it publishes once a day, and the APIs you build while you are still running them; an API you have not run for 30 days is no longer checked. A failed check is repeated about 30 minutes later before it counts, then re-checked, less often over time, until it passes. These checks alert the CleanedWeb team. Your workspace notifications show failed runs of your own APIs and saved schema changes, so a site change reaches you there when one of your runs fails.
Catalog APIs are repaired by CleanedWeb. When a catalog API keeps failing, CleanedWeb rebuilds it from the site as it is now. The rebuild replaces the published version only after it passes qualification. It keeps the same API ID, URL and keys, and the previous version is kept so the change can be rolled back; a failed check right after the swap rolls it back. A catalog run uses the newest minor of the major you pinned (the latest major when you pin none), so a repair doesn't change your requests. If the rebuild removes or retypes a field or an input, it doesn't replace the version you pin: where a shim keeps the old contract answering (a removed field stays as null, a new required input ships with a default) it is a minor of the same API, and otherwise it is published as the next major of the same API, with the same API ID, URL and keys, while the major you pinned keeps answering for 60 days. A change that only retypes a field waits for one confirming rebuild before it ships, and an API that was never served takes minors only. See Catalog API versions. While an API is failing it can be withdrawn from the public catalog, and it is listed again only after repeated passing checks. Repair isn't guaranteed: when rebuilds keep failing, the API goes to the CleanedWeb team.
Your own APIs never break their contract on their own. A change that removes or retypes a field, or changes what an existing request collects, is never applied to an API you built until you review and activate it. The one exception is a strictly additive update, which CleanedWeb can activate for you: new optional output fields, new optional inputs, or an input moved with every call it accepted kept. It is recorded in the API's version history and can be reverted. Every new version has a new definition version, so runs that pin expected_version to the previous one answer 409 with builder_definition_version_conflict until you review the change and update the pin; runs that don't pin a version use the active one. See versioning.
Some changes can't be repaired within the same scope, for example when a site starts requiring a login or removes the records. An API you built then fails its runs until you rebuild or update it.
A partial run is still partial#
Say a GitHub topic API returned some repositories after GitHub changed its topic page, but could not finish within the requested scope. The response, shortened, could look like this:
{
"records": [{ "title": "creeper", "stargazer_count": "12", "url": "https://github.com/z7r1k3/creeper" }],
"summary": {
"status": "partial",
"within_scope_passed": false,
"catalog_complete": false
}
}
Nonempty records and HTTP 200 do not override summary.status or summary.within_scope_passed. catalog_complete: false also rules out a full-catalog claim.
Preserve evidence and decide#
Keep the records with their summary, requested limits, source identifier, definition version, and non-secret run or trace identifier. Compare the returned fields and examples with the accepted schema. If your workflow requires complete coverage or a required field is missing, hold the result from downstream use while you investigate.
Check whether the source definition or schema changed. Review a new definition before updating the version pinned by your application. Some changes require a different collection approach or make the source unavailable; a successful retry alone does not prove recovery.
Continue safely#
Read pagination and output to distinguish a deliberately bounded run from an incomplete one, and errors and retries before repeating an uncertain execution. API status covers the catalog APIs; it does not establish the health of every generated API.