HTTP error
HTTP 503 Service Unavailable
HTTP 503 Service Unavailable means the server cannot handle the request right now, usually because it is overloaded or down for maintenance. It is meant to be temporary, and the response may include a Retry-After header.
Why a pipeline gets a 503
- The site is overloaded, sometimes by your own crawler.
- Planned maintenance or a deployment is in progress.
- A service the site depends on is down.
- A protection layer is shedding load, or showing a challenge page with a 503 status.
- A rate limiter answers 503 where others would use 429.
What to do
- Retry with exponential backoff and jitter, a cap on the wait and a maximum number of attempts. Honour
Retry-Afterwhen it is there. - Add a circuit breaker: after repeated failures, stop calling for a while instead of adding load to a struggling server.
- Look at the body. A short HTML page full of scripts is a bot check, not an outage, and retrying faster will not help.
- Keep your last good data and mark it stale. Do not write an empty result, and do not mark records as removed because one request failed.
- Alert when the outage outlasts your freshness window (see data freshness), and check the site's status page if it has one.
In CleanedWeb
On the CleanedWeb API a 503 is never an empty result set. builder_billing_unavailable means billing could not be confirmed, so the run was refused rather than run for free. runs_unavailable means the run store could not answer. A 503 with no code means execution or catalog storage was unavailable.
Retry later with the same Idempotency-Key and the same request. A repeated request returns the existing run, so a retry does not start duplicate work. The status page covers the published catalog APIs; it does not establish the health of every generated API.
Tools and docs
- HTTP status checker See the status code, redirects and headers a URL answers a script with.
- API status Live health of the published catalog APIs.
- Errors and retries Which failures are safe to retry, and how to recover an uncertain run.
- Idempotency and recovery Retry a lost or failed request without starting a second run.
Related terms
-
HTTP 429 Too Many Requests
You sent requests too fast. How to read Retry-After, back off and slow down.
-
Rate limiting
How servers cap request rates, and how to throttle and back off when collecting data.
-
Cloudflare challenge page
The interstitial a site shows instead of its content. How to recognise it and respond.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.