HTTP error
HTTP 429 Too Many Requests
HTTP 429 Too Many Requests means the client has sent too many requests in a given time. It is the standard signal for rate limiting, and the response often says how long to wait.
Why a pipeline gets a 429
- Parallel workers, or no delay between pages of a long list.
- A retry loop that adds load exactly when the server is asking for less.
- Several jobs sharing one IP address or API key, so their requests count together.
- A limit lower than you assumed, or one that applies per endpoint or per client name.
- A burst after a backlog, such as a scheduler catching up on missed runs.
HTTP/1.1 429 Too Many Requests
Retry-After: 30
What to do
- Read
Retry-After. It is a number of seconds or an HTTP date. Wait at least that long before the next request to that host. - Without the header, back off exponentially with random jitter, a cap on the wait, and a maximum number of attempts.
- Lower concurrency and add a delay between requests. A 429 says your rate is too high, not that you need more retries.
- Cache responses so you do not fetch the same URL twice.
- If 429s persist at a reasonable rate, ask the provider for a higher limit, or for a feed or export.
- Do not rotate IP addresses or keys to dodge the limit.
Keep the original request and treat the 429 as a retryable failure. It is never a sign that there is no data.
In CleanedWeb
The CleanedWeb API asks the same of you: on a 429, respect Retry-After when it is present, reduce request frequency, and preserve any existing run identity rather than starting a new run. Do not raise collection limits automatically after an error.
Waiting for capacity is not an error. A new run waits in the queue (status: queued) while the workspace already has a run in progress in the same lane, and a 202 answer means the run continues. After Retry-After seconds, read GET /v1/runs/{run_id}/result; do not send the POST again.
Tools and docs
- HTTP status checker See the status code, redirects and headers a URL answers a script with.
- Errors and retries The status table, retry boundaries and how to recover an uncertain run.
- Schedule API runs One active execution per source, with limits you set.
- Idempotency and recovery Keep the request identity so a retry returns the same run.
Related terms
-
Rate limiting
How servers cap request rates, and how to throttle and back off when collecting data.
-
HTTP 503 Service Unavailable
The server cannot answer right now. How to retry with backoff and not read it as no data.
-
HTTP 403 Forbidden
The server understood the request and refuses it. Why it happens and how to respond.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.