CleanedWeb

HTTP error

HTTP 429 Too Many Requests

HTTP 429 Too Many Requests means the client has sent too many requests in a given time. It is the standard signal for rate limiting, and the response often says how long to wait.

Why a pipeline gets a 429

HTTP/1.1 429 Too Many Requests
Retry-After: 30

What to do

  1. Read Retry-After. It is a number of seconds or an HTTP date. Wait at least that long before the next request to that host.
  2. Without the header, back off exponentially with random jitter, a cap on the wait, and a maximum number of attempts.
  3. Lower concurrency and add a delay between requests. A 429 says your rate is too high, not that you need more retries.
  4. Cache responses so you do not fetch the same URL twice.
  5. If 429s persist at a reasonable rate, ask the provider for a higher limit, or for a feed or export.
  6. Do not rotate IP addresses or keys to dodge the limit.

Keep the original request and treat the 429 as a retryable failure. It is never a sign that there is no data.

In CleanedWeb

The CleanedWeb API asks the same of you: on a 429, respect Retry-After when it is present, reduce request frequency, and preserve any existing run identity rather than starting a new run. Do not raise collection limits automatically after an error.

Waiting for capacity is not an error. A new run waits in the queue (status: queued) while the workspace already has a run in progress in the same lane, and a 202 answer means the run continues. After Retry-After seconds, read GET /v1/runs/{run_id}/result; do not send the POST again.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.