CleanedWeb

Glossary

Rate limiting

Rate limiting is a server capping how many requests a client can make in a window of time, for example a set number per minute for each API key or IP address. Requests over the cap are delayed or refused, usually with HTTP 429.

How a server signals it

Staying under a limit

  1. Find the published limit, and treat it as a ceiling to stay well below, not a target.
  2. Throttle on your side: a queue with fixed concurrency, and a delay between requests to the same host.
  3. On a 429, wait for Retry-After. Without it, back off exponentially (1 s, 2 s, 4 s and so on) with random jitter, a cap, and a maximum number of attempts.
  4. Cache what you have already fetched, so a retry or a re-run does not repeat requests.
  5. Do not let several workers share one key and all retry at once; that turns one limit hit into a burst.

A site with no published limit still has one. Send fewer requests than you think you need, identify your client honestly, and read the site's robots.txt. Repeatedly hitting a limit can get a client blocked for longer, and a pipeline that reads a 429 as an empty result quietly loses data.

In CleanedWeb

CleanedWeb's API asks the same of its callers: on a 429, respect Retry-After when it is present and reduce request frequency. A source run that is waiting for capacity is queued (status: queued), not rejected, and a 202 answer carries Retry-After for reading the result. Do not resend the POST to check progress; read GET /v1/runs/{run_id}/result instead.

The docs do not promise a universal concurrency allowance. Start with one active execution per source, follow the limits available for your API and workspace, and do not raise collection limits automatically after an error.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.