Glossary
Rate limiting
Rate limiting is a server capping how many requests a client can make in a window of time, for example a set number per minute for each API key or IP address. Requests over the cap are delayed or refused, usually with HTTP 429.
How a server signals it
- Status code. 429 Too Many Requests is the standard answer. Some servers use 503 instead.
Retry-After. A header that gives the wait as a number of seconds or as an HTTP date. When it is present it is the best guide to when to try again.- Quota headers. Many APIs add headers such as
X-RateLimit-RemainingandX-RateLimit-Reset. Names and meanings vary between services, so read the provider's documentation.
Staying under a limit
- Find the published limit, and treat it as a ceiling to stay well below, not a target.
- Throttle on your side: a queue with fixed concurrency, and a delay between requests to the same host.
- On a 429, wait for
Retry-After. Without it, back off exponentially (1 s, 2 s, 4 s and so on) with random jitter, a cap, and a maximum number of attempts. - Cache what you have already fetched, so a retry or a re-run does not repeat requests.
- Do not let several workers share one key and all retry at once; that turns one limit hit into a burst.
A site with no published limit still has one. Send fewer requests than you think you need, identify your client honestly, and read the site's robots.txt. Repeatedly hitting a limit can get a client blocked for longer, and a pipeline that reads a 429 as an empty result quietly loses data.
In CleanedWeb
CleanedWeb's API asks the same of its callers: on a 429, respect Retry-After when it is present and reduce request frequency. A source run that is waiting for capacity is queued (status: queued), not rejected, and a 202 answer carries Retry-After for reading the result. Do not resend the POST to check progress; read GET /v1/runs/{run_id}/result instead.
The docs do not promise a universal concurrency allowance. Start with one active execution per source, follow the limits available for your API and workspace, and do not raise collection limits automatically after an error.
Tools and docs
- Errors and retries What each status means for your client and when a retry is safe.
- Schedule API runs One attempt at a time, with limits set in your own scheduler.
- robots.txt tester Check which paths a site asks crawlers to leave alone.
Related terms
-
HTTP 429 Too Many Requests
You sent requests too fast. How to read Retry-After, back off and slow down.
-
HTTP 503 Service Unavailable
The server cannot answer right now. How to retry with backoff and not read it as no data.
-
robots.txt
The file where a site tells crawlers what they may fetch, and what it does not do.
-
Pagination
How lists are split into pages and how to collect every record without gaps or repeats.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.