HTTP error
HTTP 403 Forbidden
HTTP 403 Forbidden means the server understood the request and refuses to fulfil it. Unlike 401, sending the same request again will not help unless something about the request or the client changes.
Why a scraper or pipeline gets a 403
- The content needs a login, or a role or subscription your client does not have. It is not a public page.
- A firewall or bot-protection rule flagged the client: a burst of requests, an IP range such as a cloud provider's, unusual headers, or a client that identifies itself as an automated tool. The block may come as a challenge page.
- The site restricts automated access, which its robots.txt or terms usually say.
- A signed URL or token has expired, or the region is restricted.
- Earlier requests triggered a temporary block that has not lifted yet.
What to do
- Open the URL in an ordinary browser. If it asks you to sign in, the data is not public, and a public-page collector should leave it alone.
- Read the site's robots.txt and terms. A 403 for automated clients is often an explicit no.
- If the 403s began after a burst, cut concurrency, add delays and wait. Temporary blocks tend to lift.
- Identify your client honestly, with a descriptive user agent and a contact address. Do not pretend to be a browser.
- If you need the data, ask the owner for permission, or use an official API, feed or data export.
- Do not rotate IP addresses, forge headers or defeat checks to get past the block. The site has said no, and doing that can breach its terms.
A 403 is rarely worth retrying, so cap the attempts. Log it with the URL, and never record it as an empty result: a refused request is not a page with no records.
In CleanedWeb
CleanedWeb reads public pages only. A run whose source refuses access ends with stop_reason: blocked and status failed, so check the run's summary instead of expecting records.
When CleanedWeb's own API answers 403, the credential is not tied to a workspace account or lacks permission. Use a workspace key with the run scope, and check the key's scopes, the selected API's access and the workspace membership. Repeating the request does not grant access.
Tools and docs
- HTTP status checker See the status code, redirects and headers a URL answers a script with.
- robots.txt tester See whether a site asks crawlers to stay out of the URLs you need.
- Authentication and limits API key scopes, access restrictions and rotation.
- Errors and retries How to respond to each status from the CleanedWeb API.
Related terms
-
HTTP 401 Unauthorized
The request lacks valid credentials. How it differs from 403 and how to fix it.
-
Cloudflare challenge page
The interstitial a site shows instead of its content. How to recognise it and respond.
-
robots.txt
The file where a site tells crawlers what they may fetch, and what it does not do.
-
HTTP 429 Too Many Requests
You sent requests too fast. How to read Retry-After, back off and slow down.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.