CleanedWeb

HTTP error

HTTP 403 Forbidden

HTTP 403 Forbidden means the server understood the request and refuses to fulfil it. Unlike 401, sending the same request again will not help unless something about the request or the client changes.

Why a scraper or pipeline gets a 403

What to do

  1. Open the URL in an ordinary browser. If it asks you to sign in, the data is not public, and a public-page collector should leave it alone.
  2. Read the site's robots.txt and terms. A 403 for automated clients is often an explicit no.
  3. If the 403s began after a burst, cut concurrency, add delays and wait. Temporary blocks tend to lift.
  4. Identify your client honestly, with a descriptive user agent and a contact address. Do not pretend to be a browser.
  5. If you need the data, ask the owner for permission, or use an official API, feed or data export.
  6. Do not rotate IP addresses, forge headers or defeat checks to get past the block. The site has said no, and doing that can breach its terms.

A 403 is rarely worth retrying, so cap the attempts. Log it with the URL, and never record it as an empty result: a refused request is not a page with no records.

In CleanedWeb

CleanedWeb reads public pages only. A run whose source refuses access ends with stop_reason: blocked and status failed, so check the run's summary instead of expecting records.

When CleanedWeb's own API answers 403, the credential is not tied to a workspace account or lacks permission. Use a workspace key with the run scope, and check the key's scopes, the selected API's access and the workspace membership. Repeating the request does not grant access.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.