CleanedWeb

HTTP error

Cloudflare challenge page

A Cloudflare challenge is an interstitial page that Cloudflare, a widely used CDN and security provider, shows instead of the requested page when it wants to check that a visitor is a person and not an automated client. Other security providers have similar checks.

What it looks like, and why a pipeline meets it

The site owner switched this check on, for the whole site or for some paths. Clients without a real browser cannot pass it, and heavy traffic or a poor IP reputation can trigger it. It can start without warning when the owner changes a setting.

What to do

  1. Detect it. Check the status, the cf-mitigated header, the title, and whether the fields you expect are present.
  2. Fail the run instead of parsing the page. A challenge page that is read as a record is worse than an error, because it looks like data.
  3. Back off. Hammering a challenged site only makes things worse.
  4. Look for a sanctioned route: an official API, a feed, a data export, or a partner programme. Owners can allow known automated clients, so asking is a real option.
  5. Respect the choice. Do not use CAPTCHA-solving services, browser spoofing or rotating IP addresses to get past a check that a site owner turned on.

In CleanedWeb

Challenge pages are not records. When you review a source's sample records during inspection and build, a challenge or login page must not appear as one. A site that blocks automated access cannot be turned into an API this way. A run that a site blocks ends with stop_reason: blocked and status failed, and you don't run or configure proxies or blocked-request handling. Check the run instead of expecting records.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.