HTTP error
Cloudflare challenge page
A Cloudflare challenge is an interstitial page that Cloudflare, a widely used CDN and security provider, shows instead of the requested page when it wants to check that a visitor is a person and not an automated client. Other security providers have similar checks.
What it looks like, and why a pipeline meets it
- A short HTML page, often titled "Just a moment...", that runs a script or shows a widget instead of the content you asked for.
- A status of 403 or 503, and sometimes 200. Cloudflare's challenge responses commonly carry a
cf-mitigated: challengeheader. - None of the fields you expect, and none of the page's real content.
The site owner switched this check on, for the whole site or for some paths. Clients without a real browser cannot pass it, and heavy traffic or a poor IP reputation can trigger it. It can start without warning when the owner changes a setting.
What to do
- Detect it. Check the status, the
cf-mitigatedheader, the title, and whether the fields you expect are present. - Fail the run instead of parsing the page. A challenge page that is read as a record is worse than an error, because it looks like data.
- Back off. Hammering a challenged site only makes things worse.
- Look for a sanctioned route: an official API, a feed, a data export, or a partner programme. Owners can allow known automated clients, so asking is a real option.
- Respect the choice. Do not use CAPTCHA-solving services, browser spoofing or rotating IP addresses to get past a check that a site owner turned on.
In CleanedWeb
Challenge pages are not records. When you review a source's sample records during inspection and build, a challenge or login page must not appear as one. A site that blocks automated access cannot be turned into an API this way. A run that a site blocks ends with stop_reason: blocked and status failed, and you don't run or configure proxies or blocked-request handling. Check the run instead of expecting records.
Tools and docs
- HTTP status checker See the status code, redirects and headers a URL answers a script with.
- Website data checker See what a site publishes for machines before you decide how to get the data.
- Choosing a source Check the sample records, and try a different public page when they are empty.
- Troubleshooting What to save and check when a run does not return usable records.
Related terms
-
HTTP 403 Forbidden
The server understood the request and refuses it. Why it happens and how to respond.
-
HTTP 503 Service Unavailable
The server cannot answer right now. How to retry with backoff and not read it as no data.
-
robots.txt
The file where a site tells crawlers what they may fetch, and what it does not do.
-
Web scraping vs API
Scraping pages versus calling an API: when each fits and what scraping costs to maintain.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.