CleanedWeb

Free tool

HTTP status and headers checker

Enter a URL to see the status code it answers, every redirect on the way with its time, and the response headers. The checker says what the answer means for a crawler or data pipeline: a 403, a 429 with its Retry-After, a 503 or a Cloudflare challenge instead of the page.

One GET request from our server, following up to 8 redirects. The answer is what a script gets, not what your browser shows after it passes a check.

Reading the answer

2xx: the page

200 OK means the server sent the page. It does not mean the data is there: a page that builds its list with JavaScript can answer 200 with an empty list.

3xx: go elsewhere

301 and 308 are permanent moves; 302 and 307 are temporary. Each redirect is another round trip, so link to the final address.

4xx: the request was refused

404 means nothing is there; 401 needs credentials; 403 refuses this client; 429 says slow down. See the glossary for each.

5xx: the server failed

500, 502 and 503 are the server's problem, often temporary. Retry later with backoff rather than at once.

Questions

Why does my browser show the page when the checker gets a 403?

Many sites treat scripts differently from browsers: by address, user agent, missing cookies or a JavaScript check. The checker shows what an automated client gets, which is what a data pipeline would see.

Which request does the checker send?

One GET per hop with a CleanedWebSiteChecker user agent, accepting HTML, from our server. It reads at most the first 256 KB of the page. Cookies are not kept between hops, and cookie values in the answer are hidden.

Can I check a private or local address?

No. Only public websites on the standard ports can be checked, and a redirect to a private address stops the trace.

More free tools

All free tools

Need the data itself, on a schedule?

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.