Free tool
HTTP status and headers checker
Enter a URL to see the status code it answers, every redirect on the way with its time, and the response headers. The checker says what the answer means for a crawler or data pipeline: a 403, a 429 with its Retry-After, a 503 or a Cloudflare challenge instead of the page.
Reading the answer
2xx: the page
200 OK means the server sent the page. It does not mean the data is there: a page that builds its list with JavaScript can answer 200 with an empty list.
3xx: go elsewhere
301 and 308 are permanent moves; 302 and 307 are temporary. Each redirect is another round trip, so link to the final address.
4xx: the request was refused
404 means nothing is there; 401 needs credentials; 403 refuses this client; 429 says slow down. See the glossary for each.
5xx: the server failed
500, 502 and 503 are the server's problem, often temporary. Retry later with backoff rather than at once.
Questions
Why does my browser show the page when the checker gets a 403?
Many sites treat scripts differently from browsers: by address, user agent, missing cookies or a JavaScript check. The checker shows what an automated client gets, which is what a data pipeline would see.
Which request does the checker send?
One GET per hop with a CleanedWebSiteChecker user agent, accepting HTML, from our server. It reads at most the first 256 KB of the page. Cookies are not kept between hops, and cookie values in the answer are hidden.
Can I check a private or local address?
No. Only public websites on the standard ports can be checked, and a redirect to a private address stops the trace.
More free tools
-
Website data checker
Enter a URL to see the JSON-LD, embedded JSON, feeds, sitemap, robots.txt rules and llms.txt a crawler finds there.
-
JSON to CSV converter
Paste an API response or NDJSON; nested fields become columns and you download a spreadsheet-ready CSV.
-
JSON Schema generator
Paste sample records to get a JSON Schema, TypeScript interfaces and Pydantic models that cover every record.
-
robots.txt tester
Load or paste a robots.txt, pick a crawler and see whether each URL is allowed, with the rule and line that decides it.
-
Sitemap checker
Find a site's sitemap and check its URLs, child sitemaps, duplicates, escaping and lastmod dates against the sitemap protocol.
-
llms.txt generator
Enter a site to get a draft llms.txt from its home page, sitemap and page titles, ready to edit and publish.
-
Shopify store to CSV
Enter a Shopify store or collection to download its public products as CSV, one row per product or per variant.
-
Structured data extractor
Enter a page address to list its JSON-LD, microdata, RDFa, Open Graph and Twitter data as items you can browse, copy and download as JSON.
Need the data itself, on a schedule?
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.