Free tool
Website data checker
Enter a page address to see the machine-readable data a crawler finds there: JSON-LD and microdata, JSON embedded in the HTML, linked feeds, open Shopify and WordPress endpoints, the sitemap, robots.txt rules and llms.txt.
What the checker looks at
Structured data on the page
JSON-LD blocks and schema.org microdata, counted by type (Product, JobPosting, Event, BreadcrumbList and so on), plus Open Graph tags.
JSON in the HTML
Framework state such as __NEXT_DATA__ or __NUXT__. When a page ships its data this way, the records are usually there before any JavaScript runs.
Feeds and platform endpoints
RSS, Atom and JSON feeds linked from the page. On Shopify stores it tries /products.json; on WordPress sites, /wp-json/.
Crawl rules
Whether robots.txt allows this page for all crawlers, any crawl delay, which AI crawlers it blocks, how many URLs the sitemap lists, and whether the site publishes llms.txt.
Questions
Does a good result mean the site is easy to scrape?
It means the site already publishes some of its data in a form machines can read. Pagination, search forms, sign-in walls and bot protection still decide whether a full collection works, and the checker does not test those.
Why does the checker see less than my browser?
It reads the HTML the server returns, without running JavaScript. Pages that build their content in the browser show little text here; that is also what most simple crawlers see.
Which user agent does it send?
CleanedWebSiteChecker/1.0, with a link back to this page. It reads at most six addresses per check and keeps each result for ten minutes.
Is there a limit?
A few checks per minute from each visitor. For recurring collection from a site, build an API instead.
More free tools
-
JSON to CSV converter
Paste an API response or NDJSON; nested fields become columns and you download a spreadsheet-ready CSV.
-
JSON Schema generator
Paste sample records to get a JSON Schema, TypeScript interfaces and Pydantic models that cover every record.
-
robots.txt tester
Load or paste a robots.txt, pick a crawler and see whether each URL is allowed, with the rule and line that decides it.
-
Sitemap checker
Find a site's sitemap and check its URLs, child sitemaps, duplicates, escaping and lastmod dates against the sitemap protocol.
-
llms.txt generator
Enter a site to get a draft llms.txt from its home page, sitemap and page titles, ready to edit and publish.
-
Shopify store to CSV
Enter a Shopify store or collection to download its public products as CSV, one row per product or per variant.
Need the data itself, on a schedule?
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.