CleanedWeb

Free tool

Website data checker

Enter a page address to see the machine-readable data a crawler finds there: JSON-LD and microdata, JSON embedded in the HTML, linked feeds, open Shopify and WordPress endpoints, the sitemap, robots.txt rules and llms.txt.

Checks one page plus the site's robots.txt, sitemap and llms.txt. Public websites only.

What the checker looks at

Structured data on the page

JSON-LD blocks and schema.org microdata, counted by type (Product, JobPosting, Event, BreadcrumbList and so on), plus Open Graph tags.

JSON in the HTML

Framework state such as __NEXT_DATA__ or __NUXT__. When a page ships its data this way, the records are usually there before any JavaScript runs.

Feeds and platform endpoints

RSS, Atom and JSON feeds linked from the page. On Shopify stores it tries /products.json; on WordPress sites, /wp-json/.

Crawl rules

Whether robots.txt allows this page for all crawlers, any crawl delay, which AI crawlers it blocks, how many URLs the sitemap lists, and whether the site publishes llms.txt.

Questions

Does a good result mean the site is easy to scrape?

It means the site already publishes some of its data in a form machines can read. Pagination, search forms, sign-in walls and bot protection still decide whether a full collection works, and the checker does not test those.

Why does the checker see less than my browser?

It reads the HTML the server returns, without running JavaScript. Pages that build their content in the browser show little text here; that is also what most simple crawlers see.

Which user agent does it send?

CleanedWebSiteChecker/1.0, with a link back to this page. It reads at most six addresses per check and keeps each result for ten minutes.

Is there a limit?

A few checks per minute from each visitor. For recurring collection from a site, build an API instead.

More free tools

All free tools

Need the data itself, on a schedule?

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.