Use case
Structured web data for RAG, agents and LLM apps
Give a retrieval pipeline, an agent or an LLM app web data that arrives as records with named fields and a versioned schema, not as page text to parse. The same API works from your code and from an MCP client.
The problem
Page text is the usual way to get web data into a model: fetch the page, strip the markup, chunk it. That keeps the words but loses the structure. The price, the date and the author end up in the same blob as the navigation and the footer, and the model has to guess which is which.
Pipelines built that way are also hard to keep steady. When a page changes, the chunks change, and nothing tells you that what you index today differs in kind from what you indexed last week.
How CleanedWeb solves it
A website becomes an API that returns typed records. Your pipeline reads fields, so it can filter, rank and cite on them without guessing.
-
Discover the fields
CleanedWeb reads the site and proposes the fields it finds. You keep the ones your application needs, such as title, body text, author and date.
-
Build the API
It builds and validates a REST API with inputs and a limit on how much one call returns, so an agent cannot ask for an unbounded crawl.
-
Get a versioned contract
The schema and its version are published with the API. Your indexer can pin the version it was written for and notice when a contract changes, instead of embedding records of a new shape.
-
Run it on your schedule
Refresh your index from your own scheduler. Each response carries a run summary that says whether the run completed its scope, so you can skip a partial refresh.
-
Repairs when the site changes
When a site changes, CleanedWeb rebuilds the API for it, and a rebuild that would change the contract is not swapped in silently.
Agents can use the same APIs over MCP: search the catalog, read an API's fields and sample records, and run it. See the MCP and agent quickstart docs.
What a record looks like
An illustrative record from a fictional documentation site, with the fields a retrieval pipeline would index and cite.
{
"url": "https://docs.example.com/guides/rate-limits",
"title": "Rate limits",
"section": "Guides",
"author": null,
"updated_at": "2026-01-10",
"body_text": "Requests are limited per API key. When a limit is reached ...",
"tags": ["limits", "errors"]
}
Embed the body_text field and keep the rest as metadata, so retrieved passages can be filtered by date or section and cited with their URL.
Docs for this use case
- MCP clients Let an agent search the catalog, read fields and run APIs.
- Agent quickstart Have a coding agent set up the connection itself.
- Fields and schemas Read an API's fields before you index them.
- Versions and changes Pin the contract your pipeline expects.
- Source changes What happens to your API when a site changes.
Free tools that help
- Website data checker See the structured data and robots.txt rules a site exposes before you build.
- robots.txt tester Check whether a site allows GPTBot, ClaudeBot or your own crawler on the pages you need.
- llms.txt generator Draft the llms.txt that tells models which of your own pages matter.
- JSON Schema generator Generate a schema, TypeScript types or Pydantic models from sample records.
- JSON to CSV converter Inspect a batch of records as a table.
Questions
How is this different from fetching pages and extracting text?
A page fetch returns content you then have to interpret. A CleanedWeb API returns records with named, typed fields, and the schema is versioned, so your pipeline reads values instead of guessing them from text.
Can an agent call these APIs directly?
Yes. CleanedWeb provides an MCP server that lets a client search the catalog, read an API's fields and sample records, and run it. Your own APIs are available the same way.
Does it work for sites that need a login?
No. CleanedWeb reads public pages. A site that requires sign-in cannot be turned into an API this way.
How do I keep my index current?
Call the API from your own scheduler, check the run summary, and re-embed only the records that changed. CleanedWeb does not push updates to your index.
More use cases
-
Price monitoring
Track competitor and marketplace prices from an API that returns price, currency and availability as records.
-
Job data
Collect postings from job boards and career pages as records with title, employer, location and date.
-
Product catalog sync
Keep your catalog current with supplier and store sites using records you can diff and import.
Turn a site into an API
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.