CleanedWeb

Use case

Structured web data for RAG, agents and LLM apps

Give a retrieval pipeline, an agent or an LLM app web data that arrives as records with named fields and a versioned schema, not as page text to parse. The same API works from your code and from an MCP client.

The problem

Page text is the usual way to get web data into a model: fetch the page, strip the markup, chunk it. That keeps the words but loses the structure. The price, the date and the author end up in the same blob as the navigation and the footer, and the model has to guess which is which.

Pipelines built that way are also hard to keep steady. When a page changes, the chunks change, and nothing tells you that what you index today differs in kind from what you indexed last week.

How CleanedWeb solves it

A website becomes an API that returns typed records. Your pipeline reads fields, so it can filter, rank and cite on them without guessing.

  1. Discover the fields

    CleanedWeb reads the site and proposes the fields it finds. You keep the ones your application needs, such as title, body text, author and date.

  2. Build the API

    It builds and validates a REST API with inputs and a limit on how much one call returns, so an agent cannot ask for an unbounded crawl.

  3. Get a versioned contract

    The schema and its version are published with the API. Your indexer can pin the version it was written for and notice when a contract changes, instead of embedding records of a new shape.

  4. Run it on your schedule

    Refresh your index from your own scheduler. Each response carries a run summary that says whether the run completed its scope, so you can skip a partial refresh.

  5. Repairs when the site changes

    When a site changes, CleanedWeb rebuilds the API for it, and a rebuild that would change the contract is not swapped in silently.

Agents can use the same APIs over MCP: search the catalog, read an API's fields and sample records, and run it. See the MCP and agent quickstart docs.

What a record looks like

An illustrative record from a fictional documentation site, with the fields a retrieval pipeline would index and cite.

{
  "url": "https://docs.example.com/guides/rate-limits",
  "title": "Rate limits",
  "section": "Guides",
  "author": null,
  "updated_at": "2026-01-10",
  "body_text": "Requests are limited per API key. When a limit is reached ...",
  "tags": ["limits", "errors"]
}

Embed the body_text field and keep the rest as metadata, so retrieved passages can be filtered by date or section and cited with their URL.

Docs for this use case

Free tools that help

Questions

How is this different from fetching pages and extracting text?

A page fetch returns content you then have to interpret. A CleanedWeb API returns records with named, typed fields, and the schema is versioned, so your pipeline reads values instead of guessing them from text.

Can an agent call these APIs directly?

Yes. CleanedWeb provides an MCP server that lets a client search the catalog, read an API's fields and sample records, and run it. Your own APIs are available the same way.

Does it work for sites that need a login?

No. CleanedWeb reads public pages. A site that requires sign-in cannot be turned into an API this way.

How do I keep my index current?

Call the API from your own scheduler, check the run summary, and re-embed only the records that changed. CleanedWeb does not push updates to your index.

More use cases

All use cases

Turn a site into an API

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.