CleanedWeb

Glossary

Web scraping vs API

Web scraping means fetching a website's pages and extracting data from their HTML. An API returns data as structured responses over a documented interface. The practical difference is who maintains the contract between the data and your code: with scraping you do, with an API its provider does.

When each fits

Before you scrape, look for what the site already publishes for machines: structured data, feeds, a public API or a data export. The free website data checker lists them for a URL.

What scraping costs to keep running

Selectors break when a layout changes. Lists need pagination, detail pages need a second request, and failures need rate limiting, retries and handling for blocked requests. The worst failure is silent: a wrong value that looks plausible.

Scraping also carries obligations the code does not show. Read robots.txt and the site's terms, keep your request rate gentle, and take care with personal data. This page is not legal advice; ask a lawyer when the data or the use is sensitive.

A third option: an API built over the site

CleanedWeb builds an API over a public website. You give it a collection page, review the fields it finds, and call a REST API that returns structured records with a versioned schema. CleanedWeb does the page reading, and rebuilds a catalog API when its site changes; a contract change to an API you built waits for your review. You don't run or configure proxies or blocked-request handling, and a run that a site blocks ends with stop_reason: blocked.

It is not an official API. The site's owner does not maintain it, it depends on what the site shows, and it reads public pages only, never pages behind a login. Keep your own code for sites that need a login, form flows, or logic an API cannot express.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.