Glossary
Web scraping vs API
Web scraping means fetching a website's pages and extracting data from their HTML. An API returns data as structured responses over a documented interface. The practical difference is who maintains the contract between the data and your code: with scraping you do, with an API its provider does.
When each fits
- Use an official API when the site offers one that covers the data you need, on terms you can accept. You get documented fields, authentication, published limits and a way to ask for help.
- Scrape when there is no API, or the API leaves out data the pages show, and the pages are public. You take on the upkeep.
Before you scrape, look for what the site already publishes for machines: structured data, feeds, a public API or a data export. The free website data checker lists them for a URL.
What scraping costs to keep running
Selectors break when a layout changes. Lists need pagination, detail pages need a second request, and failures need rate limiting, retries and handling for blocked requests. The worst failure is silent: a wrong value that looks plausible.
Scraping also carries obligations the code does not show. Read robots.txt and the site's terms, keep your request rate gentle, and take care with personal data. This page is not legal advice; ask a lawyer when the data or the use is sensitive.
A third option: an API built over the site
CleanedWeb builds an API over a public website. You give it a collection page, review the fields it finds, and call a REST API that returns structured records with a versioned schema. CleanedWeb does the page reading, and rebuilds a catalog API when its site changes; a contract change to an API you built waits for your review. You don't run or configure proxies or blocked-request handling, and a run that a site blocks ends with stop_reason: blocked.
It is not an official API. The site's owner does not maintain it, it depends on what the site shows, and it reads public pages only, never pages behind a login. Keep your own code for sites that need a login, form flows, or logic an API cannot express.
Tools and docs
- Website data checker See what a site already publishes: JSON-LD, feeds, Shopify and WordPress APIs.
- CleanedWeb vs. custom scrapers The product comparison, with what moves and what stays.
- Move from Scrapy or Playwright scrapers Replace one spider at a time and compare the output.
Related terms
-
Structured data
JSON-LD and schema.org markup: what sites publish for machines and where it falls short.
-
robots.txt
The file where a site tells crawlers what they may fetch, and what it does not do.
-
Cloudflare challenge page
The interstitial a site shows instead of its content. How to recognise it and respond.
-
API versioning
How APIs label changes, why you pin a version, and how major versions, deprecation and sunset work.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.