CleanedWeb
Markdown

CleanedWeb documentation

Turn a website into a REST API. Check the records it returns, test the API, and bring clean data into your application.

On this page

Start here#

See it work#

Every catalog API is one REST endpoint. This request asks the GitHub Topics Repositories API for two repositories tagged web-scraping:

Shell
curl --fail-with-body --max-time 120 \
  "https://api.cleanedweb.com/v1/catalog/public/apis/generated_047a8ebb25911ccf62f4a212/endpoints/generated_047a8ebb25911ccf62f4a212_search/run" \
  -H "Authorization: Bearer ${CLEANEDWEB_API_KEY}" \
  -H "Idempotency-Key: ${CLEANEDWEB_IDEMPOTENCY_KEY}" \
  -H 'Content-Type: application/json' \
  --data '{"input":{"topics":"web-scraping"},"max_items":2}'

The response's records array holds one object per repository, with the same fields every time. This record, shortened, is from the API's saved sample:

JSON · example
{
  "title": "creeper",
  "owner_login": "z7r1k3",
  "description": "Web Crawler and Scraper",
  "typed_stargazer_count": 12,
  "programming_language": "Python",
  "url": "https://github.com/z7r1k3/creeper"
}

A run needs an API key with the run scope and uses your workspace's Units; your first 5,000 records are free. Run a catalog API covers the idempotency key, waiting for a longer run and reading the result.

Connect your application#

Use standard HTTP from your server with cURL, JavaScript, or Python. Start with a small collection request and inspect its records together with the execution summary.

Need a hand?#

Use troubleshooting to diagnose an issue, check source status for monitored coverage, or contact support with your source and run reference.

Updated Get help
Search across the documentation · Esc to close