CleanedWeb documentation
Turn a website into a REST API. Check the records it returns, test the API, and bring clean data into your application.
On this page
Start here#
See it work#
Every catalog API is one REST endpoint. This request asks the GitHub Topics Repositories API for two repositories tagged web-scraping:
curl --fail-with-body --max-time 120 \
"https://api.cleanedweb.com/v1/catalog/public/apis/generated_047a8ebb25911ccf62f4a212/endpoints/generated_047a8ebb25911ccf62f4a212_search/run" \
-H "Authorization: Bearer ${CLEANEDWEB_API_KEY}" \
-H "Idempotency-Key: ${CLEANEDWEB_IDEMPOTENCY_KEY}" \
-H 'Content-Type: application/json' \
--data '{"input":{"topics":"web-scraping"},"max_items":2}'
The response's records array holds one object per repository, with the same fields every time. This record, shortened, is from the API's saved sample:
{
"title": "creeper",
"owner_login": "z7r1k3",
"description": "Web Crawler and Scraper",
"typed_stargazer_count": 12,
"programming_language": "Python",
"url": "https://github.com/z7r1k3/creeper"
}
A run needs an API key with the run scope and uses your workspace's Units; your first 5,000 records are free. Run a catalog API covers the idempotency key, waiting for a longer run and reading the result.
Connect your application#
Use standard HTTP from your server with cURL, JavaScript, or Python. Start with a small collection request and inspect its records together with the execution summary.
Need a hand?#
Use troubleshooting to diagnose an issue, check source status for monitored coverage, or contact support with your source and run reference.