Choosing a source
An API reads one collection: one kind of record from one list on one site. Start from the page that lists the records you want, not from the site's homepage.
On this page
Start from the list page#
Say you want public GitHub repositories about web scraping. https://github.com holds repositories, people, issues and product pages. https://github.com/topics/web-scraping lists only the repositories tagged with that topic, one per entry. Paste the topic page. A search results page or a category page works the same way.
That is how the catalog's GitHub Topics Repositories API reads GitHub: one topic page at a time, one record per repository.
Open the list the way you want it before you copy the address. Controls the page offers, such as a language filter or a sort order, become inputs of the API: the GitHub API takes a programming_language and a sort (most stars, recently updated and so on) because the topic page has those controls.
Check what inspection found#
After you paste the URL, Source inspection reads the page. If the site has more than one list, pick yours under Choose a collection to inspect. Then read the sample records before you create anything:
- One row per record. Each row should be one repository, not a navigation link, a promoted box or an ad.
- The fields you need. Every GitHub sample has the repository name, description, topics and link.
- Fields from each record's own page. GitHub shows owner, stars, forks and language on each repository's page, not in the topic list. The API reads those pages too, so those fields can be missing from a record whose page was not read. Fields and schemas shows what that looks like.
When to try a different URL#
- Inspection picked the wrong list. Paste a more specific page, such as one category or one search.
- The page needs a login. CleanedWeb reads public pages only, so it cannot build an API for it.
- The samples are empty or are only links. Try the same list from another page, such as a search result instead of a category. If that fails too, send the URL to support.
Once the samples look right, continue with fields and schemas to check the record your code will read.