CleanedWeb
Markdown

Fields and schemas

Every API has a schema: the fields a record can have, the type of each, and which ones every record has. Read it before your code relies on a field.

On this page

Two real records#

These are saved samples from the catalog's GitHub Topics Repositories API, shortened to a few fields and topics:

JSON · example
{
  "title": "creeper",
  "owner_login": "z7r1k3",
  "description": "Web Crawler and Scraper",
  "stargazer_count": "12",
  "typed_stargazer_count": 12,
  "forks_count": "1",
  "programming_language": "Python",
  "topics": ["python", "crawler", "scraper", "web-crawler"],
  "url": "https://github.com/z7r1k3/creeper"
}
JSON · example
{
  "title": "web-scraping-google-sheets",
  "owner_login": null,
  "description": "Guide to Using Google Sheets for Basic Web Scraping",
  "stargazer_count": null,
  "typed_stargazer_count": null,
  "forks_count": null,
  "programming_language": null,
  "topics": ["web-scraping", "web-crawler-python", "scraper-api"],
  "url": "https://github.com/oxylabs/web-scraping-google-sheets"
}

Both records have the same field names. The second has no owner, stars, forks or language. The next two sections explain why, and what your code should do about it.

Read the types#

The schema gives each field's type. Part of this API's schema:

JSON · example
{
  "stargazer_count": { "type": "string" },
  "typed_stargazer_count": { "type": "integer", "minimum": 0 },
  "forks_count": { "type": "string" },
  "topics": { "type": "array", "items": { "type": "string" } },
  "url": { "type": "string", "format": "uri" }
}

stargazer_count and forks_count keep the counts as the page shows them, as text such as "12". typed_stargazer_count and typed_forks_count hold the same counts as integers (12), so use those for sorting and arithmetic. Not every API has typed copies; where a count is only text, convert it in your code and handle a value that is not a plain number.

Plan for missing fields#

The schema also lists the fields every record has. Here that is "required": ["title", "url"]; every other field can be null.

Owner, stars, forks and language come from each repository's own page. With enrich_records set to true (the default), the API reads those pages within the run's time budget, and a record whose page was not read comes back without them, like the second record above. The run's summary.record_enrichment says how many records were enriched and why any were not.

So treat a null as unknown: a missing star count is not zero stars. If a field is essential to you, drop or re-run the records that lack it, rather than filling in a default.

Where to find the schema#

  • A catalog API: its page in the catalog lists every field with its type under What you get, and links the full schema as contract.json.
  • An API you built: its page in your workspace shows the saved schema, sample records and definition version.

Pin that definition version in your code so a change to the schema fails loudly instead of changing your data; see versions and changes. The quickstart shows a first request.

Updated Get help
Search across the documentation · Esc to close