CleanedWeb

HTTP error

HTTP 404 Not Found

HTTP 404 Not Found means the server has nothing at the requested URL. It does not say whether the resource never existed, was removed, or moved. A 410 Gone says it was removed on purpose.

Why a pipeline gets a 404

The reverse also happens: a soft 404 is a page that says an item is gone but returns status 200. Check the content as well as the status.

What to do

  1. Do not retry a 404 in a tight loop. It will not change within seconds.
  2. Follow redirects (301 and 308) and store the new URL, so you stop requesting the old one.
  3. To track removals, wait for the 404 to repeat across several runs, or for a 410, before you mark a record as gone. Routing glitches happen (see data freshness).
  4. If many URLs fail at once, suspect your URL building. Compare a failing URL with a working one, and log where each URL came from.
  5. Stop a list at its real end condition, an empty page or no next cursor, not at the first 404.

Keep the two cases apart in your data: a record that is not found, and a list that has no records.

In CleanedWeb

A 404 from the CleanedWeb API means the API, endpoint or delivery was not found, or that a run was not found or is not accessible to your key. Re-read the catalog, because the API may have been unpublished. A catalog major that never existed answers 404 with catalog_version_not_found and lists the majors that do answer. A saved result can still be read after its API is unpublished or its version changes.

Tools and docs

Related terms

All glossary entries

Build an API from a website

CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.