HTTP error
HTTP 404 Not Found
HTTP 404 Not Found means the server has nothing at the requested URL. It does not say whether the resource never existed, was removed, or moved. A 410 Gone says it was removed on purpose.
Why a pipeline gets a 404
- Your URL is wrong: a typo, a trailing slash, bad encoding, or a slug that changed.
- The item was removed, which is normal for sold-out products and expired job postings.
- The site restructured its URLs and did not redirect the old ones.
- You asked for a page past the end of a list.
- A sitemap or an older crawl handed you a stale URL.
- Some sites answer 404 for a private resource instead of 403.
The reverse also happens: a soft 404 is a page that says an item is gone but returns status 200. Check the content as well as the status.
What to do
- Do not retry a 404 in a tight loop. It will not change within seconds.
- Follow redirects (301 and 308) and store the new URL, so you stop requesting the old one.
- To track removals, wait for the 404 to repeat across several runs, or for a 410, before you mark a record as gone. Routing glitches happen (see data freshness).
- If many URLs fail at once, suspect your URL building. Compare a failing URL with a working one, and log where each URL came from.
- Stop a list at its real end condition, an empty page or no next cursor, not at the first 404.
Keep the two cases apart in your data: a record that is not found, and a list that has no records.
In CleanedWeb
A 404 from the CleanedWeb API means the API, endpoint or delivery was not found, or that a run was not found or is not accessible to your key. Re-read the catalog, because the API may have been unpublished. A catalog major that never existed answers 404 with catalog_version_not_found and lists the majors that do answer. A saved result can still be read after its API is unpublished or its version changes.
Tools and docs
- HTTP status checker See the status code, redirects and headers a URL answers a script with.
- API catalog Find the APIs that are published now.
- Catalog API versions What the unknown-version answer lists, and what happens after a sunset.
- Sitemap checker Inspect the URLs a site lists, with duplicates and lastmod dates.
Related terms
-
Sitemap
The XML file that lists a site's URLs, and how to use it to find pages and spot changes.
-
Data freshness
How current your collected data is, how to choose a refresh interval and detect changes.
-
Pagination
How lists are split into pages and how to collect every record without gaps or repeats.
-
HTTP 403 Forbidden
The server understood the request and refuses it. Why it happens and how to respond.
Build an API from a website
CleanedWeb turns a website into a maintained REST API that returns structured records, with a versioned schema and repairs when the site changes.