Write the acceptance matrix before running the demo
We define the market, transaction type, asset class, filters, required fields, freshness ceiling, uniqueness rules, row limits, and acceptable partial-coverage behavior before sending a request. This prevents a visually impressive sample from quietly redefining success.
The matrix includes different failure surfaces: dense cities, sparse regions, developments with duplicate-heavy inventory, rentals, sales, nested media or contact fields, pagination limits, and the local fields the real workflow depends on.
Reconcile the response with the rows you actually saved
We compare the reported count with the records actually persisted, then check schema version, source and property identifiers, uniqueness, currencies, units, omission behavior, provenance, snapshot time, cursors, and truncation markers. A true empty result must be distinguishable from unavailable acquisition or incomplete coverage.
For duplicate-heavy inventory, we inspect both canonical properties and their underlying source offers. One flat row count cannot tell you whether the service removed duplicates correctly, lost contributing offers, or merged separate units.
Interrupt the workflow and prove it can recover
We repeat the same matrix over time and deliberately exercise limits, retries, timeouts, stale snapshots, source outages, interrupted pagination, and duplicate webhook delivery. Recovery should resume from a durable cursor or receipt, not from an operator remembering the last successful page.
Then we inspect how price and status changes, removals, reactivations, parser changes, and schema versions behave across runs. A stable endpoint is not a stable dataset if identity or semantics drift between snapshots.
Reconcile cost to accepted records, not internal requests
We require a clear billable object, a pre-request spend ceiling, idempotent retries, and a receipt that reconciles usage to accepted delivery. Internal crawling, retries, and provider architecture should not reappear as unexplained customer charges.
The evaluation ends with persisted sample rows, row-parity checks, required-field statistics, freshness distribution, known failure cases, and a signed-off scope. The last response body is supporting evidence; it is never the acceptance boundary by itself.
- Compare reported, returned, and persisted row counts.
- Test empty, partial, truncated, stale, and failed states separately.
- Replay retries and interrupted delivery with the same request identity.
- Reconcile the final usage receipt to accepted billable objects.