Store enough context to reproduce the source claim
We preserve the source, source URL, loaded URL, external ID, collection time, and canonical source URL when one exists. We also record the acquisition or parser version whenever it can change how the source content becomes a field.
The observation envelope travels with the offer or remains reachable through a durable reference. A URL in a support note fails as provenance: pages change, identifiers get recycled, and the exact value seen at collection time disappears.
Record the input and rule behind every derived value
For unit conversion, currency handling, enum mapping, geocoding, and calculated fields, we retain the input, output, rule name, and rule version. A derived price per area points to the exact compatible price and area observations used to calculate it.
When several offers contribute to a preferred field, we keep the competing assertions and the selection rule, including observation time and confidence where relevant. The preferred value is a view over evidence, not a replacement for it.
Design lineage around the questions incidents produce
During an incident, we need to answer: which source asserted this price, when did we observe it, what parser produced it, did another source disagree, was the area converted, and which match attached the offer to this property? Each answer should come from structured fields, not a manual reconstruction from logs.
This model also supports targeted repair. If one enum rule was wrong, we can find the affected fields and recompute them without recollecting unrelated sources or rewriting the original observations.
Follow sampled fields all the way back to evidence
We sample delivered records and trace high-impact fields—price, typed area, status, contact, and preferred source—back to their observations. We then repeat the trace after a parser migration, source removal, property split, and rematch.
Provenance completeness is measurable: required delivered fields with a valid observation path divided by required delivered fields sampled. We report that alongside field completeness because a populated field without evidence is a different kind of missing data.
- Trace values after parser and normalization-rule changes.
- Confirm a property split does not orphan offer evidence.
- Keep source disagreement queryable after choosing a preferred value.
- Measure lineage completeness separately from field completeness.