Keep immutable observations before deriving events
For every accepted collection, we persist the source identity, canonical identity, observed time, comparable fields, and collection completeness. Events are comparisons between two accepted observations under a versioned rule set; we never create them by overwriting the previous row.
That history lets us replay the comparison and distinguish a source edit from a parser correction. It also prevents a newly deployed rule from silently rewriting the meaning of older events.
Name only what the observations prove
Our event envelope identifies the affected offer or property, source, previous value, new value, observation time, and rule version. Useful names include first observed, price changed, status changed, material facts changed, removed from source, and reactivated.
Removed means the offer passed a documented absence policy—not that a transaction closed. We first rule out incomplete pagination, source unavailability, acquisition failures, and parser regressions. A broken observation window is its own operational state, never a zero-inventory event.
Make every change safe to retry and replay
We give each event a stable ID and make delivery idempotent so a retry cannot apply the same price change twice. Consumers persist a cursor or receipt and can replay a bounded window after interruption without guessing where continuity broke.
Periodic exports carry a snapshot timestamp and completeness marker. Webhook envelopes are signed and duplicates are expected by contract. Exactly-once side effects come from the consumer storing event IDs, not from pretending a network delivery can never repeat.
Measure the gaps behind the event feed
We monitor collection gaps, stale records, event lag, duplicate IDs, reactivation spikes, removal spikes, and unexplained identity-cluster churn. These measures expose feeds that look active while their underlying observations are incomplete.
The most useful denominator is expected observation opportunity, not only delivered event count. A quiet market and a broken collector can both produce zero changes; continuity data is what tells them apart.
- Persist the accepted observations used for every event.
- Separate source unavailability from listing removal.
- Reconcile consumer cursors after interruption.
- Alert on unusual removal, reactivation, and cluster-churn rates.