Gaps, Nulls and Silent Data Failures
The characteristic failure in rank collection is not an error but a plausible wrong value. Four monitors that catch it, and the manual check that catches the rest.
A rank pipeline that breaks loudly is a good outcome. The damaging failures return rows that look correct.
The failure modes
Complete stop. Job fails, no rows. Detected by any monitoring.
Partial stop. One source or one marketplace stops. Invisible if you monitor total row counts.
Nulls. Rows arrive with the rank field empty, because the selector no longer matches.
Stale values. The same value repeated because a cache is being read rather than the live page.
Wrong element. The parser finds a number that is not the rank — a review count, a price, a different rank type. This is the worst case and it produces a plausible series that is simply about something else.
Identifier drift. The tracked identifier now resolves to a different product.
Rank type switch. The page structure changed and you are now capturing a category rank where you previously captured overall.
The four monitors
Row count by source, against expectation. Catches complete and partial stops.
Distinct products with data today. Catches coverage loss that row counts miss.
Field completeness for rank, price, availability, title. Catches nulls.
Value change rate. The proportion of products whose rank changed since the last observation. A sudden drop toward zero means stale values; a jump toward 100 percent means something else changed.
That fourth one is the least common and the most useful, because it catches the stale-cache failure that the others miss entirely.
Range validation on write
Reject or flag implausible values. A rank of zero, a negative number, a value beyond the catalogue size, a rank that moved by four orders of magnitude in an hour.
Flag rather than silently drop, so the anomaly is investigable.
Validate the title against the previous observation. A changed title on a tracked identifier is either a listing edit or a different product, and both need a human.
The manual reconciliation
The only check that catches a parser reading the wrong number correctly.
Twenty products, monthly. Open the marketplace, look, compare to what you stored.
Check the rank value, the rank type, the price, the availability and the title.
Twenty minutes. It has caught, in practice, every category of silent failure listed above at some point.
Automated monitoring verifies that rows arrived. Manual reconciliation verifies that they mean what you think. These are different questions and only the second one is about correctness.
Handling gaps in analysis
Never fill them in storage.
Exclude them from calculations rather than interpolating.
State the coverage in any output: "22 of 30 days observed".
Check whether gaps are random or systematic. Gaps concentrated at particular times, or on particular products, are informative about the collection rather than random loss.
A gap rate above a few percent makes fine-grained analysis unreliable and should be fixed before more analysis is built on it.
The audit trail
Store the collection method version with every row.
Log method changes with dates and descriptions.
Log incidents — blocks, outages, marketplace changes.
When a series shows a discontinuity, the log is the first place to look, and it resolves the question in seconds when it exists and takes a day of archaeology when it does not.
The principle
Assume the pipeline is broken until the monitors say otherwise, rather than assuming it works until someone complains.
The cost of that posture is four dashboards and twenty minutes a month. The cost of the alternative is a quarter of decisions made on a series that was measuring a review count.
A daily five-minute check
The routine that keeps a pipeline honest, and it fits in the time before a first coffee.
Row count by source against yesterday and against the weekly average.
Distinct products with data today.
Field completeness for rank, price and availability.
Value change rate — the proportion of products whose rank moved.
Any validation flags raised overnight.
Any gap longer than the expected interval.
Six numbers on one screen. Most mornings they are unremarkable and the check takes seconds.
The value is entirely in the mornings when they are not, because the alternative is discovering three weeks later that the price field has been null since a site update, by which time the analysis built on it has already been circulated.