A Checklist for a Defensible Analysis

Most rank analysis fails on one of a small number of points. Running through them takes twenty minutes and prevents the failure that costs credibility.

Data integrity

Was the pipeline healthy over the period? Row counts, distinct products, field completeness, value change rate.

What was the gap rate? State it. Above a few percent, fine-grained conclusions are unreliable.

Did the collection method change? Check the log before interpreting any discontinuity.

Do the tracked identifiers still resolve to the intended products? Titles unchanged, categories unchanged.

Was a manual reconciliation done recently? Automated monitoring confirms rows arrived, not that they are correct.

Method

Is the rank type consistent across everything being compared — all overall, or all the same category?

Was availability recorded, and are outages excluded rather than averaged through?

Was price recorded, and did it change during the period?

Is there a control basket, and did it move?

Is the observation window adequate for the velocity? Split-half test: do the two halves of the period agree?

Is the noise floor established for each product, so that movements can be judged against normal variation?

Does the period span a seasonal boundary? If so, is it handled?

Interpretation

Have the mundane explanations been ruled out in order — data, availability, price, basket, seasonality — before any interesting one?

Is any causal claim supported by an intervention or a natural experiment, or is it association?

If a sample was selected on the outcome, is there a comparison group?

Presentation

Are estimates presented as ranges, in the cell rather than in a footnote?

Is the conversion basis named and dated?

Is the scope stated: products, marketplace, period, cadence, rank type?

Is the chart inverted and logarithmic, with gaps shown and the basket plotted?

Is the conclusion phrased to survive being quoted onward without its caveats?

The one-paragraph summary test

Before sending, write the conclusion as a single paragraph containing the finding, the basis, the period, what was ruled out and the uncertainty.

If that paragraph is uncomfortable to write, the analysis is not ready.

If it is comfortable, it is also the paragraph to lead with, because it is what a reader needs and it is what protects the work.

The checks people skip

In order of how often they are skipped and how much damage that causes:

The control basket. Skipped most; causes the most wrong conclusions.

Availability. Skipped often; produces the most convincing wrong conclusions.

The noise floor. Rarely established; makes every wiggle a finding.

Manual reconciliation. Almost never done; the only defence against a parser reading the wrong number.

The split-half test. Almost never done; free, and it settles the window question immediately.

Adding those five to a routine analysis costs perhaps an hour and moves the work from plausible to defensible.

The peer read

The cheapest quality control available, and it is a fifteen-minute conversation rather than a review process.

Give the one-paragraph summary to a colleague who did not do the work.

Ask them what they would challenge.

Ask them what they think the number means, and see whether it matches what you intended.

Ask them what they would do differently as a result of reading it.

If their answer to the third question is wrong, the presentation is at fault, not the reader.

If their answer to the fourth is nothing, the analysis did not need doing, which is worth knowing before it is circulated.

Do this on anything consequential. It catches the misreadings that the author cannot see, because the author knows what they meant.