How Long to Observe
The window has to be long enough to average out noise and short enough that the market has not changed underneath it. Both bounds are real and they conflict.
Every rank analysis has to choose an observation window, and the choice is a trade-off between two failures.
The lower bound: noise
Too short and variation dominates. A three-day window on a slow product captures two sales and a decay, which supports no conclusion.
The requirement scales inversely with velocity. A top-hundred product's rank is stable enough to characterise in days. A long-tail product needs weeks, and may never be characterisable.
A practical test: split your window in half and compute the summary for each half. If the two halves disagree substantially, the window is too short.
That test is cheap, it uses data you already have, and almost nobody runs it.
The upper bound: drift
Too long and the market changes. New competitors enter, prices move, the season turns, the catalogue grows.
A six-month window blends several different competitive situations into one average that describes none of them.
Seasonal boundaries are the hard limit. A window spanning a peak and a trough is measuring the calendar.
Reasonable defaults
Fast-moving products, competitive tracking: two to four weeks.
Mid-velocity products: four to eight weeks.
Slow products: eight to twelve weeks, with the caveat that the conclusion is coarse.
Campaign measurement: a baseline period of at least a week before, the campaign period, and a post-period of at least a week after equilibrium is reached — which given decay means starting the post-period several days after the campaign ends.
Launch assessment: the first weeks are not representative. New products receive visibility that fades, and a launch rank is not a steady-state rank.
Matching windows across comparisons
The compared series must use the same window. Comparing a four-week average for one product against a twelve-week average for another compares different things.
Where availability differs, either restrict both to the period when both were available, or state the difference.
Where one product launched mid-window, exclude the pre-launch period rather than treating absence as poor performance.
Rolling windows
For ongoing monitoring rather than a one-off analysis:
A rolling median over a fixed window smooths noise while remaining current.
Choose the window from the velocity, as above.
Plot the raw series underneath it, because the rolling line hides events and the events are frequently the point.
Recompute the noise floor periodically. A product's velocity changes and the appropriate window changes with it.
When the window cannot be long enough
Some questions cannot be answered with rank data because the required window exceeds the period over which the market is stable.
Long-tail products with a decision that needs answering this month. The noise floor is wider than the effect you are looking for.
Fast-changing categories where four weeks contains three competitor launches.
Post-launch assessment of a product still receiving launch visibility.
In these cases the honest output is that the data does not support the conclusion, with an explanation of why, and a suggestion of what would — usually first-party sales data, a longer wait, or a different measure entirely.
Producing a confident answer from an inadequate window is the most common way rank analysis loses credibility, because the answer will be contradicted by the next month's data and the audience will remember.
Running the split-half test
The cheapest validity check available and almost nobody runs it.
Take your observation window and divide it in two.
Compute your summary statistic separately for each half — median rank, estimated units, whatever the analysis produces.
Compare. If the two halves give substantially different answers, the window is too short, or something changed mid-window and the average describes neither state.
Investigate which. A trend across the halves is a real change; random disagreement is noise.
Report the result. "The two halves of the window agree within X" is a sentence that materially strengthens an analysis and costs one line of code.
Where the halves disagree and you cannot extend the window, the honest output is a description of both states rather than an average of them.