Comparing Two Products Fairly
A head-to-head comparison is the most common rank analysis and the easiest to get wrong. Six conditions that have to hold before the comparison means anything.
Comparing two products by rank looks trivial and is the analysis most likely to produce a confidently wrong answer.
The six conditions
One: same category rank, or both overall. Mixing a category rank with an overall rank compares different scales. Record which you captured.
Two: same observation window. Both products measured over the same dates, with the same cadence, at the same time of day.
Three: comparable categories, if using category ranks. A category of three hundred products and one of three hundred thousand produce incomparable numbers.
Four: known availability. A product that was out of stock for part of the window has a depressed rank that says nothing about demand.
Five: sufficient window. Long enough to average out noise, which depends on velocity. Fast products need days; slow ones need weeks.
Six: price recorded. Rank without price is half the picture, particularly if one product ran a discount.
Fail any one and the comparison is provisional.
What to compare
Not average rank. Rank is a compressed non-linear scale and averaging it produces a number with no meaning. An average of rank 100 and rank 100,000 is not rank 50,050 in any useful sense.
Median rank is better than mean, and still inherits the scale problem.
Ratio of estimated units is the honest comparison, using the same conversion table for both. The errors partly cancel, which is why ratios travel better than absolutes.
Time spent above a threshold. "Product A was in the top 1,000 for eighteen of thirty days; product B for four" is defensible, easy to check and hard to misread.
Rank distribution. Plot both series and describe the distributions rather than reducing each to one number.
Presenting it
Both series on one chart, inverted log axis, same window.
A basket median line so that market-wide movement is visible.
Availability and price marked.
A stated conclusion with its basis: "Over the thirty days observed, A held a better rank than B on 26 days, and the estimated unit ratio is roughly three to one using [named table, dated]. Both products were available throughout."
That sentence is defensible. "A outsells B three to one" is not, because it omits the window, the basis and the uncertainty.
The comparisons that are not valid
Across marketplaces. Different catalogues, different sizes, different demand.
Across time periods separated by months. The catalogue grew and the competitive field changed.
Between a product and a category average. The average of a rank distribution is not a meaningful reference point.
Using different conversion tables for each product.
Where one product is a bundle or variant and the other is a single item, since rank may be shared across variants in some catalogues and separate in others. Check how the marketplace treats variants before comparing.
The variant problem
Worth stating because it silently invalidates comparisons.
Some marketplaces aggregate rank across product variants — sizes, colours, formats — and some do not. Where they aggregate, a product with twelve variants shows one rank reflecting all of them, and a single-variant competitor is being compared to twelve products.
Check the listing structure before comparing. A parent listing with many children is not comparable to a standalone item.
For books specifically, different formats frequently carry separate ranks, so comparing a paperback rank to a competitor's hardback rank compares two subsets of each title's sales.
The honest output
Most head-to-head comparisons should conclude with a direction and a rough magnitude, both qualified.
"A is consistently ahead of B, by roughly a factor of two to four in estimated units, over the observed month, with both available throughout."
That is what the data supports. Anything more precise is arithmetic performed on estimates, and the precision is manufactured at the point of multiplication.
The comparison table worth producing
A defensible head-to-head fits in a small table, and the columns are the argument.
Product, with the listing structure noted — standalone, parent with N variants, bundle.
Median rank over the window, with the range.
Days observed, and days excluded, with the reason.
Availability: proportion of the window in stock.
Price: median, and whether it changed.
Days in the top N, for a threshold that matters in that category.
Estimated units, as a range, with the basis named once beneath the table.
Seven columns. They pre-empt the questions a sceptical reader will ask, and a table containing them is much harder to misread than a single ratio in a sentence.