When a Change Is Real

Every rank series is noisy. The noise comes from other people's sales, from the update cadence, and from single transactions in slow categories. Without a baseline for normal variation, every wiggle looks like news.

Establishing the normal range

Observe a quiet period. Two to four weeks with no known events.

Record the distribution: the median, and the range within which most observations fall.

Note the shape. A slow product's series is a sawtooth with a wide range; a fast product's is smooth with a narrow one.

That range is your noise floor. Movements inside it are not events, however suggestive.

Do this per product. Noise characteristics vary enormously with velocity and category depth, and a threshold that works for one product is meaningless for another.

What counts as a real change

Movement well outside the established range, sustained across several observations.

Sustained is the operative word. A single reading outside the range is a sample, not a finding. Two consecutive readings, at hourly cadence, is still thin. A day of readings outside the range is something.

Direction consistent with a known cause, where one exists.

Not shared by the basket. If comparable products moved the same way at the same time, the cause is market-wide.

Velocity determines everything

Fast products — smooth series, narrow range, small movements are detectable. A 20 percent rank change is meaningful.

Slow products — sawtooth series, enormous range, only large sustained changes are detectable. A product oscillating between rank 400,000 and 700,000 on individual sales cannot support a claim about a 30 percent change.

Very slow products — in the deep long tail, rank is essentially a record of the last sale and its decay. Analysis of trend is not available at this velocity, and no technique recovers it.

Know which regime your product is in before analysing it. This single check prevents a category of wasted work.

The multiple comparisons trap

Tracking fifty products and looking for movements guarantees finding some, by chance.

With enough series, extreme values appear constantly and they are not events.

If you are scanning for anomalies, set the detection threshold with the number of series in mind, and expect false positives proportional to how many you watch.

Confirm before acting. An anomaly detected in a scan is a candidate for investigation, not a finding. Check availability, price, listing changes and the basket before concluding anything.

Confounders to rule out, in order

Before attributing a movement to demand:

Availability. Out of stock produces decay; restock produces recovery.

Price change, yours or a competitor's.

Listing change — title, images, category, variant structure.

Category restructure, which moves everyone in a category at once.

Collection problem — a gap, a changed selector, a different identifier captured.

Seasonal transition.

Basket-wide movement.

Five minutes through this list resolves the large majority of apparent events, and skipping it is how a collection bug becomes a competitor's product launch in a slide deck.

Presenting uncertainty

State the noise floor. "This product's rank normally varies between X and Y; the observed movement to Z is outside that range."

State the duration. "Sustained for six days."

State what was ruled out. "Availability confirmed throughout; the comparison basket did not move."

Use ranges rather than points for any derived estimate.

An analysis presented this way can be challenged on its substance. One presented as "rank improved 34%" invites a challenge to the whole method the first time someone notices the product moves 34 percent on a quiet Tuesday.

Setting an alert threshold

For ongoing monitoring rather than one-off analysis, an alert needs a threshold that does not fire constantly.

Base it on the product's own noise floor, not on a fixed percentage. A 30 percent move is nothing for a long-tail product and dramatic for a top-hundred one.

Require persistence. Two or three consecutive observations outside the range, rather than one.

Require basket divergence. Suppress the alert if the basket moved similarly.

Suppress during known events — your own promotions, seasonal transitions, known outages.

Tune with the number of tracked products in mind. Watching fifty products means fifty chances to fire by chance daily.

Review muted and ignored alerts monthly. An alert stream nobody reads is worse than none, because it creates a belief that anomalies would be noticed.