Storing Rank Data So It Stays Useful

Rank data is one of the cheapest datasets to store and one of the most expensive to recreate, because it cannot be recreated at all.

The schema

One row per observation, with more fields than seem necessary at the start.

Identity: marketplace, marketplace identifier, your internal product identifier.

The measurement: rank type — overall or category, with the category path — and the rank value.

Time: the timestamp of observation, in UTC, and the local time of the marketplace, because intraday patterns matter.

Context, captured at the same moment: price, currency, availability, seller, review count, average rating, title, brand.

Provenance: source, collection method, and a flag for anything unusual about the fetch.

The context fields are the ones people skip and they are what makes the dataset answer questions later. Price and availability in particular convert an ambiguous rank movement into an explicable one, and they cannot be added retrospectively.

Volume

A daily observation for a thousand products is 365,000 rows a year. Trivial.

Hourly for a thousand products is about nine million rows a year. Still small by any modern standard.

Storage is not the constraint. Query patterns and collection reliability are.

Do not aggregate away the raw observations to save space. The space is free and the aggregate cannot be un-aggregated when someone asks a question you did not anticipate.

Gaps

Store gaps as absence, not as zero and not as interpolation. A missing observation is missing.

Record failed collection attempts separately, with the reason. A rising failure rate is the earliest signal that collection is breaking.

Never fill gaps in the stored data. Interpolate at analysis time if you must, visibly, and never in the source of truth.

Retention

Keep everything. The whole value of this dataset is longitudinal, and a retention policy that discards two-year-old observations discards the only thing you cannot buy.

The exception is review text, which is personal data in most privacy regimes and carries its own retention obligations. Counts and averages do not.

Back it up off the collection machine. A dataset that exists only on the server that collects it is one failure from being gone.

Versioning the method

Collection methods change and the changes create discontinuities that look like market events.

Record the collection method version with every observation.

Log changes with dates: a new parser, a changed cadence, a switched provider, a different rank type captured.

When a series shows a step change, check the method log first. This resolves a meaningful share of apparent events and it takes seconds if the log exists.

Health monitoring

Four numbers, checked daily, catch most problems.

Observations collected against expected.

Distinct products with data today.

Field completeness for price, availability and title.

Timestamp lag between observation time and storage time.

Any of these moving is a reason to look before trusting the day's data. Collection degrades quietly — a parser that stops finding the price field returns rows with nulls and no error.

The reconciliation

Monthly, verify a sample by hand. Take twenty products, look at the marketplace, compare to what you stored.

This catches the failures that monitoring misses: the identifier that now points at a different product, the category that was restructured, the rank type that silently changed.

Twenty products takes twenty minutes and it is the only check that verifies the data means what you think it means rather than merely that rows arrived.

Starting small

An operator with no history should start today, even badly.

Fifty products, daily, with the context fields. A script and a spreadsheet will do.

In two years that is a genuine asset and it cost almost nothing. In two years it also cannot be created by anyone who did not start today, which is the entire argument.

What to do with the data you already have

Organisations frequently discover an old tracking dataset from a previous initiative. It is usually worth rescuing.

Establish what rank type it captured, which may not be documented. Sampling against a known period sometimes settles it.

Establish the cadence and the gaps.

Establish whether identifiers were stable and whether the products are the ones the labels claim.

Establish whether the method changed during its life, which usually shows as a discontinuity affecting all series at once.

If those four can be answered, the data is usable with caveats and it is history you cannot otherwise buy.

If they cannot, the data is not usable for analysis and should be kept but not relied on. A series of unknown provenance produces conclusions of unknown validity, and the temptation to use it anyway is strongest when someone urgently wants a multi-year chart.