What Analytics Vendors' Claims Actually Mean
The tools in this space make specific claims about accuracy, coverage and history. What each one means in practice and which questions get evasive answers.
Marketplace analytics tools compete on numbers that sound like measurements. Most of them are estimates with undisclosed method.
"Accurate sales estimates"
What it means: a conversion model fitted to some sample of known sales.
What to ask: what is the sample, how large, which categories, which marketplaces, how old, and what is the error range?
What a good answer sounds like: a description of the sample, an acknowledged error range that varies by band, and a statement of which categories are well covered.
What an evasive answer sounds like: proprietary methodology, machine learning, a percentage accuracy figure with no definition of accuracy.
A single accuracy percentage is meaningless without saying accurate against what and within what tolerance.
"Complete category coverage"
What to ask: how is the product list assembled, how often is it refreshed, and what proportion of the category is missed?
Coverage is almost never complete. Long tails are large and constantly changing, and no collector enumerates all of it.
Ask for a coverage figure by category, and expect it to vary enormously.
"Historical data going back X years"
The most valuable thing a provider has, because it cannot be recreated.
What to ask: is the history continuous, what is the gap rate, was the collection method consistent, and can I export it?
Method changes create discontinuities in a long series and providers rarely flag them.
Export is the critical question. A provider whose history you cannot extract is a provider you cannot leave.
"Real-time data"
What it means in practice: collected on some cadence, which is not real time.
What to ask: what is the actual refresh interval, does it vary by product popularity, and what is the lag between observation and availability?
Deep-catalogue products are refreshed slowly everywhere, and "real time" describes the top.
"Proprietary algorithm"
A statement that the method will not be disclosed.
This is a legitimate commercial position and it means you cannot assess the estimate's reliability independently.
The mitigation is calibration: check their figures against your own known sales for your own products, across several bands. If they match well in your bands, they are probably reasonable nearby. That test is available to any seller and almost nobody runs it.
"Track unlimited products"
Check the cadence at scale. Unlimited products at weekly refresh is a different product from a hundred at hourly.
Check the API limits, which are frequently where the real constraint sits.
The questions that sort providers
Can I export everything, in a standard format, on demand, without asking?
What is the gap rate, by category, over the last quarter?
How has the collection method changed over the history you sell?
What is the error range on your unit estimates, by rank band?
Which categories and marketplaces are poorly covered?
A provider who answers all five plainly is a serious operation. One who deflects on the first is selling a dataset you will not be able to leave with.
Testing before committing
Take twenty products you have first-party data for. Compare their estimates to your actual sales, across several rank bands.
Take twenty competitor products and compare the provider against manual observation for a week.
Check the history by looking at a period you remember — a launch, a promotion, an outage — and seeing whether the data reflects it.
Request an export during the trial and confirm it works and is usable.
Four tests, a week of effort, and they distinguish a data business from a dashboard with a conversion table in it.
The trial protocol
A structured two-week trial that distinguishes providers, run identically across candidates.
Week one, coverage. Load your watch list. Record how many products they have, how far back the history goes, and the gap rate over the last quarter.
Week one, accuracy. Compare their estimates against your first-party sales for twenty of your own products across several rank bands.
Week two, freshness. Compare their daily values against your own manual observation for ten competitor products.
Week two, export. Request a full export and confirm it arrives, in a usable format, without a support conversation.
Throughout, note the answers to the five questions about method, gaps, history, error and coverage.
Score against criteria written before the trial, because otherwise the provider with the better interface wins regardless of the data.