Building Your Own Tracking, Realistically
The collection is a weekend. The maintenance is indefinite, and it is what determines whether the dataset is trustworthy in year two.
Building rank tracking looks like a small project because the first version works quickly. The cost is entirely in what happens afterwards.
What the first version takes
Fetching, parsing, storing, scheduling. For a modest watch list against a single marketplace, a competent developer produces something working in a couple of days.
This is why so many organisations have one, and why so many of those datasets are unreliable.
What breaks, continuously
Page structure changes. Marketplaces redesign, and the selector that found the rank finds nothing. Sometimes it finds the wrong thing, which is worse.
Blocks and rate limits. Detection improves, and a pipeline that ran for months stops.
Identifier changes. Products relisted, merged, split.
Category restructures.
Field format changes. Prices, currencies, availability wording.
Regional differences. A parser that works on one marketplace fails on another for reasons that take a day to find.
None of this announces itself. The pipeline continues to produce rows.
The silent failure problem
The characteristic failure is not an error. It is nulls, or stale values, or a rank captured from the wrong element.
Guard against it explicitly:
Validate on write. A rank outside a plausible range, a missing price, a title that changed — reject or flag rather than store silently.
Alert on field completeness, not just on job success.
Alert on absence. A source that stops returning data produces no errors.
Compare a sample against manual observation monthly. The only check that catches a parser reading the wrong number correctly.
The maintenance commitment
Budget ongoing engineering time, not a project. Something breaks every few weeks in a multi-marketplace pipeline.
Assign an owner. An unowned pipeline decays and the decay is invisible until someone needs the data.
Document the parsers, because the person who wrote them will leave and the selectors are incomprehensible without notes.
Test against stored fixtures so a parser change can be verified before deployment.
When building is the right choice
A narrow, specific watch list that commercial coverage handles badly.
A marketplace or category with poor commercial coverage.
A need for context fields no provider captures.
Cadence requirements a provider does not offer.
Existing engineering capacity with someone to own it.
When it is not
When a provider covers your need at bearable cost. The history alone is worth more than the build.
When nobody will maintain it. An unmaintained tracker produces a dataset that is worse than none, because it is trusted.
When the marketplace is one you sell on. Use the API, and do not put a revenue-generating account at risk over data collection.
When you need history you do not have. Building today gives you data from today.
The hybrid that usually wins
API for your own products. Authoritative, permitted, reliable.
A provider for market and competitive history.
A small own-built layer for the specific gap — the handful of competitors in a niche category nobody covers, or the context fields you need and cannot buy.
Manual verification across all three.
This spreads the failure modes, keeps the critical path off your own infrastructure, and puts your engineering effort where it is actually differentiated rather than on re-solving a problem several companies solve full-time.
The honest test before starting
Who owns this in eighteen months? If the answer is a name and allocated time, build. If it is "we'll see", buy — because the version of this project that fails is not the one that never worked, it is the one that worked for a year and then quietly stopped being right.
A minimal build specification
For a team that has decided to build, this is what the first version needs and nothing more.
A scheduler, running at your chosen cadence.
A fetcher with retry, backoff, and honest client identification.
A parser per marketplace, with fixture tests.
Validation on write: range checks on rank, presence checks on required fields, title comparison against the previous observation.
Storage with the nine-field schema described elsewhere.
Four health metrics exposed somewhere a human sees them daily.
A gap log and a method version.
No dashboard in version one. The output is a table someone queries. Dashboards are built after the questions are known, and building one first tends to freeze the wrong questions into the design.