Completeness, accuracy and staleness are decided by architecture, not effort. Why the comparison has a known shape, and a protocol for running it yourself.
- Architecture decides the result before any test: 5 field groups are never rendered publicly, so no scraper returns them.
- Score unreachable fields as unavailable, not 0% accurate; mixing coverage with accuracy makes a benchmark easy to dismiss.
- On shared fields the gap is derived versus counted: public reach is modelled, while an authenticated read is the platform's own number.
- Staleness is forced by cost: a sampled record is commonly months old, while an authenticated read is current at call time.
- Benchmark matched samples across all account sizes, verify against the native app, and report per field, never blended.
Comparisons between scraped and API-sourced data usually turn into an argument about effort, as though a sufficiently well built scraper would close the gap. For most fields it would not, because the gap is not about engineering.
Quality in this context breaks into 3 measurable things: whether the field is there at all, whether the value is right, and how old it is. Each one is settled by where the data comes from before anybody writes a line of code.
Here is why each is structurally decided, and a protocol you can run to measure the size of the gap for the fields you actually use.
Why is completeness decided in advance?
Because platforms render some things to the account holder and nobody else, so no amount of collection skill reaches them.
| Field group | Public source | Authenticated |
|---|---|---|
| Handle, bio, follower count | Available | Available |
| Captions, public like and comment counts | Available | Available |
| True impressions and reach | Never rendered publicly | Available where the platform exposes it |
| Audience demographics | Never rendered publicly | Available on 2 of the 4 major platforms |
| Saves and shares | Never rendered publicly | Available via insights |
| Expiring content performance | Gone in 24 hours | Capturable while live |
| Earnings | Never rendered on any public surface | YouTube only, through 1 scope |
Rows 3 to 7 are not a quality difference. They are an availability difference, and a benchmark that scores them as 0% accurate is making an argument that is easy to dismiss as unfair. Report them as unavailable and the point lands harder.
The detail per platform is in social data API coverage, and the boundary itself in authenticated versus public social data.
What decides accuracy on the fields both can reach?
Whether the number was counted or derived.
A public source showing reach is modelling it, usually from follower count and recent engagement with a platform-specific multiplier. A public source showing audience demographics is sampling visible followers and projecting the distribution. Both are legitimate techniques and both produce estimates.
An authenticated read of the same field returns what the platform recorded. There is no model to be wrong about.
That gives a benchmark 2 distinct things to measure, and they should not be combined.
- Agreement on directly observable fields. Follower count, post count, caption text. Both sources should match the native app, and where they do not, somebody has a bug.
- Divergence on derived fields. Compare the modelled value against the measured one. This is not an accuracy test of the scraper, it is a measurement of how far the model sits from reality, which is a more honest framing and a more interesting number.
The derived-field comparison degrades predictably: models tighten on large accounts and loosen on small ones, because the sample thins. The sampling arithmetic is in the audience demographics post.
Before benchmarking anything, confirm which fields are available from each source. The list is public. See the coverage list
Why is staleness forced rather than chosen?
Because keeping a large index current costs more than anyone will pay.
Refreshing 350 million profiles daily at $1 per 1,000 records runs to about $127,750,000 a year, so every index at that scale refreshes by tier: a small hot set often, the long tail rarely. On a representative tier structure that puts the median sampled record around 160 days old. The full arithmetic is in the freshness benchmark.
An authenticated read has no index behind it, so the value is current at call time and the only staleness is whatever you cache. That is architecture rather than effort, and it means a staleness comparison is really a comparison between a stored snapshot and a live fetch.
Which gives a benchmark a clean third measurement: for each field, how far is the stored value from the live one, and how old does the source say it is.
How do you run the benchmark?
On matched samples, verified against the platform rather than against each other.
# Protocol. Same accounts, same day, both sources.
# 1. Build the sample across the size distribution. Do NOT use
# a convenience sample of large accounts, where everything
# looks better.
SAMPLE = {
"nano": [...], # under 10k followers
"micro": [...], # 10k to 100k
"mid": [...], # 100k to 1m
"macro": [...], # over 1m
}
# 2. Pull every account from both sources within the same hour.
# 3. Record the ground truth by hand from the native app.
# Not from either vendor. This is the step people skip.
for account in all_accounts:
row = {
"truth": manual_from_app(account),
"public": public_source(account),
"api": authenticated_source(account),
}
# 4. Classify each field per source, 4 outcomes not 2:
# UNAVAILABLE - the source cannot return this field at all
# NULL - returned empty, honestly
# WRONG - returned a value that disagrees with truth
# CORRECT - matches truth
#
# UNAVAILABLE and WRONG are completely different findings.
# Collapsing them is what makes a benchmark look rigged.Step 3 is the one that decides whether the benchmark is worth publishing. Comparing 2 vendors against each other tells you they disagree, not which is right. Recording the truth by hand is slow and it is the only thing that turns a comparison into a measurement.
What makes a benchmark dishonest?
6 things, and most published vendor comparisons do at least 2 of them.
| Trap | Why it invalidates the result |
|---|---|
| Convenience sample of large accounts | Quality degrades on small accounts, and most creators are small |
| Scoring unavailable fields as inaccurate | Conflates coverage with quality and invites a fair objection |
| Comparing vendors against each other | Establishes disagreement, not correctness |
| Treating a zero as a value | A silent zero is often an unavailable field wearing a number's clothes |
| Blended quality score | Hides the field that is broken, the same way a blended success rate does |
| Running it once | A single run cannot distinguish a real difference from a bad day or an A/B variant |
Row 4 deserves a note because it cuts both ways. A source returning 0 for impressions is not reporting zero impressions, it is reporting that it does not have the field. A source returning null is being honest. Treating those identically penalises the honest one.
What should you report?
- Per field, per source, per size band. 3 dimensions, no averaging across any of them.
- Availability separately from accuracy. 2 tables, not 1 blended score.
- Mean absolute error on derived fields, against the measured value, with the size band attached.
- Record age where the source exposes it, and the gap from the live value where it does not.
- Your methodology in full, including the sample, so someone can disagree with your numbers by rerunning them.
That last point is what separates a benchmark from marketing. A comparison nobody can reproduce is a claim, and a protocol anyone can run is evidence.
Summary
Completeness, accuracy and staleness are each decided by where the data comes from. Some fields are never rendered publicly, so no scraper reaches them. Fields both sources reach differ by whether the value was counted or modelled. And staleness follows from refresh economics rather than from engineering choices.
So a benchmark is worth running to size the gap for your own fields, not to settle which approach is better in general. Use matched samples across the size distribution, verify against the platform rather than against a vendor, separate unavailable from wrong, and publish the protocol so the result can be checked.
Want to run this against your own fields? The coverage list and docs are open, no sales call needed. Read the docs
Is scraped data less accurate than API data?
On shared fields, both should match the platform. The bigger difference: public sources model fields like reach and audience demographics, while an authenticated read returns the recorded value.
Which fields can scraped data never provide?
True impressions and reach, audience demographics, saves and shares, expiring content performance, and earnings. None are rendered publicly, so they are unavailable rather than inaccurate.
How old is scraped data typically?
It depends on the refresh tier. Daily refresh of a 350 million profile index costs about $127,750,000 a year, so large indexes refresh the long tail rarely, leaving a median record about 160 days old.
How do you benchmark data quality fairly?
Pull the same accounts from both sources within the same hour, across the full size distribution, then record the truth by hand from the native app rather than comparing vendors against each other.
Why not just use a single quality score?
It hides the field that is broken. A source can score well overall and return nothing useful for the 3 fields your product renders. Report per field, per source and per account size band.
