Why Scrapers Break: A Failure Rate Analysis

Everyone knows scrapers break when a site changes its markup, and the retrieval layer is where most teams expect trouble. That is the failure people plan for, and it is the easiest one, because it throws an exception and somebody fixes it the same day.

The failures that actually cost money are the ones that return a 200 and a plausible-looking value. Those run for weeks, quietly filling your database with wrong data, and the first sign is usually a customer asking why a number looks odd.

Here is a taxonomy of how scrapers break, how often, and what each kind costs before you notice. The cost model that sits on top of this is in the API versus scraping comparison.

What are the 3 failure classes?

Sorted by how quickly you find out, which is the opposite of how much they cost.

The inversion in that table is the point. Loud failures are the ones your monitoring is built for and the ones that cost least. Silent failures are invisible to exception tracking and cost the most, because by the time you find one you have to decide what to do with a few weeks of records you can no longer trust.

What actually causes breakage?

Rows 4 to 7 are where the real damage is. The scraper runs, the selector matches, the value parses, and the data is wrong. Nothing in a conventional monitoring setup notices, because nothing failed.

The format change row is worth dwelling on. A follower count rendered as 1.2M parses to 1.2 if your code expects an integer, so a creator with 1.2 million followers quietly becomes a creator with 1 follower. That value is in range, has the right type, and is wrong by 6 orders of magnitude. It is the same class of trap as the string versus integer problem in API responses.

How often does this happen?

Using the one observation with a real number behind it: a team tracking 14 marketplaces reported 9 structure changes in a single quarter. That works out to about 2.6 per source per year.

At 20 sources you are fixing something weekly. At 50 you are fixing something twice a week, which is most of a role. This is the arithmetic behind labour being 64 to 71% of build cost.

And this counts structure changes only. Anti-bot responses, consent walls and A/B tests arrive on top, and they arrive more often.

If the fields you need are available through an API, none of this applies. The per-platform list is public. See the coverage list

What does a silent failure cost?

It depends entirely on how often you check, because a silent failure runs until a human looks, unlike an announced API change.

150,000 contaminated records from 1 undetected break, on 1 source, with a monthly check. Now put that next to a breakage every 7 days across 20 sources and the picture gets uncomfortable.

The follow-on question is worse than the detection one: what do you do with the bad records? If you cannot tell which ones came from the broken period, you either re-collect everything since the last known-good date or you keep data you know is wrong. Most teams quietly choose the second.

Why do A/B tests look like flakiness?

Because they are intermittent by design, and intermittent failures get dismissed.

If a site serves a new layout to 10% of requests, your scraper succeeds 90% of the time. Spot check it 3 times and all 3 pass with probability 0.73, so the likely conclusion is that it works and something odd happened once.

Meanwhile 10% of your records are wrong, continuously, until the experiment concludes and the variant either disappears or becomes the default. If it becomes the default, your failure rate jumps from 10% to 100% overnight and you finally notice.

The lesson is that an intermittent failure is a failure, not noise. If a check fails once in ten, that is a signal about a tenth of your data.

How do you detect the silent ones?

By watching distributions rather than exceptions. Nothing throws, so you have to notice that the shape of the data changed.

# Exception monitoring catches loud failures only.
# These 4 checks catch the expensive ones.

# 1. Field fill rate. A selector that stopped matching returns
#    nulls, not errors.
assert fill_rate("follower_count") > 0.95

# 2. Value distribution. Compare today against a trailing window.
#    A format change moves the whole distribution at once.
assert median_today / median_trailing_30d < 10

# 3. Record count. Pagination breaking usually halves the volume
#    rather than zeroing it.
assert abs(count_today - mean_count_7d) / mean_count_7d < 0.25

# 4. Known-value canaries. A handful of records you verify by hand
#    and re-check every run. The cheapest high-signal check there is.
for canary in CANARIES:
    assert fetch(canary.url)[canary.field] == canary.expected

# Canaries catch semantic breakage that no statistical check will,
# because you know what the answer should be.

The canaries are the highest-value item on that list and the one most often skipped. Pick 10 records per source whose correct values you verified by hand, re-check them every run, and alert on any mismatch. It takes an afternoon to set up and it catches the failures that statistics cannot.

One caveat on check 2. A real event can move a distribution, so a threshold alert needs a human to confirm rather than an automatic rollback. The aim is to notice within a day rather than within a quarter.

Summary

Scrapers break about 2.6 times per source per year on structure alone, which is weekly at 20 sources, and the breakages everyone plans for are the cheap ones. The expensive failures return a 200 with a plausible wrong value and run until somebody checks.

So monitor distributions and fill rates rather than exceptions, keep hand-verified canaries on every source, and treat an intermittent failure as a real one rather than as noise. And decide in advance what you will do with the contaminated records, because that decision is harder than the fix.

Want data that fails loudly with a documented error instead of quietly with a plausible one? Get a demo