What Is Web Data Infrastructure? Definition, Components, and the Layer Most Definitions Leave Out

Web data infrastructure is the stack that sits between a data source and usable structured output: getting a request to the target, receiving a usable response, turning that response into fields, running the whole thing reliably at scale, and governing what you collected.

That much is uncontroversial. What is worth noticing is that every published definition of this category was written by a company selling one layer of it, and each definition ends at the boundary of what that company sells. Proxy vendors define it as an access problem. Scraping platforms define it as a rendering and extraction problem. Both are correct about their layer and both stop at the same place: the public web.

There is a fifth layer past that boundary, and if the data you need lives behind an account holder's login, it is the only layer that matters. Below: each layer, what it costs, where the build-versus-buy line falls, and how to tell which ones you actually need.

Why does every definition stop in the same place?

Because the category was named by its vendors. Look at three current definitions side by side and the pattern is hard to miss.

Source type	How they define the category	Where it stops
Residential proxy vendor	A collection, processing and delivery stack whose foundation is the access layer. Explicit advice: build your scrapers and extraction, rent the IPs	Proxies. Everything above is your problem
Scraping API platform	A single API providing headless Chrome rendering, rotating proxies, concurrency and AI extraction	Rendering and extraction
Crawling platform	The layer between the public web and the products built on it, with a six-stage path: receive, route, render, extract, store, deliver	Explicitly, the public web

That third definition is admirably honest and it names the boundary in its own words: the layer between the public web and the products built on it. Which is the right description of what those platforms do, and an incomplete description of web data infrastructure, because a great deal of the data products actually need is not on the public web at all.

None of this is deceptive. A proxy company writing a category definition that ends at proxies is doing normal marketing. It just means the map you find when you search this term has an edge, and the edge is not the edge of the territory.

What are the five layers?

Here is the full stack, bottom to top, with the governance concern that has to run through all of it.

Layer	What it does	Typical products	Bills by
1. Network access	Determines where your request appears to originate	Residential, ISP, mobile and datacenter proxies	Gigabyte
2. Retrieval	Gets a usable response back	Unblockers, headless browsers, CAPTCHA handling, fingerprint management	Request or record
3. Extraction	Turns a response into structured fields	Parsers, scraping APIs, LLM-based extraction	Record or compute unit
4. Orchestration	Runs it reliably: scheduling, retries, rate limits, queues, storage, delivery, monitoring	Crawling platforms, workflow engines, your own code	Record, compute, or salary
5. Authenticated access	Returns data the public web never renders, on the account holder's authorisation	Consented APIs, OAuth data layers	Connected account
Governance	Cross-cutting: provenance, lawful basis, retention, deletion, audit trail	Policy, plus whatever your vendors expose	Not billed. Costs you anyway

Layers one to four answer one question: how do I reliably get what the public web will show me? Layer five answers a different one: how do I get what it will not? They are complementary rather than competing, and a team that needs both and buys only the first four will discover the gap late.

Layer 1: network access

The job is to make your request appear to originate from an acceptable place. That is the whole product. It changes the network path, not what the target is willing to serve.

Four types, chosen by workload rather than by target: rotating residential for broad geographic collection, ISP or static residential for stable long sessions, mobile where cellular network characteristics matter, and datacenter for tolerant targets and internal tooling. Pricing runs from roughly $2.50 per gigabyte at the residential end, which makes bandwidth-heavy targets expensive fast.

Almost nobody should build this layer. Maintaining a clean global IP pool is a full-time infrastructure business, and even the proxy vendors say so when advising on the rest of the stack. We went through the selection criteria and the legal boundary in social media proxies, and when an API is the better buy.

Layer 2: retrieval

The job is turning a request into a usable response. This is where most of the visible engineering effort in scraping goes, and where the arms race lives: anti-bot detection, CAPTCHA handling, browser fingerprinting, JavaScript rendering, session management.

The metric that matters here is success rate per target, not overall. An independent benchmark of eleven scraping APIs across fifteen protected sites found only four exceeded 80 percent success against a 58.2 percent industry average, and one provider averaged 63.7 percent overall while returning 0 percent on two specific major platforms. A blended average conceals exactly the failure you care about.

Buy this layer unless retrieval is your product. The maintenance is continuous and the failure mode is silent: a 200 response containing a consent wall or a challenge page counts as a success in your metrics and as garbage in your database.

Layer 3: extraction

The job is turning a response into fields. Historically this meant hand-written selectors per site, which broke every time a layout changed. LLM-based extraction reads pages by meaning instead, which makes the layer more resilient and less per-site to maintain.

This is the layer teams over-build. It feels like the interesting problem and it is the one most amenable to being solved once. Before writing a custom extractor, check whether a scraping API already returns the fields structured, because at roughly $0.75 to $1.00 per thousand records the buy option is frequently cheaper than the engineer maintaining the build option.

One caution on AI extraction. It changes how parsing works and nothing else. You still fetch the pages, you still pay for retrieval, and you still face the same legal boundary. A model reading a page it should not have fetched is not a compliance improvement.

Layer 4: orchestration

The job is running all of the above reliably: scheduling, retries with backoff, rate limit budgeting, queueing, deduplication, storage, delivery and monitoring.

This is the layer teams under-build, and it is where projects quietly fail. Nobody demos orchestration. It is also the layer where the cost per usable result is actually determined, because retries, failed jobs and unvalidated successes all land here.

The metric worth instrumenting from day one is cost per validated result: total spend divided by outputs that passed a quality check. Not per request, not per record returned. If you count 200 responses as successes you will have a healthy dashboard and a rotting dataset.

Layer 5: authenticated access, the layer definitions leave out

The job is returning data the public web does not render, on the authorisation of the person it belongs to. This is not a harder version of layers one to four. It is a different mechanism, and it is the only route to a specific and commercially important set of fields.

The test is simple enough to run in thirty seconds. Open the target in a logged-out browser. Whatever is on the screen, layers one to four can get for you at scale. Whatever is not on the screen, they cannot get at any price, because the platform is not serving it to anyone who is not the account holder.

On social platforms specifically, five field groups sit on the far side of that line, and they are the ones most products are built on:

Field group	Why layers 1 to 4 cannot reach it	What layer 5 returns
True impressions and reach	Rendered only inside the account holder's own analytics	Exact counts per post
Audience demographics	Released to the account owner only	Age, gender and location distributions
Ephemeral content performance	Expires within 24 hours, never publicly archived	Captured while live
Earnings and payouts	Never rendered on any public surface	Platform earnings where exposed
Verified cross-platform identity	No public page reliably links one person across networks	Confirmed, not inferred

The mechanism is OAuth: the account holder connects through your product, approves scopes, and the platform serves the data to you on their behalf. Because the permission comes from the account holder rather than from a network trick, the output is first-party, auditable, and carries a documented legal basis. We went through the full comparison in authenticated versus public social data.

And the honest limit on layer five: it only covers people who connect. Layers one to four work on everyone and return less. Layer five works on fewer people and returns more. That trade is the entire architecture decision.

Where does governance sit?

Across all five, which is why it is drawn as a cross-cutting concern rather than a layer. Four questions have to be answerable at every layer or they are answerable at none.

• Provenance. For any given field, which layer produced it, from which source, at what time. If you cannot answer this per field, you cannot answer a client challenge or an audit.
• Lawful basis. Why were you entitled to collect this. Layer five answers with a consent record. Layers one to four answer with an argument about public availability, which is jurisdiction-dependent and contested.
• Retention and deletion. How long you hold it and what happens on a deletion request, including for data collected through layers where the subject never interacted with you.
• Audit trail. Who accessed what, when. This becomes a hard requirement the moment you sell into hiring, lending, immigration or anything regulated, as covered in social data use cases across HR, fintech, edtech and govtech.

Governance is the layer nobody budgets and everybody eventually needs. It is also the one that cannot be retrofitted cheaply, because provenance you did not record at collection time cannot be reconstructed afterwards.

What should you build and what should you buy?

The line moves per layer, and getting it wrong in either direction is expensive in a different way.

Layer	Default	Build only if	Common mistake
1. Network access	Buy	Proxy infrastructure is your product	Underestimating bandwidth cost on image-heavy targets
2. Retrieval	Buy	Retrieval is your product	Comparing blended success rates instead of per-target
3. Extraction	Buy first, build later	Your fields are genuinely bespoke	Over-building. It feels like the interesting problem
4. Orchestration	Build	Always. This is your product logic	Under-building. Nobody demos retries
5. Authenticated access	Buy	You need one platform only and have an engineer to spare	Assuming it is a harder version of layer 2, and scoping it as such
Governance	Build	Always. It is policy, not software	Retrofitting. Provenance not recorded cannot be reconstructed

The one worth arguing with is layer five. Native platform APIs are free, so a single-platform product with engineering capacity should build directly against them. Past two platforms the maintenance compounds faster than the integration count, because each one has its own auth model, its own rate limits and its own deprecation schedule.

How do the costs compare across layers?

They do not, at least not on a single number, because each layer bills for a different thing. This is the most common budgeting error in the category.

Layer	Billing unit	What actually drives your bill
Network access	Per gigabyte, from about $2.50	Page weight. Image-heavy and headless-rendered targets cost multiples of plain HTML
Retrieval and extraction	Per record, roughly $0.75 to $2.50 per thousand	Success rate. Failed and unvalidated results still consume budget
Compute platforms	Per compute unit	Runtime, which is close to unforecastable before you have run the workload
Datasets	Per record, around $2.50 per thousand	Volume and freshness. You are buying somebody else's prior collection
Authenticated access	Per connected account and usage	Connection count, not read count. Re-reading a connected account is near free

That last row is the one that inverts the maths. A per-record model bills you every time you refresh the same subject, so a product re-reading 5,000 accounts daily pays 150,000 times a month for 5,000 subjects. A per-account model does not. Which is cheaper depends entirely on your access pattern rather than on the rate, and we worked the arithmetic in social data API pricing.

How do you work out which layers you need?

Three questions, in order. The first one settles more architecture than the other two combined.

# Which layers do you actually need?

def layers_required(field_visible_logged_out,
                    subject_will_authorise,
                    platforms_count):
    layers = set()

    if field_visible_logged_out:
        layers |= {1, 2, 3, 4}          # the public-web stack
    else:
        if not subject_will_authorise:
            return "IMPOSSIBLE"        # stop. Redesign the product.
        layers |= {5, 4}               # consented access plus orchestration

    return sorted(layers)

# The 30-second version of question one:
#   open the target in a logged-out private window.
#   On the screen  -> layers 1 to 4.
#   Not on screen  -> layer 5, or nothing.

assert layers_required(True,  False, 1) == [1, 2, 3, 4]   # public collection
assert layers_required(False, True,  4) == [4, 5]         # consented data
assert layers_required(False, False, 1) == "IMPOSSIBLE"   # the expensive one

# Note layer 4 appears in both. Orchestration is not optional
# in either architecture, and it is the layer most often
# discovered rather than planned.

The IMPOSSIBLE branch is not hypothetical. Needing a field that is not rendered publicly, from a subject who will never authorise you, is not a hard engineering problem. It is a product that cannot be built, and no vendor in any layer changes that.

Where does Phyllo fit?

Layer five, and only layer five. We do not sell proxies, unblockers, headless browsers or general web scraping, and if you need layers one to four you should buy them from the companies that build them properly.

Phyllo's social data API is consented access across 25+ platforms through one integration: OAuth handled, token lifecycle managed, fields normalised into one schema, changes pushed over webhooks. It returns the five field groups from the layer five table, plus verified identity across a creator's connected accounts. Per-platform coverage is public at getphyllo.com/coverage, the API reference needs no sales call, and we hold GDPR compliance and SOC 2 Type II. For public conversation and mention tracking where there is no account holder to ask, social listening sits on the other side of the line.

Where we are the wrong purchase. If your fields are visible logged out, you do not need layer five and you should not pay for it. Buy proxies from a proxy vendor, retrieval from a scraping platform, and build your orchestration. That advice costs us nothing, because a product that only needs the public web was never going to be a customer, and pretending otherwise would waste a quarter of somebody's time.