Modern scraping operations

What changed about web scraping? The scripts still work, until they don’t. Then you’re debugging at 2 AM because a site changed a single CSS class and your entire pipeline went silent. Five years ago, scraping was a scripting problem. You wrote a parser, pointed it at a URL, and collected data. Today, scraping is an infrastructure problem. Cloudflare alone protects over 24 million active websites, and in July 2025 it began blocking AI crawlers by default across its entire network. The number of web security services tracked by Wappalyzer nearly doubled from 36 to 60 between 2022 and 2024. The web got harder to scrape, and it happened fast. This article examines why the landscape shifted, what production scraping systems look like now, and how teams are adapting their architecture to operate reliably at scale.

​

The economics shifted

The headline number most teams track is cost per gigabyte of proxy traffic. That number has fallen significantly over the past five years. But it’s the wrong metric. What actually matters is cost per successful payload: the total spend (proxy, compute, API fees) divided by the number of usable records you extract. That number has been climbing, because the techniques required to extract data have gotten more expensive even as individual input costs dropped. Here’s why:

• Sites that once responded to simple HTTP requests with datacenter proxies now require residential proxies (roughly 10x the cost) or full browser rendering (another 5-10x multiplier in compute)
• According to AImultiple’s web scraping challenges research, blocking is now the primary challenge developers face, with static IP blocking replaced by continuous behavioral trust scoring
• The share of requests that need heavyweight extraction methods (residential IPs, headless browsers, or both) has grown substantially, driven by widespread adoption of anti-bot services

This creates what you might call scraping shock: data remains technically reachable but becomes economically painful to extract at previous cost levels.

​

Three forces driving the shift

​

1. Anti-bot systems got smarter, faster

In September 2024, Cloudflare introduced one-click AI bot blocking. Over 1 million customers activated it. By July 2025, Cloudflare began blocking AI crawlers by default across its entire network. Detection moved beyond IP reputation and cookie validation. Modern anti-bot systems now analyze:

• TLS fingerprints. Python’s Requests library has a different TLS signature than Chrome, making the library itself identifiable.
• Browser fingerprints. Screen resolution, WebGL renderer, canvas fingerprint, installed fonts, audio context, and GPU timing.
• Behavioral signals. Mouse movement patterns, scroll velocity, click timing, and keystroke cadence.
• Header consistency. Real browsers send different headers per request type; bots send identical headers every time.

The popular puppeteer-stealth library was deprecated in February 2025 because it could no longer bypass current Cloudflare versions. As developers frequently report in scraping communities: stock Selenium and Puppeteer leak automation signals everywhere, and techniques that work today stop working next month.

​

2. Platforms locked down deliberately

This isn’t just technical escalation. It’s strategic. Platforms are treating their data as a competitive asset, especially against AI companies.

• Reddit filed a lawsuit in October 2025 against multiple data collection companies and Perplexity AI for circumventing security measures.
• X/Twitter implemented IP reputation scoring in October 2024, bound guest tokens to browser fingerprints in January 2025, and permanently banned datacenter IPs.
• Cloudflare’s “AI Labyrinth” creates honeypot traps that force AI agents to waste compute resources on fake content.

These aren’t isolated incidents. They represent a coordinated shift where platforms view unrestricted data access as a threat to their business model.

​

3. AI crawlers changed the game for everyone

AI crawler traffic from GPTBot rose 147% between July 2024 and July 2025. Meta-ExternalAgent traffic rose 843%. These crawlers collectively generate around 50 billion requests daily, roughly 1% of all web traffic according to Cloudflare. This flood of AI traffic triggered a defensive response that affects all scrapers, not just AI crawlers. Sites that never bothered with anti-bot protection now have it. The collateral damage hits every team running a data pipeline.

​

What production scraping looks like now

The gap between “scraping works on my laptop” and “scraping works in production” has never been wider. Low-code tools and LLMs help get from 0 to 1 (getting a scraper working against a single page). But getting from 1 to 100 (running reliably at scale) requires systems thinking. The operational reality of scraping at scale is sobering: successfully operating at even fewer than 1 million page views per day requires managing viewport size matching, user-agent rotation, multiple VLANs per container, and datacenter IP avoidance. Here’s what a production scraping system actually looks like: The architecture has five layers, each solving a distinct operational concern:

​

Orchestration layer

This is the control plane. It manages what gets scraped, when, and how. At minimum, it includes:

• Job scheduler. Prioritizes and queues scrape jobs based on data freshness requirements.
• Request router. Decides the extraction method per target (HTTP client, headless browser, or managed API) based on the site’s difficulty profile.
• Retry logic with backoff. Handles transient failures without hammering targets.

Most teams start without this layer. They run scrapers as cron jobs and handle routing manually. That works at hundreds of pages per day. It breaks at hundreds of thousands.

​

Extraction layer

This is where the actual scraping happens. The key insight is that not every request needs the same tool:

• HTTP clients handle 0.5-2 seconds per request at minimal resource cost. They work for static content and sites without heavy JavaScript rendering.
• Headless browsers handle JavaScript-rendered content but consume 200-500 MB of RAM per instance and take 3-15 seconds per request. That’s a 5-10x infrastructure multiplier.
• Managed scraping APIs offload anti-bot handling, proxy rotation, and browser management to a third party. The per-request cost is higher, but you eliminate infrastructure maintenance.

The production pattern is a tiered approach: try the cheapest method first, escalate only when it fails, and cache the result so subsequent requests to the same domain use the right method automatically.

​

Validation layer

This is the layer most teams skip, and the one that causes the most damage at scale. Scaling without validation means scaling error. A blocked scraper doesn’t always return an HTTP error. It may return valid-looking HTML with wrong data: a CAPTCHA page, a soft block, or a region-specific version of the content. Your downstream systems ingest it silently. Production validation includes:

• Schema validation before data enters storage: expected fields, types, and value ranges.
• Content fingerprinting to detect when a response matches a known block page pattern rather than real content.
• Field completeness checks that flag records where critical fields are missing or suspiciously uniform.

Teams that add schema validation to their scraping pipelines consistently report significant reductions in downstream data quality issues and ML model errors.

​

Observability layer

Treat scraping like a production service. If you run a web application without monitoring, you’d be negligent. Scraping pipelines deserve the same discipline. Key metrics to track:

• Success rate per domain. The percentage of requests that return valid, usable data (not just HTTP 200).
• Cost per successful payload. The total cost (proxy + compute + API) divided by usable records extracted.
• Selector health. Automated checks that detect when CSS selectors or XPath expressions stop matching expected elements.
• Latency distribution. p50 and p99 response times per domain, which signal when a site starts throttling.

Teams that implement observability-driven scraper redesigns consistently see dramatic improvements: fewer job failures, faster detection of site changes, and lower operational overhead from reduced manual debugging.

​

Storage and delivery layer

Clean, validated data flows into storage and downstream systems. This layer is straightforward but matters for pipeline integrity:

• Deduplication before write. Scraping the same product page twice shouldn’t create two records.
• Timestamping. Every record carries a collection timestamp so downstream consumers know data freshness.
• Schema evolution. When sites change their data structure, your storage schema needs to accommodate both old and new formats during the transition.

​

The build vs. buy decision

At some point, every scraping team asks: should we maintain this infrastructure ourselves, or use a managed service? The breakeven math depends on scale and site difficulty: Self-hosted browser infrastructure costs $200-800/month plus ongoing engineering time. At scale, the engineering time dominates: quarterly selector fixes, proxy pool management, and anti-bot adaptation are recurring costs that don’t shrink with automation. The practical pattern is a hybrid approach:

• Self-host for sites where simple HTTP requests work and data formats are stable.
• Use managed APIs for sites with heavy anti-bot protection, frequent layout changes, or JavaScript-heavy rendering.

Bright Data’s product suite maps to this layered model. Web Unlocker handles the anti-bot layer for HTTP requests. Scraping Browser provides managed headless browsers without the RAM and scaling headaches. Web Scraper API handles the full extraction pipeline for supported sites, returning structured data directly. The decision isn’t binary. Most production teams use a mix of self-hosted and managed components, optimized per domain based on difficulty, cost, and data freshness requirements.

​

Common architectural mistakes

These patterns show up repeatedly in scraping teams that struggle at scale:

• No validation layer. The single most common failure. Teams scale their extraction without validating output, then discover weeks later that half their data is CAPTCHA pages stored as legitimate records.
• Defaulting to browsers. Headless browsers are 5-10x more expensive than HTTP clients. Teams that route every request through Puppeteer or Playwright are burning money on sites that would respond fine to a well-configured HTTP request with proper headers.
• Hardcoded selectors without monitoring. Static XPath and CSS selectors become invalid silently when sites change their DOM. Without automated selector health checks, data quality degrades without triggering any alert.
• Treating scraping as a script, not a service. Cron jobs without health checks, no alerting on success rate drops, no cost tracking per domain. This works at small scale. It creates invisible failures at production scale.
• Ignoring the economics. Scraping costs can spike 5-10x overnight when a site upgrades its defenses. Teams without cost-per-payload tracking don’t notice until the monthly bill arrives.

​

Where this is heading

Three trends will shape scraping operations over the next two years:

1. Legal frameworks are crystallizing. Reddit’s lawsuit, the EU AI Act’s data sourcing requirements, and platform-specific terms of service are creating a compliance layer that scraping systems need to account for. Teams that treat compliance as an afterthought will face operational disruption.
2. AI is on both sides. Sites use ML models to detect bots. Scrapers use LLMs to parse unstructured content. The cost of LLM-based parsing is still orders of magnitude higher than traditional approaches at scale, but it’s falling. The teams that figure out when to use AI parsing and when to use traditional selectors will have a cost advantage.
3. Managed infrastructure is becoming the default. As anti-bot systems get more sophisticated, the maintenance burden of self-hosting increases. The trend line points toward more teams buying infrastructure and focusing engineering effort on data quality and business logic.

The scraping teams that thrive won’t be the ones writing the cleverest bypass scripts. They’ll be the ones running the most disciplined operations, with clear cost tracking, automated quality validation, and architecture that adapts to an adversarial web without manual intervention.

​

Further reading

• How Bright Data handles anti-bot blocking: deep dive into detection layers and how managed services defeat them
• How to reduce web scraping costs: practical techniques for optimizing at the request level
• Ethical web scraping best practices: compliance principles and regulatory context
• Understanding asynchronous requests: when to use async vs sync for performance and cost