Skip to main content
Archive API is a massive, continuously expanding, cached repository by Bright Data, designed to capture and deliver public web data at scale.It provides full web pages and metadata, making it ideal for AI training, machine learning, and large-scale data analysis.Unlike traditional web crawls, Archive API prioritizes relevance, freshness, and usability, giving you access to the most important parts of the internet as they are scraped daily.
As of August 2026, the Bright Data Archive holds over~115 PB across ~800 billion pages from ~380 million domains, and keeps growing.Collection runs continuously, so the totals grow every day:That growth rate makes Archive the largest up-to-date web data repository available for AI and data-driven applications.
You can start accessing data immediately through our Archive API. The API allows you to search, retrieve and filter data snapshots from Archive seamlessly and efficiently.
  • Data from the last 24 hours: Will take from within minutes and up to a few hours to deliver (depending on snapshot size)
  • Data older than 24 hours: Will take up to 48 hours to process and start delivery (depending on snapshot size)
Archive API is priced by data age and billed per CPM, meaning cost per thousand pages.Before you run a dump, GET /webarchive/search/{search_id} returns dump_cost_usd for that search, plus a cost_breakdown object splitting the estimate between cache pages and archive pages.
Archive API offers four delivery destinations:
  • Amazon S3 bucket: Have your Data Snapshot delivered directly to your S3 bucket.
  • Azure Blob Storage: Deliver the data snapshot directly to your Azure Blob container.
  • Google Cloud Storage: Deliver the data snapshot directly to your GCS bucket.
  • Webhook: Retrieved via webhook for real-time integration into your systems.
Webhook delivery is not suitable for large dumps, which can reach 1 GB. See Deliver to cloud for the settings each destination requires.
Absolutely! Archive API allows filtering by category, domains, date, languages, and country before retrieving data, ensuring you only get what you need.
When working with large-scale web data, freshness, relevance, and accessibility are key. While Common Crawl provides a broad historical snapshot of the web, Bright Data’s Archive API offers real-time, continuously updated data with advanced filtering and delivery options. Here’s how they compare: