Skip to main content
The Web Scraper API lets you extract data from websites programmatically. It offers both synchronous and asynchronous scraping methods for different use cases, from quick data retrieval to complex, large-scale extraction jobs. The API handles real-time processing for up to 20 URL inputs, and batch processing for larger collections, accommodating various scraping requirements.
overview.png

​
Scraping methods

​
Synchronous scraping (/scrape)

Synchronous scraping allows you to initiate a scrape and receive results in a single request, ideal for real-time quick data retrieval. 
  curl "https://api.brightdata.com/datasets/v3/scrape?dataset_id=gd_l1viktl72bvl7bjuj0&format=json" \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://www.linkedin.com/in/elad-moshe-05a90413/"}]'

​
Key features of synchronous scraping:

  • Real-time results in the same request
  • Perfect for single URL and quick extractions
  • Simplified error handling
  • 1-minute timeout (automatically switches to async if exceeded)

​
Asynchronous scraping (/trigger)

Asynchronous scraping initiates a job that runs in the background, allowing you to handle larger and more complex scraping tasks in batch mode. Batch mode allows up to 100 concurrent requests, and each batch can process up to 1GB of inputs file size, making it ideal for high-volume data collection projects. Discovery tasks (finding related products, scraping multiple pages) require the asynchronous scraping (/trigger) due to their need to navigate and extract data across multiple web pages. 
curl "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&uncompressed_webhook=true" \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {"url": "https://www.linkedin.com/in/elad-moshe-05a90413/"},
    {"url": "https://www.linkedin.com/in/jonathan-myrvik-3baa01109"},
    {"url": "https://www.linkedin.com/in/aviv-tal-75b81/"},
    {"url": "https://www.linkedin.com/in/bulentakar/"},
    {"url": "https://www.linkedin.com/in/nnikolaev/"}
  ]'

​
Key features of asynchronous scraping:

  • Handles multiple URLs in batch processing (up to 1GB of inputs file size)
  • No timeout limitations for long-running jobs
  • Progress monitoring via status checks
  • Ideal for large datasets
  • Required for “discovery” tasks that need to crawl multiple pages or perform complex data extraction

​
Understanding Synchronous and Asynchronous Scraping

When dealing with web scraping, it’s essential to choose the right method based on your needs:

​
Synchronous Scraping (/scrape):

  • Purpose: Designed for quick, single-url jobs where instant results are required.
  • Response: Returns the scraped data directly within the same request.
  • Ideal for: Quick data checks, single-page extractions without long runtimes.
Have a simple one-link case? Consider trying Synchronous Requests for instant results.

​
Asynchronous Scraping (/trigger):

  • Purpose: Used for large, complex, or long-running scraping jobs.
  • Response: Immediately returns a snapshot_id instead of data. This snapshot_id is used to track job progress or download the data later.
  • Ideal for: Batch processing, multiple URLs, or high-volume data collection. Requires managing multiple pages or complex data extractions.
Looking for large-scale scraping? Asynchronous Requests might be suited for your complex tasks.

​
How To Collect?

​
Trigger a Collection (Demo)

  1. Choose your target website from our API offerings
  2. Select the appropriate scraper for your needs
  3. Decide between synchronous or asynchronous scraping based on your requirements:
    • Use synchronous (/scrape) for immediate results and simple extractions
    • Use asynchronous (/trigger) for complex scraping, multiple URLs, or large datasets
  4. Provide your input URLs via JSON or CSV
  5. Enable error reporting to track any issues
  6. Select your preferred delivery method

​
Via Webhook:

  1. Set your webhook URL and Authorization header if needed
  2. Select your preferred file format (JSON, NDJSON, JSON lines, CSV)
  3. Choose whether to send it compressed or not
  4. Test webhook to validate that the operation runs successfully (using sample data)
  5. Copy the code and run it.
via-webhook.png

​
Via Deliver to external storage:

  1. Select your preferred delivery location (S3, Google Cloud, Snowflake, or any other available option)
  2. Fill out the needed credentials according to your pick
  3. Select your preferred file format (JSON, NDJSON, JSON lines, CSV)
  4. Copy the code and run it.
via-deliver-to-external-storage.png

​
Limit records

While running a discovery API, you can set a limit of the number of results per input provided
limit-per-input-disabled.png
In the example below, we’ve set a limitation of 10 results per input
limit-per-input-10.png

​
Management APIs

Additional actions you can do using our different API endpoints
management-apis.png

​
Get snapshot list

Check your snapshot history with this API. It returns a list of all available snapshots, including the snapshot ID, creation date, and status. (link to endpoint playground)
get-snapshot-list.png
If you are confused by receiving a snapshot_id, it means an asynchronous call was made. Consider using the synchronous /scrape endpoint for simpler, immediate tasks.

​
Monitor Progress

Check your data collection status with this API. It should return “collecting” while gathering data, “digesting” when processing, and “ready” when available. (link to endpoint playground)
monitor-progress.png

​
Cancel snapshot

Cancel a running collection, stop your data collection before finishing with this API. It should return “ok” while managing to stop the collection. (link to endpoint playground)

​
Monitor Delivery

Check your delivery status with this API. It should return “done” while the delivery was completed, “canceled” when the delivery was canceled, and “Failed” when the delivery was not completed. (link to endpoint playground)
monitor-progress.png

​
System limitations

​
File size

Inputup to 1GB
Webhook deliveryup to 1GB
API Downloadup to 5GB (for bigger files use API delivery)
Delivery APIunlimited

​
Rate Limits & Concurrent Requests

To ensure stable performance and fair usage, the Web Scraper API enforces rate limits based on the type of request: single input or batch input. Exceeding these limits will result in a 429 error response.

​
What is the Rate Limit?

The Web Scraper API supports the following maximum number of concurrent requests:
MethodRate-limit
Up to 20 inputs per requestup to 1500 concurrent requests
Over 20 inputs per requestup to 100 concurrent requests
If your application exceeds these limits, the API will return the following error:
429 Client Error: Too Many Requests for URL
This error indicates that your request rate has surpassed the allowed threshold.

​
How to Avoid Hitting Rate Limits

To reduce the number of concurrent requests and stay within the rate limits:
  • Use the batch input method whenever possible.
  • A single batch request can include up to 1GB of input data (file size).
  • By combining multiple inputs into one batch request, you can minimize the total number of requests sent concurrently.
When using the Discovery Scraper the system automatically utilizes batch input method where rate limit is up to 100 requests.

​
Best Practices for Managing API Rate Limits

  • Monitor your request volume and adjust your concurrency accordingly.
  • Use batch requests to group multiple scraping tasks into fewer API calls.
  • Contact our support team if you need help optimizing your requests.