How to scrape in bulk with async requests

Use the asynchronous /trigger endpoint to scrape more than 20 URLs in one job, run discovery collections or deliver results to a webhook or cloud storage. The workflow is the same for every Bright Data scraper: trigger, poll, download. Only the dataset_id changes.

Not sure whether to use sync or async? Read Understanding sync vs. async requests.

Prerequisites

A Bright Data account with an active API key
The dataset_id for the scraper you want to run, from the table below
Familiarity with the synchronous request flow for your scraper, for example LinkedIn Scraper API endpoints

Which dataset ID do I use?

Every scraper has its own dataset_id. The examples on this page use LinkedIn profiles. Substitute the ID for the platform you are scraping.

Scraper	Primary `dataset_id`	Endpoint reference
LinkedIn	`gd_l1viktl72bvl7bjuj0`	LinkedIn Scraper API
Instagram	`gd_l1vikfch901nx3by4`	Instagram Scraper API
TikTok	`gd_l1villgoiiidt09ci`	TikTok Scraper API
Amazon	`gd_l7q7dkf244hwjntr0`	Amazon Scraper API
ChatGPT	`gd_m7aof0k82r803d5bjm`	ChatGPT Scraper API
Facebook	`gd_mf0urb782734ik94dz`	Facebook Scraper API
X (Twitter)	`gd_lwxmeb2u1cniijd7t4`	X (Twitter) Scraper API
YouTube	`gd_lk538t2k2p1k3oos71`	YouTube Scraper API
Reddit	`gd_lvz8ah06191smkebj4`	Reddit Scraper API
Google	`gd_m8ebnr0q2qlklc02fz`	Google Scraper API

Each scraper exposes several endpoints, for example Amazon has separate IDs for products, reviews and sellers. The full list for each platform is on its introduction page, and every ID is browsable in the scraper library.

Step 1: Trigger the collection

Send a POST request to the /trigger endpoint with your input URLs:

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {"url": "https://www.linkedin.com/in/satyanadella"},
    {"url": "https://www.linkedin.com/in/jeffweiner08"},
    {"url": "https://www.linkedin.com/in/rbranson"},
    {"url": "https://www.linkedin.com/in/sherylsandberg"},
    {"url": "https://www.linkedin.com/in/raboram"}
  ]'

import requests

response = requests.post(
    "https://api.brightdata.com/datasets/v3/trigger",
    params={
        "dataset_id": "gd_l1viktl72bvl7bjuj0",
        "format": "json",
    },
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json=[
        {"url": "https://www.linkedin.com/in/satyanadella"},
        {"url": "https://www.linkedin.com/in/jeffweiner08"},
        {"url": "https://www.linkedin.com/in/rbranson"},
        {"url": "https://www.linkedin.com/in/sherylsandberg"},
        {"url": "https://www.linkedin.com/in/raboram"},
    ],
)

snapshot = response.json()
print("Snapshot ID:", snapshot["snapshot_id"])

const response = await fetch(
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json",
  {
    method: "POST",
    headers: {
      "Authorization": "Bearer YOUR_API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify([
      { url: "https://www.linkedin.com/in/satyanadella" },
      { url: "https://www.linkedin.com/in/jeffweiner08" },
      { url: "https://www.linkedin.com/in/rbranson" },
      { url: "https://www.linkedin.com/in/sherylsandberg" },
      { url: "https://www.linkedin.com/in/raboram" },
    ]),
  }
);

const snapshot = await response.json();
console.log("Snapshot ID:", snapshot.snapshot_id);

You should see a 200 response with a snapshot_id:

{
  "snapshot_id": "sd_m1a2b3c4d5e6f7g8h"
}

Save this ID. You need it to check progress and download results.

Step 2: Monitor progress

Poll the snapshot status until it shows ready. This takes 30 seconds to several minutes depending on the number of URLs.

curl "https://api.brightdata.com/datasets/v3/progress/sd_m1a2b3c4d5e6f7g8h" \
  -H "Authorization: Bearer YOUR_API_KEY"

import time

snapshot_id = "sd_m1a2b3c4d5e6f7g8h"

while True:
    status_response = requests.get(
        f"https://api.brightdata.com/datasets/v3/progress/{snapshot_id}",
        headers={"Authorization": "Bearer YOUR_API_KEY"},
    )
    status = status_response.json().get("status")
    print(f"Status: {status}")

    if status == "ready":
        break
    time.sleep(10)

const snapshotId = "sd_m1a2b3c4d5e6f7g8h";

let status = "starting";
while (status !== "ready") {
  const statusResponse = await fetch(
    `https://api.brightdata.com/datasets/v3/progress/${snapshotId}`,
    { headers: { "Authorization": "Bearer YOUR_API_KEY" } }
  );
  const statusData = await statusResponse.json();
  status = statusData.status;
  console.log("Status:", status);

  if (status !== "ready") {
    await new Promise((r) => setTimeout(r, 10000));
  }
}

Status values:

Status	Meaning
`starting`	The job is queued and has not begun scraping
`running`	Scraping is in progress
`ready`	Results are available for download
`failed`	The collection encountered an error
`canceled`	The job was canceled before it finished

Step 3: Download results

Once the status is ready, download the scraped data:

curl "https://api.brightdata.com/datasets/v3/snapshot/sd_m1a2b3c4d5e6f7g8h?format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -o results.json

results_response = requests.get(
    f"https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}",
    params={"format": "json"},
    headers={"Authorization": "Bearer YOUR_API_KEY"},
)

results = results_response.json()
print(f"Collected {len(results)} records")

const resultsResponse = await fetch(
  `https://api.brightdata.com/datasets/v3/snapshot/${snapshotId}?format=json`,
  { headers: { "Authorization": "Bearer YOUR_API_KEY" } }
);

const results = await resultsResponse.json();
console.log(`Collected ${results.length} records`);

You have triggered, monitored and downloaded a batch scraping job.

Skip polling with webhooks

If you don’t want to poll for status, add an endpoint parameter to receive results automatically:

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&endpoint=https://your-server.com/webhook" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://www.linkedin.com/in/satyanadella"}]'

See Deliver results to webhooks and cloud storage for the full setup.

Limits and constraints

Constraint	Value
Max input file size	1 GB
Max concurrent jobs	5000
Webhook delivery size	Up to 1 GB
API download size	Up to 5 GB
`deadline` query parameter	How long the trigger request waits before returning an error: a duration such as `3m`, `15min` or `2h`, or an ISO timestamp such as `2026-09-30T18:00:00.000Z`. Without it the request returns a `snapshot_id` at once

How to limit records per input

When running a discovery collection, you can cap the number of results returned per input. Set this in the scraper’s configuration in the Control Panel.

With a limit of 10, each input returns at most 10 records.

Troubleshooting

Getting a 429 Too Many Requests error?

You receive 429 Too Many Requests when you exceed the concurrent job limit, 5,000 active jobs or snapshots. Bright Data blacklists any IP that collects 25 or more 429 responses within 5 minutes, and a blacklisted IP is blocked from every API request until support clears it. So a 429 is a signal to slow down, not to retry at once.

Stop sending new requests as soon as you receive a 429.
Wait before retrying: the number of seconds in the Retry-After header if present, otherwise exponential backoff of 2, 4, 8, 16 and 32 seconds.
Reduce concurrency if 429s keep coming; you are over the limit.

Ten or more 429 responses within 5 minutes means you are close to the 25-in-5-minutes blacklist threshold. Reduce the request rate right away.

response = requests.post(api_url, json=data)

if response.status_code == 429:
    requests.post(api_url, json=data)  # retrying at once is what gets an IP blacklisted

import time
import requests

resp = requests.post(api_url, json=data)

if resp.status_code == 429:
    retry_after = resp.headers.get("Retry-After")
    wait_s = int(retry_after) if retry_after else 2
    time.sleep(wait_s)
    # retry after waiting, and double the wait if the 429 repeats

If your IP is already blacklisted, contact Bright Data support with the blocked address to request whitelisting.

Snapshot status shows ‘failed’?

Check that all input URLs are valid for the scraper you are calling. A LinkedIn URL sent to the Amazon dataset_id fails. Review the error details in the snapshot response or in the Logs tab of your Bright Data dashboard.

Results are incomplete or missing some URLs?

Some URLs may fail individually while the overall job succeeds. Check the snapshot response for any errors field. Retry failed URLs in a separate request.

FAQ

How many URLs can I send in one async request?

Async requests accept up to 1 GB of input data per job, which is tens of thousands of URLs. Use the synchronous endpoint only when you have 20 URLs or fewer and want the results in the same response.

How are inputs processed across concurrent requests?

All inputs are processed in parallel. Processing capacity is shared equally across all running jobs for a given scraper. Start time does not affect priority either, jobs are not processed in first-in-first-out or last-in-first-out order. A job started an hour ago and a job started a minute ago receive the same share of capacity.

Do I need a different dataset ID for each platform?

Yes. The dataset_id query parameter selects both the platform and the endpoint, for example Amazon products versus Amazon reviews. The table above lists the primary ID per platform, and each platform’s introduction page lists the rest.

How long does a snapshot stay available?

Snapshots are downloadable for 30 days after the job completes. Deliver results to a webhook or cloud storage if you need them retained longer.

Get started

Data extraction

Web search

AI training data

Proxy networks

How to scrape in bulk with async requests

Prerequisites

Which dataset ID do I use?

Step 1: Trigger the collection

Step 2: Monitor progress

Step 3: Download results

Skip polling with webhooks

Limits and constraints

How to limit records per input

Troubleshooting

Getting a 429 Too Many Requests error?

Snapshot status shows ‘failed’?

Results are incomplete or missing some URLs?

FAQ

How many URLs can I send in one async request?

How are inputs processed across concurrent requests?

Do I need a different dataset ID for each platform?

How long does a snapshot stay available?

​Prerequisites

​Which dataset ID do I use?

​Step 1: Trigger the collection

​Step 2: Monitor progress

​Step 3: Download results

​Skip polling with webhooks

​Limits and constraints

​How to limit records per input

​Troubleshooting

​Getting a 429 Too Many Requests error?

​Snapshot status shows ‘failed’?

​Results are incomplete or missing some URLs?

​FAQ

​How many URLs can I send in one async request?

​How are inputs processed across concurrent requests?

​Do I need a different dataset ID for each platform?

​How long does a snapshot stay available?

​Related

Prerequisites

Which dataset ID do I use?

Step 1: Trigger the collection

Step 2: Monitor progress

Step 3: Download results

Skip polling with webhooks

Limits and constraints

How to limit records per input

Troubleshooting

Getting a 429 Too Many Requests error?

Snapshot status shows ‘failed’?

Results are incomplete or missing some URLs?

FAQ

How many URLs can I send in one async request?

How are inputs processed across concurrent requests?

Do I need a different dataset ID for each platform?

How long does a snapshot stay available?

Related