> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brightdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# How to scrape in bulk with async requests

> Trigger bulk Scraper API jobs with the async /trigger endpoint, monitor snapshot progress and download results. Works with every Bright Data scraper.

Use the asynchronous `/trigger` endpoint to scrape more than one URL in one job, run discovery collections or deliver results to a webhook or cloud storage. The workflow is the same for every Bright Data scraper: trigger, poll, download. Only the `dataset_id` changes.

> [!TIP]
>
> Not sure whether to use sync or async? Read [Understanding sync vs. async requests](https://docs.brightdata.com/concepts/sync-vs-async.md).

## Prerequisites

* A [Bright Data account](https://brightdata.com/?hs_signup=1\&utm_source=docs) with an active API key
* The `dataset_id` for the scraper you want to run, from the table below
* Familiarity with the synchronous request flow for your scraper, for example [LinkedIn Scraper API endpoints](https://docs.brightdata.com/products/scrapers/linkedin/send-first-request.md)

## Which dataset ID do I use?

Every scraper has its own `dataset_id`. The examples on this page use Instagram profiles discovered by username (`gd_l1vikfch901nx3by4`). Substitute the ID for the platform you are scraping.

| Scraper | Primary `dataset_id` | Endpoint reference |
| :- | :- | :- |
| LinkedIn | `gd_l1viktl72bvl7bjuj0` | [LinkedIn Scraper API](https://docs.brightdata.com/products/scrapers/linkedin/introduction.md) |
| Instagram | `gd_l1vikfch901nx3by4` | [Instagram Scraper API](https://docs.brightdata.com/products/scrapers/instagram/introduction.md) |
| TikTok | `gd_l1villgoiiidt09ci` | [TikTok Scraper API](https://docs.brightdata.com/products/scrapers/tiktok/introduction.md) |
| Amazon | `gd_l7q7dkf244hwjntr0` | [Amazon Scraper API](https://docs.brightdata.com/products/scrapers/amazon/introduction.md) |
| ChatGPT | `gd_m7aof0k82r803d5bjm` | [ChatGPT Scraper API](https://docs.brightdata.com/products/scrapers/chatgpt/introduction.md) |
| Facebook | `gd_mf0urb782734ik94dz` | [Facebook Scraper API](https://docs.brightdata.com/products/scrapers/facebook/introduction.md) |
| X (Twitter) | `gd_lwxmeb2u1cniijd7t4` | [X (Twitter) Scraper API](https://docs.brightdata.com/products/scrapers/twitter/introduction.md) |
| YouTube | `gd_lk538t2k2p1k3oos71` | [YouTube Scraper API](https://docs.brightdata.com/products/scrapers/youtube/introduction.md) |
| Reddit | `gd_lvz8ah06191smkebj4` | [Reddit Scraper API](https://docs.brightdata.com/products/scrapers/reddit/introduction.md) |
| Google | `gd_m8ebnr0q2qlklc02fz` | [Google Scraper API](https://docs.brightdata.com/products/scrapers/google/introduction.md) |

Each scraper exposes several endpoints, for example Amazon has separate IDs for products, reviews and sellers. The full list for each platform is on its introduction page, and every ID is browsable in the [scraper library](https://docs.brightdata.com/products/scrapers/overview.md).

For the dataset ID of every pre-built scraper, including sites without an introduction page, read [scrapers.json](https://docs.brightdata.com/scrapers.json). It lists the dataset ID, name, domain and collection methods of each scraper and refreshes daily. [scrapers-full.json](https://docs.brightdata.com/scrapers-full.json) adds the input schema and output fields of every collection method. To get the same details through the API, call [Get all scrapers](https://docs.brightdata.com/api-reference/scrapers/management-apis/get-scrapers.md) with your API key.

## Step 1: Trigger the collection

Send a `POST` request to the `/trigger` endpoint with an object body holding your inputs:

```bash cURL theme={null}
curl -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input":[{"user_name":"zoobarcelona"}],"limit_per_input":null}' \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1vikfch901nx3by4&notify=false&include_errors=true&type=discover_new&discover_by=user_name"
```

```python Python (requests) theme={null}
import requests

response = requests.post(
    "https://api.brightdata.com/datasets/v3/trigger",
    params={
        "dataset_id": "gd_l1vikfch901nx3by4",
        "notify": "false",
        "include_errors": "true",
        "type": "discover_new",
        "discover_by": "user_name",
    },
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"input": [{"user_name": "zoobarcelona"}], "limit_per_input": None},
)

snapshot = response.json()
print("Snapshot ID:", snapshot["snapshot_id"])
```

```javascript Node.js (Axios) theme={null}
import axios from "axios";

const response = await axios.post(
  "https://api.brightdata.com/datasets/v3/trigger",
  { input: [{ user_name: "zoobarcelona" }], limit_per_input: null },
  {
    params: {
      dataset_id: "gd_l1vikfch901nx3by4",
      notify: false,
      include_errors: true,
      type: "discover_new",
      discover_by: "user_name",
    },
    headers: { Authorization: "Bearer YOUR_API_KEY" },
  }
);

console.log("Snapshot ID:", response.data.snapshot_id);
```

```javascript Node.js (fetch) theme={null}
const params = new URLSearchParams({
  dataset_id: "gd_l1vikfch901nx3by4",
  notify: "false",
  include_errors: "true",
  type: "discover_new",
  discover_by: "user_name",
});

const response = await fetch(`https://api.brightdata.com/datasets/v3/trigger?${params}`, {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ input: [{ user_name: "zoobarcelona" }], limit_per_input: null }),
});

const snapshot = await response.json();
console.log("Snapshot ID:", snapshot.snapshot_id);
```

You should see a `200` response with a `snapshot_id`:

```json theme={null}
{
  "snapshot_id": "sd_m1a2b3c4d5e6f7g8h"
}
```

Save this ID. You need it to check progress and download results.

## Step 2: Monitor progress

Poll the snapshot status until it shows `ready`, and stop if it shows `failed` or `canceled`. This takes 30 seconds to several minutes depending on the number of URLs.

```bash cURL theme={null}
curl "https://api.brightdata.com/datasets/v3/progress/sd_m1a2b3c4d5e6f7g8h" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

```python Python theme={null}
import time
import requests

snapshot_id = "sd_m1a2b3c4d5e6f7g8h"

for attempt in range(60):  # stop after about 10 minutes
    status_response = requests.get(
        f"https://api.brightdata.com/datasets/v3/progress/{snapshot_id}",
        headers={"Authorization": "Bearer YOUR_API_KEY"},
    )
    status_response.raise_for_status()
    progress = status_response.json()
    status = progress.get("status")
    print(f"Status: {status}")

    if status == "ready":
        break
    if status in ("failed", "canceled"):
        raise RuntimeError(f"Snapshot {status}: {progress.get('error_message', 'no details')}")
    time.sleep(10)
else:
    raise TimeoutError("Snapshot was not ready after 10 minutes")
```

```javascript Node.js theme={null}
const snapshotId = "sd_m1a2b3c4d5e6f7g8h";
const maxAttempts = 60; // stop after about 10 minutes

for (let attempt = 1; ; attempt++) {
  const statusResponse = await fetch(
    `https://api.brightdata.com/datasets/v3/progress/${snapshotId}`,
    { headers: { "Authorization": "Bearer YOUR_API_KEY" } }
  );
  if (!statusResponse.ok) throw new Error(`HTTP ${statusResponse.status}`);
  const progress = await statusResponse.json();
  console.log("Status:", progress.status);

  if (progress.status === "ready") break;
  if (progress.status === "failed" || progress.status === "canceled") {
    throw new Error(`Snapshot ${progress.status}: ${progress.error_message ?? "no details"}`);
  }
  if (attempt === maxAttempts) throw new Error("Snapshot was not ready after 10 minutes");
  await new Promise((r) => setTimeout(r, 10000));
}
```

Status values:

| Status | Meaning |
| :- | :- |
| `starting` | The job is queued and has not begun scraping |
| `running` | Scraping is in progress |
| `ready` | Results are available for download |
| `failed` | The collection encountered an error |
| `canceled` | The job was canceled before it finished |

## Step 3: Download results

Once the status is `ready`, download the scraped data:

```bash cURL theme={null}
curl "https://api.brightdata.com/datasets/v3/snapshot/sd_m1a2b3c4d5e6f7g8h?format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -o results.json
```

```python Python theme={null}
results_response = requests.get(
    f"https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}",
    params={"format": "json"},
    headers={"Authorization": "Bearer YOUR_API_KEY"},
)

results = results_response.json()
print(f"Collected {len(results)} records")
```

```javascript Node.js theme={null}
const resultsResponse = await fetch(
  `https://api.brightdata.com/datasets/v3/snapshot/${snapshotId}?format=json`,
  { headers: { "Authorization": "Bearer YOUR_API_KEY" } }
);

const results = await resultsResponse.json();
console.log(`Collected ${results.length} records`);
```

You have triggered, monitored and downloaded a batch scraping job.

## Skip polling with webhooks

If you don't want to poll for status, add an `endpoint` parameter to receive results automatically:

```bash theme={null}
curl -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input":[{"user_name":"zoobarcelona"}],"limit_per_input":null}' \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1vikfch901nx3by4&notify=false&include_errors=true&type=discover_new&discover_by=user_name&endpoint=https://your-server.com/webhook"
```

See [Deliver results to webhooks and cloud storage](https://docs.brightdata.com/products/scrapers/scrapers-library/data-delivery.md) for the full setup.

## Limits and constraints

Input size, running job, download and delivery limits for the Scraper API are listed in one table on [Scraper API limits](https://docs.brightdata.com/products/scrapers/limits.md).

The `deadline` query parameter sets how long the trigger request waits before returning an error: a duration such as `3m`, `15min` or `2h`, or an ISO timestamp such as `2026-09-30T18:00:00.000Z`. Without it the request returns a `snapshot_id` at once.

### How to limit records per input

When running a discovery collection, you can cap the number of results returned per input. In the API, set `limit_per_input` in the request body, next to `input`.

With `"limit_per_input": 10`, each input returns at most 10 records.

## Troubleshooting

Scraper API HTTP errors and record errors are listed with their meaning on [Scraper API error codes](https://docs.brightdata.com/products/scrapers/error-codes.md).

### Getting a 429 Too Many Requests error?

You receive `429 Too Many Requests` when you exceed the concurrent job limit, 5,000 active jobs or snapshots. Bright Data applies a temporary IP block to any IP that collects 25 or more 429 responses within 5 minutes. A blocked IP is refused on every API request for one hour, then the block expires automatically. So a 429 is a signal to slow down, not to retry at once.

1. Stop sending new requests as soon as you receive a 429.
2. Wait before retrying: the number of seconds in the `Retry-After` header if present, otherwise exponential backoff of 2, 4, 8, 16 and 32 seconds with random jitter.
3. Stop after a fixed number of retries. If 429s keep coming, you are over the limit: pause and reduce concurrency, because backoff alone does not finish running jobs.

> [!WARNING]
>
> Ten or more 429 responses within 5 minutes means you are close to the 25-in-5-minutes block threshold. Reduce the request rate right away. Workers that share one outbound IP count toward the same threshold, so coordinate throttling across them.

```python Bad: immediate retry theme={null}
response = requests.post(api_url, json=data)

if response.status_code == 429:
    requests.post(api_url, json=data)  # retrying at once is what gets an IP blocked
```

```python Good: Retry-After, backoff and a retry limit theme={null}
import random
import time
import requests

MAX_RETRIES = 5


def post_with_backoff(api_url, headers, data):
    """Send a Scraper API request and retry HTTP 429 at most MAX_RETRIES times."""
    for attempt in range(MAX_RETRIES + 1):
        resp = requests.post(api_url, headers=headers, json=data)
        if resp.status_code != 429:
            return resp
        if attempt == MAX_RETRIES:
            break
        retry_after = resp.headers.get("Retry-After")
        if retry_after and retry_after.isdigit():
            wait_s = int(retry_after)
        else:
            wait_s = 2 ** (attempt + 1)  # 2, 4, 8, 16, 32 seconds
        time.sleep(wait_s + random.uniform(0, 1))  # jitter keeps workers from retrying in step
    raise RuntimeError("Still HTTP 429 after 5 retries. Pause and reduce concurrency.")
```

The function stops after five retries instead of looping forever. Do not restart it in a tight loop or run many copies of it from one IP, because their 429 responses add up toward the block threshold.

If your IP is already blocked, stop all requests from it and wait for the temporary IP block to expire after one hour, then resume at a lower rate. For critical impact, or a block that lasts longer than one hour, see [What do I do if my IP is already blocked?](https://docs.brightdata.com/products/scrapers/error-codes.md#what-do-i-do-if-my-ip-is-already-blocked)

### Snapshot status shows 'failed'?

Check that all input URLs are valid for the scraper you are calling. A LinkedIn URL sent to the Amazon `dataset_id` fails. Review the error details in the snapshot response or in the [Logs tab](https://brightdata.com/cp/scrapers) of your Bright Data dashboard.

### Results are incomplete or missing some URLs?

Some URLs may fail individually while the overall job succeeds. Check the snapshot response for any `errors` field. Retry failed URLs in a separate request.

## FAQ

### How many URLs can I send in one async request?

Async requests accept up to 1 GB of input data per job, which is tens of thousands of URLs. Bright Data recommends the synchronous `/scrape` endpoint only for a single input, because more inputs are less likely to finish within its 1-minute window.

### How are inputs processed across concurrent requests?

All inputs are processed in parallel. Processing capacity is shared equally across all running jobs for a given scraper.

Start time does not affect priority either, jobs are not processed in first-in-first-out or last-in-first-out order. A job started an hour ago and a job started a minute ago receive the same share of capacity.

### Do I need a different dataset ID for each platform?

Yes. The `dataset_id` query parameter selects both the platform and the endpoint, for example Amazon products versus Amazon reviews. The table above lists the primary ID per platform, and each platform's introduction page lists the rest.

### How long does a snapshot stay available?

Snapshots are downloadable for 30 days after the job completes. Deliver results to a webhook or cloud storage if you need them retained longer.

## Related

* [Deliver results to webhooks and cloud storage](https://docs.brightdata.com/products/scrapers/scrapers-library/data-delivery.md)
* [Understanding sync vs. async requests](https://docs.brightdata.com/concepts/sync-vs-async.md)
* [Scraper API error codes](https://docs.brightdata.com/products/scrapers/error-codes.md)
