Deliver scraper results to webhooks and S3

Bright Data Scraper API jobs can push results straight to your infrastructure instead of waiting for you to download them. Add an endpoint parameter for webhook delivery, or configure a cloud destination once and have every job write to it. The setup is identical for all scrapers; only the dataset_id changes.

The examples use the LinkedIn profiles scraper (gd_l1viktl72bvl7bjuj0). Swap in the dataset_id for your platform from the async requests table.

Prerequisites

A Bright Data account with an active API key
Familiarity with the async request workflow
For webhooks, a publicly accessible HTTPS endpoint (or a testing tool like webhook.site)
For S3, an Amazon S3 bucket and permission to create IAM roles

Which delivery method should I use?

Method	Best for	Setup
Webhook	Event-driven pipelines, payloads under 1 GB	One query parameter, no configuration
Amazon S3	Large payloads, data lakes, retained history	One-time IAM role, then automatic
Google Cloud Storage, Azure, Snowflake, SFTP	Existing cloud data platforms	One-time credentials, see Delivery options

How to deliver results to a webhook

When you trigger an async collection with an endpoint URL, Bright Data sends a POST request to that URL with the scraped data once the job completes. No polling required.

Your app --> POST /trigger (with webhook URL) --> Bright Data scrapes --> POST to your webhook

Step 1: Set up a test webhook

For testing, use webhook.site to get a temporary public URL:

Open webhook.site in your browser
Copy the unique URL displayed (e.g., https://webhook.site/abc-123-def)
Keep the page open to monitor incoming requests

Step 2: Trigger a collection with the webhook URL

Add the endpoint query parameter to your async /trigger request:

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&endpoint=https://webhook.site/abc-123-def&uncompressed_webhook=true" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {"url": "https://www.linkedin.com/in/satyanadella"},
    {"url": "https://www.linkedin.com/in/jeffweiner08"}
  ]'

import requests

WEBHOOK_URL = "https://webhook.site/abc-123-def"

response = requests.post(
    "https://api.brightdata.com/datasets/v3/trigger",
    params={
        "dataset_id": "gd_l1viktl72bvl7bjuj0",
        "format": "json",
        "endpoint": WEBHOOK_URL,
        "uncompressed_webhook": "true",
    },
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json=[
        {"url": "https://www.linkedin.com/in/satyanadella"},
        {"url": "https://www.linkedin.com/in/jeffweiner08"},
    ],
)

print("Snapshot ID:", response.json()["snapshot_id"])

const WEBHOOK_URL = "https://webhook.site/abc-123-def";

const response = await fetch(
  `https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&endpoint=${encodeURIComponent(WEBHOOK_URL)}&uncompressed_webhook=true`,
  {
    method: "POST",
    headers: {
      "Authorization": "Bearer YOUR_API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify([
      { url: "https://www.linkedin.com/in/satyanadella" },
      { url: "https://www.linkedin.com/in/jeffweiner08" },
    ]),
  }
);

const data = await response.json();
console.log("Snapshot ID:", data.snapshot_id);

Key parameters:

Parameter	Description
`endpoint`	Your HTTP endpoint URL that receives the `POST` payload
`uncompressed_webhook`	Set to `true` to receive uncompressed JSON (default is gzip)
`format`	Output format: `json`, `ndjson`, or `csv`

Step 3: Verify delivery

Once the collection completes (typically 30-60 seconds for a few records), check your webhook.site page. You should see a POST request with the scraped data. The payload is the same JSON array you would receive from a direct API download:

[
  {
    "name": "Satya Nadella",
    "city": "Redmond",
    "country_code": "US",
    "current_company": { "name": "Microsoft" },
    "followers": 10842560
  },
  {
    "name": "Jeff Weiner",
    "city": "San Francisco Bay Area",
    "country_code": "US",
    "current_company": { "name": "Next Chapter" },
    "followers": 1200000
  }
]

How to handle webhooks in production

For production, point the endpoint URL to your own server endpoint.

server.js

const express = require("express");
const app = express();

app.use(express.json({ limit: "100mb" }));

app.post("/webhook/scraper", (req, res) => {
  const records = req.body;
  console.log(`Received ${records.length} records`);

  for (const record of records) {
    console.log(`- ${record.name}`);
  }

  res.status(200).json({ received: true });
});

app.listen(3000, () => console.log("Webhook server running on port 3000"));

server.py

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/webhook/scraper", methods=["POST"])
def handle_webhook():
    records = request.get_json()
    print(f"Received {len(records)} records")

    for record in records:
        print(f"- {record['name']}")

    return jsonify({"received": True}), 200

if __name__ == "__main__":
    app.run(port=3000)

Return a 200 status code within 30 seconds to acknowledge receipt. If your endpoint fails or times out, Bright Data retries delivery.

How to authenticate webhook deliveries

If your endpoint requires authentication, add the webhook_header_Authorization parameter:

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&endpoint=https://your-server.com/webhook&webhook_header_Authorization=Bearer+YOUR_SECRET_TOKEN" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://www.linkedin.com/in/satyanadella"}]'

Which IPs do webhooks come from?

If your server uses an IP allowlist, add the following 14 Bright Data webhook source IPs:

175.27.69
225.9.175
28.38.247
29.18.195
72.185.255
174.112.248
165.183.124
91.140.7
202.75.37
82.225.117
27.150.189
214.10.85
169.71.210
194.183.74

How to deliver results to Amazon S3

S3 delivery is configured once per scraper in the Control Panel. After setup, every job writes its results to your bucket automatically.

Step 1: Create the IAM role

Bright Data writes to your bucket by assuming a role in your AWS account. Create a role that trusts Bright Data’s account 422310177405, with your Bright Data customer ID as the external ID, and grant it s3:PutObject on your bucket. The full policy and trust-relationship JSON is in Delivery options.

Step 2: Configure the delivery destination

Navigate to your scraper configuration
Click the Delivery settings tab
Select Amazon S3 as the delivery destination
Enter your credentials:
- Bucket name: Your S3 bucket name
- Role ARN: The IAM role ARN from Step 1
- Region: Your S3 bucket region
- Path prefix (optional): A folder path within the bucket (e.g., linkedin/profiles/)
Select your preferred file format (JSON, NDJSON, or CSV)
Click Save

Step 3: Trigger a collection

Trigger an async collection as normal. Results are delivered to your S3 bucket without any extra parameters:

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {"url": "https://www.linkedin.com/in/satyanadella"},
    {"url": "https://www.linkedin.com/in/jeffweiner08"},
    {"url": "https://www.linkedin.com/in/rbranson"}
  ]'

Step 4: Verify delivery

Once the collection completes, check your S3 bucket for the delivered file:

aws s3 ls s3://your-bucket-name/linkedin/profiles/

You should see a file named with the snapshot ID (e.g., s_m1a2b3c4d5e6f7g8h.json). Download and inspect it:

aws s3 cp s3://your-bucket-name/linkedin/profiles/s_m1a2b3c4d5e6f7g8h.json ./results.json
cat results.json | python -m json.tool | head -20

You can also verify delivery status using the Monitor Delivery API.

Troubleshooting

Webhook not receiving data?

Verify the URL is publicly accessible (not localhost)
Check that your endpoint returns a 200 status code within 30 seconds
Verify the 14 webhook IPs above are allowlisted if you have firewall rules

Receiving compressed data?

If you omit uncompressed_webhook=true, data arrives gzip-compressed. Add uncompressed_webhook=true to your trigger URL, or decompress the payload on your server.

Payload too large for your server?

Large collections can produce payloads up to 1 GB. Set express.json({ limit: "100mb" }) in Express.js or equivalent in your framework. For very large datasets, use S3 delivery instead.

Files not appearing in S3?

Verify the IAM role ARN and external ID are correct
Check that the bucket policy allows s3:PutObject from Bright Data’s account
Ensure the bucket region matches your configuration
Review delivery status in the Bright Data dashboard under Logs

Access denied errors on S3?

Verify the trust policy on your IAM role includes Bright Data’s account (422310177405) and that your external ID matches your Bright Data customer ID, found in Account settings.

FAQ

Does delivery configuration differ per platform?

No. Webhook parameters and cloud destination setup are identical for every Bright Data scraper. Only the dataset_id in the trigger request changes.

Can I use a webhook and cloud storage at the same time?

Yes. Cloud delivery is configured per scraper in the Control Panel, and the endpoint parameter is set per request, so a single job can do both.

What happens if my webhook is down when a job finishes?

Bright Data retries delivery. If your endpoint stays unavailable, download the snapshot directly using the snapshot_id returned by the trigger call; snapshots stay available for 30 days.

Get started

Data extraction

Web access

AI training data

Proxy networks

Deliver scraper results to webhooks and S3

Prerequisites

Which delivery method should I use?

How to deliver results to a webhook

Step 1: Set up a test webhook

Step 2: Trigger a collection with the webhook URL

Step 3: Verify delivery

How to handle webhooks in production

How to authenticate webhook deliveries

Which IPs do webhooks come from?

How to deliver results to Amazon S3

Step 1: Create the IAM role

Step 2: Configure the delivery destination

Step 3: Trigger a collection

Step 4: Verify delivery

Troubleshooting

FAQ

Does delivery configuration differ per platform?

Can I use a webhook and cloud storage at the same time?

What happens if my webhook is down when a job finishes?

​Prerequisites

​Which delivery method should I use?

​How to deliver results to a webhook

​Step 1: Set up a test webhook

​Step 2: Trigger a collection with the webhook URL

​Step 3: Verify delivery

​How to handle webhooks in production

​How to authenticate webhook deliveries

​Which IPs do webhooks come from?

​How to deliver results to Amazon S3

​Step 1: Create the IAM role

​Step 2: Configure the delivery destination

​Step 3: Trigger a collection

​Step 4: Verify delivery

​Troubleshooting

​FAQ

​Does delivery configuration differ per platform?

​Can I use a webhook and cloud storage at the same time?

​What happens if my webhook is down when a job finishes?

​Related

Prerequisites

Which delivery method should I use?

How to deliver results to a webhook

Step 1: Set up a test webhook

Step 2: Trigger a collection with the webhook URL

Step 3: Verify delivery

How to handle webhooks in production

How to authenticate webhook deliveries

Which IPs do webhooks come from?

How to deliver results to Amazon S3

Step 1: Create the IAM role

Step 2: Configure the delivery destination

Step 3: Trigger a collection

Step 4: Verify delivery

Troubleshooting

FAQ

Does delivery configuration differ per platform?

Can I use a webhook and cloud storage at the same time?

What happens if my webhook is down when a job finishes?

Related