How to deliver Scraper API results

​

Prerequisites

• A Bright Data account with an active API key
• Familiarity with the async request workflow
• For webhooks, a publicly accessible HTTPS endpoint (or a testing tool like webhook.site)
• For a cloud destination, permission to create credentials in that account: an IAM role for Amazon S3, a service account key for Google Cloud, a user and role for Snowflake

​

Which destination should I use?

​

How to deliver results to a webhook

Your app --> POST /trigger (with webhook URL) --> Bright Data scrapes --> POST to your webhook

​

Step 1: Set up a test webhook

1. Open webhook.site in your browser
2. Copy the unique URL displayed (e.g., https://webhook.site/abc-123-def)
3. Keep the page open to monitor incoming requests

​

Step 2: Trigger a collection with the webhook URL

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&endpoint=https://webhook.site/abc-123-def&uncompressed_webhook=true" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {"url": "https://www.linkedin.com/in/satyanadella"},
    {"url": "https://www.linkedin.com/in/jeffweiner08"}
  ]'

import requests

WEBHOOK_URL = "https://webhook.site/abc-123-def"

response = requests.post(
    "https://api.brightdata.com/datasets/v3/trigger",
    params={
        "dataset_id": "gd_l1viktl72bvl7bjuj0",
        "format": "json",
        "endpoint": WEBHOOK_URL,
        "uncompressed_webhook": "true",
    },
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json=[
        {"url": "https://www.linkedin.com/in/satyanadella"},
        {"url": "https://www.linkedin.com/in/jeffweiner08"},
    ],
)

print("Snapshot ID:", response.json()["snapshot_id"])

const WEBHOOK_URL = "https://webhook.site/abc-123-def";

const response = await fetch(
  `https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&endpoint=${encodeURIComponent(WEBHOOK_URL)}&uncompressed_webhook=true`,
  {
    method: "POST",
    headers: {
      "Authorization": "Bearer YOUR_API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify([
      { url: "https://www.linkedin.com/in/satyanadella" },
      { url: "https://www.linkedin.com/in/jeffweiner08" },
    ]),
  }
);

const data = await response.json();
console.log("Snapshot ID:", data.snapshot_id);

​

Step 3: Verify delivery

[
  {
    "name": "Satya Nadella",
    "city": "Redmond",
    "country_code": "US",
    "current_company": { "name": "Microsoft" },
    "followers": 10842560
  },
  {
    "name": "Jeff Weiner",
    "city": "San Francisco Bay Area",
    "country_code": "US",
    "current_company": { "name": "Next Chapter" },
    "followers": 1200000
  }
]

​

How to handle webhooks in production

const express = require("express");
const app = express();

app.use(express.json({ limit: "100mb" }));

app.post("/webhook/scraper", (req, res) => {
  const records = req.body;
  console.log(`Received ${records.length} records`);

  for (const record of records) {
    console.log(`- ${record.name}`);
  }

  res.status(200).json({ received: true });
});

app.listen(3000, () => console.log("Webhook server running on port 3000"));

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/webhook/scraper", methods=["POST"])
def handle_webhook():
    records = request.get_json()
    print(f"Received {len(records)} records")

    for record in records:
        print(f"- {record['name']}")

    return jsonify({"received": True}), 200

if __name__ == "__main__":
    app.run(port=3000)

​

How to authenticate webhook deliveries

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&endpoint=https://your-server.com/webhook&auth_header=Bearer+YOUR_SECRET_TOKEN" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://www.linkedin.com/in/satyanadella"}]'

​

Which headers does a delivery carry?

​

Which IPs do webhooks come from?

54.175.27.69
34.225.9.175
100.28.38.247
100.29.18.195
52.72.185.255
35.174.112.248
54.165.183.124
3.91.140.7
52.202.75.37
98.82.225.117
100.27.150.189
18.214.10.85
35.169.71.210
44.194.183.74

​

How to deliver results to Amazon S3

​

Step 1: Create the IAM policy and role

1

Create a policy

In the IAM console, open Policies and create a policy that allows PutObject and GetObject on the bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": "arn:aws:s3:::NAME-OF-YOUR-BUCKET/*"
    }
  ]
}

2

Create a role

Open Roles, create a role, and attach the policy from step 1 under Permission policies. Note the role’s ARN; it looks like arn:aws:iam::<ROLE_ID>:role/<ROLE_NAME> and goes into the delivery settings.

The role’s trust policy lets Bright Data’s delivery role assume it. The external ID in the condition is the value you enter in the delivery settings:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::422310177405:role/brd.ec2.zs-dca-delivery"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "YOUR EXTERNAL ID"
        }
      }
    }
  ]
}

3

Attach the role to the bucket

On the bucket, grant the role from step 2 access by its ARN.

​

Step 2: Configure the delivery destination

1. Navigate to your scraper configuration
2. Click the Delivery settings tab
3. Select Amazon S3 as the delivery destination
4. Enter your credentials:
   • Bucket name: Your S3 bucket name
   • Role ARN: The IAM role ARN from Step 1
   • External ID: The value from the role’s trust policy
   • Region: Your S3 bucket region
   • Path prefix (optional): A folder path within the bucket (e.g., linkedin/profiles/)
5. Select your preferred file format (JSON, NDJSON, or CSV)
6. Click Save

​

Step 3: Trigger a collection

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {"url": "https://www.linkedin.com/in/satyanadella"},
    {"url": "https://www.linkedin.com/in/jeffweiner08"},
    {"url": "https://www.linkedin.com/in/rbranson"}
  ]'

​

Step 4: Verify delivery

aws s3 ls s3://your-bucket-name/linkedin/profiles/

aws s3 cp s3://your-bucket-name/linkedin/profiles/sd_m1a2b3c4d5e6f7g8h.json ./results.json
cat results.json | python -m json.tool | head -20

​

How to deliver results to Google Cloud Storage

1. Open the Google Cloud console, expand the menu and open IAM & Admin.
2. Click Service accounts.
   
3. Choose an existing service account or create one. If the button is missing, create a project first.
   
   
4. Enter the name, ID and description, grant the account access to the bucket, and create it.
   
5. Click the service account’s email address.
   
6. Open the Keys tab, click Add key, then Create new key.
   
7. Choose JSON. The key file downloads to your machine.
   
8. Copy the private_key value from the downloaded file into the scraper’s delivery settings, with the bucket name and the file format.
   

​

How to deliver results to Snowflake

CREATE DATABASE <database>;

CREATE SCHEMA <schema>;

CREATE WAREHOUSE <warehouse>;

CREATE STAGE <stage>;

CREATE ROLE <role_name>;
GRANT OPERATE ON WAREHOUSE <warehouse> TO ROLE <role_name>;
GRANT WRITE ON STAGE <stage> TO ROLE <role_name>;

CREATE USER <user_name>
  PASSWORD = '<password>'
  LOGIN_NAME = <login>
  MUST_CHANGE_PASSWORD = FALSE
  DISABLED = FALSE
  COMMENT = 'user for Bright Data to upload data directly into Snowflake';

GRANT ROLE <role_name> TO USER <user_name>;

ALTER NETWORK POLICY <policy_name>
SET ALLOWED_IP_LIST=(
  <existing_allowlisted_ips>,
  '35.169.71.210',
  '34.233.211.38',
  '44.194.183.74',
  '54.243.177.151');

​

Troubleshooting

​

Webhook not receiving data?

• Verify the URL is publicly accessible (not localhost)
• Check that your endpoint returns a 200 status code within 30 seconds
• Verify the 14 webhook IPs above are allowlisted if you have firewall rules

​

Receiving compressed data?

​

Payload too large for your server?

​

Files not appearing in S3?

• Verify the IAM role ARN and external ID are correct
• Check that the bucket policy allows s3:PutObject from Bright Data’s account
• Ensure the bucket region matches your configuration
• Review delivery status in the Bright Data dashboard under Logs

​

Access denied errors on S3?

​

FAQ

​

Does delivery configuration differ per platform?

​

Can I use a webhook and cloud storage at the same time?

​

What happens if my webhook is down when a job finishes?

​

Related

• How to scrape in bulk with async requests
• How to stream results and deliver files