Skip to main content
This guide shows you how to set up webhook delivery so your server receives scraped Reddit data automatically when a collection job finishes.

​
Prerequisites

​
How webhooks work

When you trigger an async collection with an endpoint URL, Bright Data sends a POST request to that URL with the scraped data once the job completes. No polling required. 
Your app --> POST /trigger (with webhook URL) --> Bright Data scrapes --> POST to your webhook

​
Step 1: Set up a test webhook

For testing, use webhook.site to get a temporary public URL:
  1. Open webhook.site in your browser
  2. Copy the unique URL displayed (e.g., https://webhook.site/abc-123-def)
  3. Keep the page open to monitor incoming requests

​
Step 2: Trigger a collection with the webhook URL

Add the endpoint query parameter to your async /trigger request: 
curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_lvz8ah06191smkebj4&format=json&endpoint=https://webhook.site/abc-123-def&uncompressed_webhook=true" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {"url": "https://www.reddit.com/r/learnpython/comments/1asdf12/"},
    {"url": "https://www.reddit.com/r/python/comments/1bsdf34/"}
  ]'

import requests

WEBHOOK_URL = "https://webhook.site/abc-123-def"

response = requests.post(
    "https://api.brightdata.com/datasets/v3/trigger",
    params={
        "dataset_id": "gd_lvz8ah06191smkebj4",
        "format": "json",
        "webhook": WEBHOOK_URL,
        "uncompressed_webhook": "true",
    },
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json=[
        {"url": "https://www.reddit.com/r/learnpython/comments/1asdf12/"},
        {"url": "https://www.reddit.com/r/python/comments/1bsdf34/"},
    ],
)

print("Snapshot ID:", response.json()["snapshot_id"])

const WEBHOOK_URL = "https://webhook.site/abc-123-def";

const response = await fetch(
  `https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_lvz8ah06191smkebj4&format=json&endpoint=${encodeURIComponent(WEBHOOK_URL)}&uncompressed_webhook=true`,
  {
    method: "POST",
    headers: {
      "Authorization": "Bearer YOUR_API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify([
      { url: "https://www.reddit.com/r/learnpython/comments/1asdf12/" },
      { url: "https://www.reddit.com/r/python/comments/1bsdf34/" },
    ]),
  }
);

const data = await response.json();
console.log("Snapshot ID:", data.snapshot_id);
Key parameters:
ParameterDescription
endpointYour HTTP endpoint URL that receives the POST payload
uncompressed_webhookSet to true to receive uncompressed JSON (default is gzip)
formatOutput format: json, ndjson or csv

​
Step 3: Verify delivery

Once the collection completes (typically 30 to 60 seconds for a few posts), check your webhook.site page. You should see a POST request with the scraped data. The payload is the same JSON array you would receive from a direct API download: 
[
  {
    "post_id": "1asdf12",
    "url": "https://www.reddit.com/r/learnpython/comments/1asdf12/",
    "title": "How do I start learning Python?",
    "num_upvotes": 1240,
    "num_comments": 86,
    "community_name": "learnpython"
  },
  {
    "post_id": "1bsdf34",
    "url": "https://www.reddit.com/r/python/comments/1bsdf34/",
    "title": "New Python 3.13 features",
    "num_upvotes": 540,
    "num_comments": 31,
    "community_name": "python"
  }
]

​
Production webhook setup

For production, point the endpoint URL to your own server endpoint.

​
How to handle webhooks in Express.js

server.js
const express = require("express");
const app = express();

app.use(express.json({ limit: "100mb" }));

app.post("/webhook/reddit", (req, res) => {
  const posts = req.body;
  console.log(`Received ${posts.length} posts`);

  for (const post of posts) {
    console.log(`- ${post.title} (r/${post.community_name})`);
  }

  res.status(200).json({ received: true });
});

app.listen(3000, () => console.log("Webhook server running on port 3000"));

​
How to handle webhooks in Flask

server.py
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/webhook/reddit", methods=["POST"])
def handle_webhook():
    posts = request.get_json()
    print(f"Received {len(posts)} posts")

    for post in posts:
        print(f"- {post['title']} (r/{post.get('community_name')})")

    return jsonify({"received": True}), 200

if __name__ == "__main__":
    app.run(port=3000)
Return a 200 status code within 30 seconds to acknowledge receipt. If your endpoint fails or times out, Bright Data retries delivery.

​
Webhook with authorization

If your endpoint requires authentication, add the webhook_header_Authorization parameter: 
curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_lvz8ah06191smkebj4&format=json&endpoint=https://your-server.com/webhook&webhook_header_Authorization=Bearer+YOUR_SECRET_TOKEN" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://www.reddit.com/r/learnpython/comments/1asdf12/"}]'

​
Allowlist webhook IPs

If your server uses an IP allowlist, add the following Bright Data webhook source IPs: 
54.175.27.69
34.225.9.175
100.28.38.247
100.29.18.195
52.72.185.255
35.174.112.248
54.165.183.124
3.91.140.7
52.202.75.37
98.82.225.117
100.27.150.189
18.214.10.85
35.169.71.210
44.194.183.74

​
Troubleshooting

  • Verify the URL is publicly accessible (not localhost)
  • Check that your endpoint returns a 200 status code within 30 seconds
  • Verify the webhook IPs above are allowlisted if you have firewall rules
If you omit uncompressed_webhook=true, data arrives gzip-compressed. Add uncompressed_webhook=true to your trigger URL, or decompress the payload on your server.
Large collections can produce payloads up to 1 GB. Set express.json({ limit: "100mb" }) in Express.js or equivalent in your framework. If you need to handle very large datasets, use S3 delivery instead.

​
Next steps

Deliver to Amazon S3

Store results directly in your S3 bucket.

All delivery options

Snowflake, Azure, GCS and more.