> ## Documentation Index > Fetch the complete documentation index at: https://docs.brightdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Deliver scraper results to webhooks and S3 > Push Bright Data Scraper API results to an HTTPS webhook or an Amazon S3 bucket on job completion. Covers trigger parameters, IAM setup and 14 webhook source IPs. Bright Data Scraper API jobs can push results straight to your infrastructure instead of waiting for you to download them. Add an `endpoint` parameter for webhook delivery, or configure a cloud destination once and have every job write to it. The setup is identical for all scrapers; only the `dataset_id` changes. The examples use the LinkedIn profiles scraper (`gd_l1viktl72bvl7bjuj0`). Swap in the `dataset_id` for your platform from the [async requests table](/products/scrapers/scrapers-library/async-requests#which-dataset-id-do-i-use). ## Prerequisites * A [Bright Data account](https://brightdata.com/cp/start) with an active API key * Familiarity with the [async request workflow](/products/scrapers/scrapers-library/async-requests) * For webhooks, a publicly accessible HTTPS endpoint (or a testing tool like [webhook.site](https://webhook.site)) * For S3, an Amazon S3 bucket and permission to create IAM roles ## Which delivery method should I use? | Method | Best for | Setup | | :------------------------------------------- | :------------------------------------------- | :------------------------------------------------------------------------------------------------- | | Webhook | Event-driven pipelines, payloads under 1 GB | One query parameter, no configuration | | Amazon S3 | Large payloads, data lakes, retained history | One-time IAM role, then automatic | | Google Cloud Storage, Azure, Snowflake, SFTP | Existing cloud data platforms | One-time credentials, see [Delivery options](/products/scrapers/scrapers-library/delivery-options) | ## How to deliver results to a webhook When you trigger an async collection with an `endpoint` URL, Bright Data sends a `POST` request to that URL with the scraped data once the job completes. No polling required. ```text theme={null} Your app --> POST /trigger (with webhook URL) --> Bright Data scrapes --> POST to your webhook ``` ### Step 1: Set up a test webhook For testing, use [webhook.site](https://webhook.site) to get a temporary public URL: 1. Open [webhook.site](https://webhook.site) in your browser 2. Copy the unique URL displayed (e.g., `https://webhook.site/abc-123-def`) 3. Keep the page open to monitor incoming requests ### Step 2: Trigger a collection with the webhook URL Add the `endpoint` query parameter to your async `/trigger` request: ```bash cURL theme={null} curl -X POST \ "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&endpoint=https://webhook.site/abc-123-def&uncompressed_webhook=true" \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '[ {"url": "https://www.linkedin.com/in/satyanadella"}, {"url": "https://www.linkedin.com/in/jeffweiner08"} ]' ``` ```python Python theme={null} import requests WEBHOOK_URL = "https://webhook.site/abc-123-def" response = requests.post( "https://api.brightdata.com/datasets/v3/trigger", params={ "dataset_id": "gd_l1viktl72bvl7bjuj0", "format": "json", "endpoint": WEBHOOK_URL, "uncompressed_webhook": "true", }, headers={ "Authorization": "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, json=[ {"url": "https://www.linkedin.com/in/satyanadella"}, {"url": "https://www.linkedin.com/in/jeffweiner08"}, ], ) print("Snapshot ID:", response.json()["snapshot_id"]) ``` ```javascript Node.js theme={null} const WEBHOOK_URL = "https://webhook.site/abc-123-def"; const response = await fetch( `https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&endpoint=${encodeURIComponent(WEBHOOK_URL)}&uncompressed_webhook=true`, { method: "POST", headers: { "Authorization": "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify([ { url: "https://www.linkedin.com/in/satyanadella" }, { url: "https://www.linkedin.com/in/jeffweiner08" }, ]), } ); const data = await response.json(); console.log("Snapshot ID:", data.snapshot_id); ``` Key parameters: | Parameter | Description | | :--------------------- | :----------------------------------------------------------- | | `endpoint` | Your HTTP endpoint URL that receives the `POST` payload | | `uncompressed_webhook` | Set to `true` to receive uncompressed JSON (default is gzip) | | `format` | Output format: `json`, `ndjson`, or `csv` | ### Step 3: Verify delivery Once the collection completes (typically 30-60 seconds for a few records), check your webhook.site page. You should see a `POST` request with the scraped data. The payload is the same JSON array you would receive from a direct API download: ```json theme={null} [ { "name": "Satya Nadella", "city": "Redmond", "country_code": "US", "current_company": { "name": "Microsoft" }, "followers": 10842560 }, { "name": "Jeff Weiner", "city": "San Francisco Bay Area", "country_code": "US", "current_company": { "name": "Next Chapter" }, "followers": 1200000 } ] ``` ### How to handle webhooks in production For production, point the `endpoint` URL to your own server endpoint. ```javascript server.js theme={null} const express = require("express"); const app = express(); app.use(express.json({ limit: "100mb" })); app.post("/webhook/scraper", (req, res) => { const records = req.body; console.log(`Received ${records.length} records`); for (const record of records) { console.log(`- ${record.name}`); } res.status(200).json({ received: true }); }); app.listen(3000, () => console.log("Webhook server running on port 3000")); ``` ```python server.py theme={null} from flask import Flask, request, jsonify app = Flask(__name__) @app.route("/webhook/scraper", methods=["POST"]) def handle_webhook(): records = request.get_json() print(f"Received {len(records)} records") for record in records: print(f"- {record['name']}") return jsonify({"received": True}), 200 if __name__ == "__main__": app.run(port=3000) ``` Return a `200` status code within 30 seconds to acknowledge receipt. If your endpoint fails or times out, Bright Data retries delivery. ### How to authenticate webhook deliveries If your endpoint requires authentication, add the `webhook_header_Authorization` parameter: ```bash theme={null} curl -X POST \ "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&endpoint=https://your-server.com/webhook&webhook_header_Authorization=Bearer+YOUR_SECRET_TOKEN" \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '[{"url": "https://www.linkedin.com/in/satyanadella"}]' ``` ### Which IPs do webhooks come from? If your server uses an IP allowlist, add the following 14 Bright Data webhook source IPs: ```text theme={null} 54.175.27.69 34.225.9.175 100.28.38.247 100.29.18.195 52.72.185.255 35.174.112.248 54.165.183.124 3.91.140.7 52.202.75.37 98.82.225.117 100.27.150.189 18.214.10.85 35.169.71.210 44.194.183.74 ``` ## How to deliver results to Amazon S3 S3 delivery is configured once per scraper in the Control Panel. After setup, every job writes its results to your bucket automatically. ### Step 1: Create the IAM role Bright Data writes to your bucket by assuming a role in your AWS account. Create a role that trusts Bright Data's account `422310177405`, with your Bright Data customer ID as the external ID, and grant it `s3:PutObject` on your bucket. The full policy and trust-relationship JSON is in [Delivery options](/products/scrapers/scrapers-library/delivery-options#aws-s3-user-role-permissions). ### Step 2: Configure the delivery destination 1. Navigate to your [scraper configuration](https://brightdata.com/cp/scrapers) 2. Click the **Delivery settings** tab 3. Select **Amazon S3** as the delivery destination 4. Enter your credentials: * **Bucket name**: Your S3 bucket name * **Role ARN**: The IAM role ARN from Step 1 * **Region**: Your S3 bucket region * **Path prefix** (optional): A folder path within the bucket (e.g., `linkedin/profiles/`) 5. Select your preferred file format (JSON, NDJSON, or CSV) 6. Click **Save** ### Step 3: Trigger a collection Trigger an async collection as normal. Results are delivered to your S3 bucket without any extra parameters: ```bash theme={null} curl -X POST \ "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '[ {"url": "https://www.linkedin.com/in/satyanadella"}, {"url": "https://www.linkedin.com/in/jeffweiner08"}, {"url": "https://www.linkedin.com/in/rbranson"} ]' ``` ### Step 4: Verify delivery Once the collection completes, check your S3 bucket for the delivered file: ```bash theme={null} aws s3 ls s3://your-bucket-name/linkedin/profiles/ ``` You should see a file named with the snapshot ID (e.g., `s_m1a2b3c4d5e6f7g8h.json`). Download and inspect it: ```bash theme={null} aws s3 cp s3://your-bucket-name/linkedin/profiles/s_m1a2b3c4d5e6f7g8h.json ./results.json cat results.json | python -m json.tool | head -20 ``` You can also verify delivery status using the [Monitor Delivery API](/api-reference/scrapers/management-apis/monitor-delivery). ## Troubleshooting * Verify the URL is publicly accessible (not `localhost`) * Check that your endpoint returns a `200` status code within 30 seconds * Verify the 14 webhook IPs above are allowlisted if you have firewall rules If you omit `uncompressed_webhook=true`, data arrives gzip-compressed. Add `uncompressed_webhook=true` to your trigger URL, or decompress the payload on your server. Large collections can produce payloads up to 1 GB. Set `express.json({ limit: "100mb" })` in Express.js or equivalent in your framework. For very large datasets, use S3 delivery instead. * Verify the IAM role ARN and external ID are correct * Check that the bucket policy allows `s3:PutObject` from Bright Data's account * Ensure the bucket region matches your configuration * Review delivery status in the Bright Data dashboard under **Logs** Verify the trust policy on your IAM role includes Bright Data's account (`422310177405`) and that your external ID matches your Bright Data customer ID, found in [Account settings](https://brightdata.com/cp/setting/customer_details). ## FAQ ### Does delivery configuration differ per platform? No. Webhook parameters and cloud destination setup are identical for every Bright Data scraper. Only the `dataset_id` in the trigger request changes. ### Can I use a webhook and cloud storage at the same time? Yes. Cloud delivery is configured per scraper in the Control Panel, and the `endpoint` parameter is set per request, so a single job can do both. ### What happens if my webhook is down when a job finishes? Bright Data retries delivery. If your endpoint stays unavailable, download the snapshot directly using the `snapshot_id` returned by the trigger call; snapshots stay available for 30 days. ## Related * [Async requests](/products/scrapers/scrapers-library/async-requests) * [Delivery options: GCS, Azure, Snowflake and SFTP](/products/scrapers/scrapers-library/delivery-options) * [Streamed and file delivery](/products/scrapers/scrapers-library/stream-and-file-delivery)