Skip to main content
This guide shows you how to call the Bright Data Scraper API from a Clay workflow, so that Clay triggers a scrape on a schedule or an event and passes the records on to the tools your team already uses.

​
Prerequisites

  • A Clay account with administrator privileges.
  • A Bright Data API key from account settings.
  • The dataset ID of the scraper you want to run. Find it on the scraper’s page in the Control Panel; the Amazon products scraper is gd_l7q7dkf244hwjntr0.

​
Step 1: Prepare the scraper

  1. Open the scraper in the Control Panel and pick the endpoint you need.
  2. Load the inputs you plan to send, as JSON or CSV.
  3. Turn on Include errors report with the results, so a failed input is reported rather than silently dropped.
  4. Turn on Deliver results to external storage or Send to webhook, depending on where Clay should collect the data.

​
Step 2: Add the request to a Clay workflow

  1. In Clay, open Workflows and choose a trigger: manual, scheduled, or an event from another service connected to Clay.
  2. Add an HTTP request block.
  3. Set the method to POST and the URL to the trigger endpoint:
  4. Add the headers:
  5. Set the body to the inputs. A bare JSON array works for every scraper:
    To have the records pushed to a URL when the job finishes, add endpoint=https://your-server.com/webhook to the query string. See How to deliver results to a webhook or S3.
  6. Send a test request with Clay’s testing tools. The response is a snapshot_id; the records arrive at the delivery destination once the job finishes. The Trigger a collection reference lists every query parameter.

​
Step 3: Use the records in Clay

  • Send the records on to a database, Google Sheets, email or any other service Clay connects to.
  • Add conditions or filters, for example only records whose price is above a threshold.

​
Step 4: Schedule the workflow

Set the workflow to run on the schedule you need, hourly or daily, or leave it on the manual trigger.

Async requests

What the trigger endpoint returns and how to poll or download a snapshot.

Deliver results

Storage destinations, formats and the IAM setup for each.