Clay integration
This article will provide you with information on integrating Clay to Bright Data web scrapes APIs, Clay is a powerful tool for managing and automating workflows. By integrating your Web Scraper API into Clay, you can:
-
Automate data scraping from targeted websites.
-
Process and generate insights on the collected data in Clay workflows.
-
Streamline the delivery of web-scraped data into other tools or services via Clay.
Prerequisites
Before you begin, ensure the following:
-
Access to Clay: You must have a Clay account with administrator privileges.
-
Web Scraper API Access: You should have your Bright Data API token including authentication details, the needed endpoints, and request/response structures in place.
Integration Steps
Step 1: Set Up the Web Scraper API
-
Choose the target website from our variety of Bright data API offerings
-
Pick the specific scraper you need
-
Update the desired list of Inputs via JSON or CSV
-
Enable the “Include errors report with the results” toggle button
-
Enable the “Deliver results to external storage” toggle OR the “Send to webhook” toggle button according to your preference
Step 2: Create an API Request Workflow in Clay
Clay allows you to create automated workflows using its user-friendly interface. To call your Web Scraper API, follow these steps:
-
Log in to Clay:
-
Log into your Clay account.
-
Navigate to the Workflows section.
-
-
Set Up a Trigger:
- Choose how you’d like to trigger the workflow (e.g., manually, scheduled, or based on an event in another service integrated with Clay).
-
Add an HTTP Request Action:
-
Add an HTTP request block to your workflow.
-
Select the HTTP method supported by your Web Scraper API (POST “Trigger a collection”).
-
-
Configure the Request:
-
In the HTTP request block, input the following:
-
URL: Enter the API endpoint (e.g.,https://api.brightdata.com/datasets/v3/trigger).
-
Headers: Include any required headers, such as:
- Body (if required): Add any parameters required in the body as JSON.
-
-
Test the Request:
- Send a test request using Clay’s testing tools to ensure that the Web Scraper API responds with the expected data.
Step 3: Process the Response in Clay
Once you’ve successfully connected to the Web Scraper API, you can process the response data:
-
Integrate with Other Services:
- Use Clay to send the data to a database, Google Sheets, email, or any other service.
-
Add Conditions/Logic:
- Create conditions or filters in your workflow. For example, only process items with a price above a certain threshold.
Step 4: Schedule or Deploy Your Workflow
Configure your workflow to run automatically based on your preferred schedule (e.g., every hour or daily) or trigger it manually when needed.