How to Set Up Bright Data With xpander.ai

• xpander.ai account
• Bright Data API key

1. In your profile dashboard and press the “New Agent” button to add a new agent:

1. Choose an appropriate name for your agent. For example, if you want to create a web scraping agent, you can call it “Web Scraper Agent”.

2. Leave all other settings in the “General” tab as they are. The defaults are enough for a simple setup like this one. By default, xpander.ai will use OpenAI’s GPT-4o as the LLM model.

1. Go to the “Tools” tab on your agent’s page, then click the “Add tools” button:

2. Search for “bright data” on the right side panel and select the Bright Data integration:

The following modal will show up:

Fill it out as follows:

Once everything is filled in, press the “Save” button.

Now, you will be prompted to select the specific Bright Data tools you want to enable in your agent:

We recommend selecting all tools to unlock full web scraping capabilities. As of this writing, the available tools are:

• Start Data Collection Job by Dataset ID: Launches a scraping job for a specified dataset using the Web Scraper APIs.
• Execute Proxy Request by URL: Sends an HTTP request through Bright Data’s proxy network for accessing the content of any web page.
• Download Dataset Snapshot by ID: Downloads a snapshot of a dataset in various formats, passing the data to the AI.

Once you have selected the desired tools, click the “Add to agent” button in the bottom-right corner:

The “Tools” tab of your agent will now show the Bright Data connector with the tools you configured:

Notice that you can click on any tool to view or adjust its configuration.

Fantastic! Your AI agent is now fully integrated with Bright Data tools and ready to scrape the web.

Now that your agent has access to the Bright Data tools for web scraping, give it a custom system prompt. This tells the agent what it is and how it should operate.

To do this, click on the “Instructions” tab and paste something like the following into the “System prompt” textarea:

You are an AI agent capable of grounding your responses by scraping data from the web

For more specialized agents, you can also add custom rules and goals.

Amazing! Your xpander scraping agent is ready.

Click on the “Agent graph” button to view your current AI agent workflow:

You will see a single agent with access to the three configured Bright Data tools for web scraping.

Well done! All that is left is to test the agent and see it in action.

Go back to the “Tester Chat” tab and try out your agent with a prompt like this:

Search for top 3 headphones under $100 and provide me info from their PDP's

This instructs your web scraping agent to dynamically look online for the top 3 headphones priced under $100 and retrieve information directly from their product detail pages (PDPs).

As you can imagine, a standard LLM would be able to handle this kind of task without access to dedicated scraping tools like those provided by Bright Data.

Paste the prompt into the chat input and send it to your agent:

The agent uses the LLM and Bright Data tools to:

1. Perform a web search and find the top 3 headphones.
2. For each product, start a data collection job and download data from Amazon.
3. Summarize the information into a short, accurate response, complete with real-world links to the Amazon product detail pages.

If you expand one of the tool sections in the interface, you will see something like this:

This proves that, behind the scenes, the AI agent automatically detected which Bright Data tools to use to complete the task. In detail, it called them with the right parameters to fetch fresh scraped data (in this case, directly from Amazon product pages).