This tool connects to Bright Data to enable your agent to crawl websites, search the web, and access structured data from platforms like LinkedIn, Amazon, and social media.

Bright Data’s tools provide robust web scraping capabilities with built-in CAPTCHA solving and bot detection avoidance, allowing you to reliably extract data from the web.

​
Why Use Bright Data With LlamaIndex?

The Bright Data tool provides the following capabilities:

​
How to Integrate Bright Data With LlamaIndex?

1

Obtain Your Bright Data API Key

2

Installation

Install the required packages:

pip install llama-index llama-index-core llama-index-tools-brightdata
3

Usage

Here’s an example of how to use the BrightDataToolSpec with LlamaIndex:

llm = OpenAI(model="gpt-4o", api_key="your-api-key")

brightdata_tool = BrightDataToolSpec(api_key="your-api-key", zone="unlocker")

tool_list = brightdata_tool.to_tool_list()

for tool in tool_list:
    tool.original_description = tool.metadata.description
    tool.metadata.description = "Bright Data web scraping tool"

agent = OpenAIAgent.from_tools(tools=tool_list, llm=llm)

query = (
    "Find and summarize the latest news about AI from major tech news sites"
)
tool_descriptions = "\n\n".join(
    [
        f"Tool Name: {tool.metadata.name}\nTool Description: {tool.original_description}"
        for tool in tool_list
    ]
)

query_with_descriptions = f"{tool_descriptions}\n\nQuery: {query}"

response = agent.chat(query_with_descriptions)
print(response)