How to Set Up Bright Data With Dify
A comprehensive web scraping and data extraction plugin powered by Bright Data’s enterprise-grade infrastructure with intelligent auto-detection. Supports 50+ platforms including Amazon, LinkedIn, Instagram, YouTube, and more.
For the most up-to-date updates, please refer to this Repository
Available Tools
Structured Data Feeds
Extract structured data from popular platforms:
- E-commerce: Amazon, eBay, Walmart, Best Buy, Etsy, Zara
- Social Media: Instagram, Facebook, TikTok, YouTube, X (Twitter)
- Professional: LinkedIn profiles, companies, jobs
- Business: Crunchbase, ZoomInfo
- Maps & Reviews: Google Maps, booking sites
- News: Reuters and other news sources
Scrape As Markdown
Convert any webpage into clean, readable markdown format perfect for:
- Content analysis
- Documentation extraction
- Article processing
Search Engine
Get search results from major search engines:
- Bing
- Yandex, etc.
Use Cases
- E-commerce Monitoring: Track product prices and availability
- Lead Generation: Extract business information from LinkedIn
- Content Research: Gather articles and news for analysis
- Market Research: Monitor competitor websites and social media
- SEO Analysis: Track search engine results and rankings
How to Integrate Bright Data With Dify
Requirements
- Python 3.11+
- Dify Plugin SDK
- Bright Data API key
Obtain Your Bright Data API Key
- Log in to your Bright Data dashboard.
- Go to Account Settings.
- Generate an API key if you haven’t already done so.
Download the Plugin
Download the latest plugin package: brightdata_plugin.difypkg
Install in Dify
- Go to Dify.ai → Plugins → Install Plugin
- Choose Choose Local Package File
- Upload the
brightdata_plugin.difypkgfile
Create Your First Workflow
- Go to Dify Studio → Workflow
- Add one of the Bright Data Web Scraper tools:
- Structured Data Feeds - Extract structured data from 20+ platforms
- Scrape As Markdown - Convert any webpage to clean markdown
- Search Engine - Get search results from Google, Bing, Yandex
- Enter your Bright Data API key when prompted
- You can connect an LLM node to process and summarize the scraped data
Example Workflow
see workflow in banner image
Sample Use Case: Extract Amazon product information and create a summary
- START → Input: Product URL
- STRUCTURED DATA FEEDS → Extract product details
- LLM → Summarize into easy-to-read text
- END → Output: Clean product summary
Important tips
- Reference every stage of the workflow to the output of the previous stage
- Set a high character limit in input fields (for the URL input field choose the “short paragraph” var option)
Advanced Option: Use Bright Data MCP
While the Bright Data plugin for Dify uses hosted APIs, advanced users can also integrate directly with Bright Data MCP (Model Context Protocol) — a local server that exposes Bright Data’s full scraping and automation toolset via HTTP.
You can call MCP tools from Dify using custom HTTP requests or external service nodes to unlock advanced capabilities like browser automation, structured data extraction, and real-time scraping.