AI
How to Set Up Bright Data With Dify
A comprehensive web scraping and data extraction plugin powered by Bright Data’s enterprise-grade infrastructure with intelligent auto-detection. Supports 50+ platforms including Amazon, LinkedIn, Instagram, YouTube, and more.
Available Tools
Structured Data Feeds
Extract structured data from popular platforms:
- E-commerce: Amazon, eBay, Walmart, Best Buy, Etsy, Zara
- Social Media: Instagram, Facebook, TikTok, YouTube, X (Twitter)
- Professional: LinkedIn profiles, companies, jobs
- Business: Crunchbase, ZoomInfo
- Maps & Reviews: Google Maps, booking sites
- News: Reuters and other news sources
Scrape As Markdown
Convert any webpage into clean, readable markdown format perfect for:
- Content analysis
- Documentation extraction
- Article processing
Search Engine
Get search results from major search engines:
- Bing
- Yandex, etc.
Use Cases
- E-commerce Monitoring: Track product prices and availability
- Lead Generation: Extract business information from LinkedIn
- Content Research: Gather articles and news for analysis
- Market Research: Monitor competitor websites and social media
- SEO Analysis: Track search engine results and rankings
How to Integrate Bright Data With Dify
1
Requirements
- Python 3.11+
- Dify Plugin SDK
- Bright Data API key
2
Obtain Your Bright Data API Key
- Log in to your Bright Data dashboard.
- Go to Account Settings.
- Generate an API key if you haven’t already done so.
3
Download the Plugin
Download the latest plugin package: brightdata_plugin.difypkg
4
Install in Dify
- Go to Dify.ai → Plugins → Install Plugin
- Choose Choose Local Package File
- Upload the
brightdata_plugin.difypkg
file
5
Create Your First Workflow
- Go to Dify Studio → Workflow
- Add one of the Bright Data Web Scraper tools:
- Structured Data Feeds - Extract structured data from 20+ platforms
- Scrape As Markdown - Convert any webpage to clean markdown
- Search Engine - Get search results from Google, Bing, Yandex
- Enter your Bright Data API key when prompted
- You can connect an LLM node to process and summarize the scraped data
6
Example Workflow
see workflow in banner image
- START → Input: Product URL
- STRUCTURED DATA FEEDS → Extract product details
- LLM → Summarize into easy-to-read text
- END → Output: Clean product summary
Important tips
- Reference every stage of the workflow to the output of the previous stage
- Set a high character limit in input fields (for the URL input field choose the “short paragraph” var option)