# Python SDK Source: https://docs.brightdata.com/api-reference/SDK Install the Bright Data Python SDK (thousands of scrapers) to scrape URLs, search the web, run platform scrapers and connect Playwright to the Browser API. This guide shows how to install the Bright Data Python SDK and call every feature it exposes: URL scraping, search engines, platform-specific scrapers (LinkedIn, Amazon, Instagram, TikTok, YouTube, Reddit, Pinterest, ChatGPT, Perplexity, Digikey), Scraper Studio, datasets, and the Browser API. Bright Data Python SDK banner ## Installation and setup Install the package via pip: ```bash theme={null} pip install brightdata-sdk ``` ### Configuration You must provide your API key. You can find it in your [Bright Data Control Panel](https://brightdata.com/cp/api_keys). **Option 1: Environment variable (recommended)** ```bash theme={null} export BRIGHTDATA_API_KEY="YOUR_API_KEY" ``` The Python SDK looks for your API key in this order: the `token` argument, `BRIGHTDATA_API_TOKEN`, `BRIGHTDATA_API_KEY`, then the key stored by `brightdata login`. `BRIGHTDATA_API_KEY` is the variable the [Bright Data CLI](/products/cli/installation) reads, so one export covers both. If both variables are set, `BRIGHTDATA_API_TOKEN` wins. Versions before 2.5.0 read only `BRIGHTDATA_API_TOKEN`; upgrade with `pip install -U brightdata-sdk` or export that variable instead. **Option 2: Direct initialization** ```python theme={null} # Async client from brightdata import BrightDataClient async with BrightDataClient(token="YOUR_API_KEY") as client: ... # Sync client from brightdata import SyncBrightDataClient with SyncBrightDataClient(token="YOUR_API_KEY") as client: ... ``` ## Basic usage Use `SyncBrightDataClient` for simple scripts. Use `BrightDataClient` with asyncio for high-concurrency workloads. ```python Synchronous theme={null} from brightdata import SyncBrightDataClient with SyncBrightDataClient() as client: # Scrape a URL data = client.scrape_url("https://example.com") print(f"Result: {data.data}") # Search Google search = client.search.google(query="Bright Data") print(f"Found: {len(search.data)}") ``` ```python Asynchronous theme={null} import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient() as client: result = await client.scrape_url("https://example.com") print(result.data) search = await client.search.google(query="Bright Data") print(f"Found: {len(search.data)}") if __name__ == "__main__": asyncio.run(main()) ``` ## Launch scrapes and web searches ```python Search Engines theme={null} from brightdata import BrightDataClient client = BrightDataClient() # Google search results = client.search.google( query="best shoes of 2025", location="United States", language="en", num_results=20 ) # Bing search results = client.search.bing( query="python tutorial", location="United States" ) # Yandex search results = client.search.yandex( query="latest news", location="Germany" ) if results.success: print(f"Cost: ${results.cost:.4f}") print(f"Time: {results.elapsed_ms():.2f}ms") ``` ```python Web Scraping theme={null} from brightdata import BrightDataClient client = BrightDataClient() # Scrape single URL result = client.scrape_url("https://example.com") # Scrape multiple URLs concurrently urls = [ "https://example1.com", "https://example2.com", "https://example3.com" ] results = client.scrape_url(urls) for r in results: if r.success: print(f"Success: {r.data[:200]}...") r.save_to_file(f"result_{hash(r.data)}.json") ``` When working with multiple queries or URLs, requests are handled concurrently for optimal performance. ## Use platform-specific scrapers for structured data Extract structured data from Amazon, LinkedIn, Facebook, Instagram, TikTok, YouTube, Reddit, Pinterest, ChatGPT, Perplexity, and Digikey. ```python Amazon theme={null} # Async import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient(auto_create_zones=False) as client: products = await client.scrape.amazon.products(url="https://amazon.com/dp/B0CRMZHDG8") reviews = await client.scrape.amazon.reviews( url="https://amazon.com/dp/B0CRMZHDG8", pastDays=30, keyWord="quality" ) sellers = await client.scrape.amazon.sellers( url="https://amazon.com/sp?seller=AXXXXXXXXXXX" ) asyncio.run(main()) # Sync from brightdata import SyncBrightDataClient with SyncBrightDataClient(auto_create_zones=False) as client: products = client.scrape.amazon.products(url="https://amazon.com/dp/B0CRMZHDG8") reviews = client.scrape.amazon.reviews( url="https://amazon.com/dp/B0CRMZHDG8", pastDays=30, keyWord="quality" ) sellers = client.scrape.amazon.sellers( url="https://amazon.com/sp?seller=AXXXXXXXXXXX" ) ``` ```python LinkedIn theme={null} # Async: collect by URL, then discover by parameters import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient(auto_create_zones=False) as client: profiles = await client.scrape.linkedin.profiles( url="https://www.linkedin.com/in/johndoe" ) companies = await client.scrape.linkedin.companies( url="https://www.linkedin.com/company/bright-data" ) jobs = await client.scrape.linkedin.jobs( url="https://www.linkedin.com/jobs/view/123456" ) posts = await client.scrape.linkedin.posts( url="https://www.linkedin.com/feed/update/urn:li:activity:123" ) # Discover by parameters job_results = await client.search.linkedin.jobs( keyword="python developer", location="New York", remote=True ) profile_results = await client.search.linkedin.profiles( first_name="John", last_name="Doe" ) asyncio.run(main()) # Sync from brightdata import SyncBrightDataClient with SyncBrightDataClient(auto_create_zones=False) as client: profiles = client.scrape.linkedin.profiles(url="https://www.linkedin.com/in/johndoe") companies = client.scrape.linkedin.companies( url="https://www.linkedin.com/company/bright-data" ) jobs = client.scrape.linkedin.jobs(url="https://www.linkedin.com/jobs/view/123456") posts = client.scrape.linkedin.posts( url="https://www.linkedin.com/feed/update/urn:li:activity:123" ) ``` ```python Facebook & Instagram theme={null} # Async import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient(auto_create_zones=False) as client: await client.scrape.facebook.posts_by_profile( url="https://www.facebook.com/nasa", num_of_posts=10 ) await client.scrape.facebook.posts_by_group( url="https://www.facebook.com/groups/example" ) await client.scrape.facebook.posts_by_url( url="https://www.facebook.com/post/123456" ) await client.scrape.facebook.comments( url="https://www.facebook.com/post/123456", num_of_comments=100 ) await client.scrape.facebook.reels(url="https://www.facebook.com/nasa") await client.scrape.instagram.profiles(url="https://www.instagram.com/nasa/") await client.scrape.instagram.posts(url="https://www.instagram.com/p/Cuf4s0MNqNr") await client.scrape.instagram.reels( url="https://www.instagram.com/reel/C5Rdyj_q7YN/" ) await client.scrape.instagram.comments( url="https://www.instagram.com/p/CesFC7JLyFl/" ) asyncio.run(main()) # Sync from brightdata import SyncBrightDataClient with SyncBrightDataClient(auto_create_zones=False) as client: client.scrape.facebook.posts_by_profile( url="https://www.facebook.com/nasa", num_of_posts=10 ) client.scrape.instagram.profiles(url="https://www.instagram.com/nasa/") ``` ```python TikTok theme={null} # Async import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient(auto_create_zones=False) as client: profiles = await client.scrape.tiktok.profiles( url="https://www.tiktok.com/@username" ) posts = await client.scrape.tiktok.posts( url="https://www.tiktok.com/@user/video/7433494424040017194" ) comments = await client.scrape.tiktok.comments( url="https://www.tiktok.com/@user/video/7216019547806092550" ) fast_posts = await client.scrape.tiktok.posts_by_profile_fast( url="https://www.tiktok.com/@bbc" ) search = await client.scrape.tiktok.posts_by_search_url_fast( url="https://www.tiktok.com/search?q=cats", num_of_posts=10 ) asyncio.run(main()) ``` ```python YouTube theme={null} # Async import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient(auto_create_zones=False) as client: videos = await client.scrape.youtube.videos( url="https://www.youtube.com/watch?v=dQw4w9WgXcQ", transcription_language="English" ) channels = await client.scrape.youtube.channels( url="https://www.youtube.com/@MrBeast/about" ) comments = await client.scrape.youtube.comments( url="https://www.youtube.com/watch?v=dQw4w9WgXcQ", num_of_comments=100 ) asyncio.run(main()) ``` ```python Reddit theme={null} # Async import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient(auto_create_zones=False) as client: posts = await client.scrape.reddit.posts( url="https://www.reddit.com/r/python/comments/abc123/" ) by_keyword = await client.scrape.reddit.posts_by_keyword( keyword="machine learning", sort_by="Top", date="Past week" ) by_subreddit = await client.scrape.reddit.posts_by_subreddit( url="https://www.reddit.com/r/datascience/" ) comments = await client.scrape.reddit.comments( url="https://www.reddit.com/r/python/comments/abc123/", days_back=30 ) asyncio.run(main()) ``` ```python Pinterest theme={null} # Async import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient(auto_create_zones=False) as client: posts = await client.scrape.pinterest.posts( url="https://www.pinterest.com/pin/3166662230556591/" ) profiles = await client.scrape.pinterest.profiles( url="https://www.pinterest.com/boredpanda/" ) # Discover by parameters by_keyword = await client.search.pinterest.posts_by_keyword( keyword="spaghetti recipes" ) by_profile = await client.search.pinterest.posts_by_profile( url="https://www.pinterest.com/grandmapowpow/", num_of_posts=20 ) asyncio.run(main()) # Sync from brightdata import SyncBrightDataClient with SyncBrightDataClient(auto_create_zones=False) as client: posts = client.scrape.pinterest.posts( url="https://www.pinterest.com/pin/3166662230556591/" ) profiles = client.scrape.pinterest.profiles( url="https://www.pinterest.com/boredpanda/" ) ``` ```python ChatGPT theme={null} # Async import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient(auto_create_zones=False) as client: result = await client.scrape.chatgpt.prompt( "Explain async programming", web_search=True ) batch = await client.scrape.chatgpt.prompts( prompts=["Explain Python", "Explain JavaScript"] ) asyncio.run(main()) # Sync from brightdata import SyncBrightDataClient with SyncBrightDataClient(auto_create_zones=False) as client: result = client.scrape.chatgpt.prompt( "Explain async programming", web_search=True ) batch = client.scrape.chatgpt.prompts( prompts=["Explain Python", "Explain JavaScript"] ) ``` ```python Digikey & Perplexity theme={null} import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient(auto_create_zones=False) as client: parts = await client.scrape.digikey.products( url="https://www.digikey.com/en/products/detail/abc" ) answer = await client.scrape.perplexity.search( prompt="What is new in battery technology?" ) asyncio.run(main()) ``` ## Run your custom Scraper Studio scrapers ```python theme={null} # Async from brightdata import BrightDataClient async with BrightDataClient() as client: data = await client.scraper_studio.run( collector="c_abc123", input={"url": "https://example.com/product/1"} ) job = await client.scraper_studio.trigger( "c_abc123", {"url": "https://example.com/product/1"} ) data = await job.wait_and_fetch(timeout=120) status = await client.scraper_studio.status("j_abc123") # Sync from brightdata import SyncBrightDataClient with SyncBrightDataClient() as client: data = client.scraper_studio.run( collector="c_abc123", input={"url": "https://example.com/product/1"} ) status = client.scraper_studio.status("j_abc123") ``` ## Datasets API Access pre-collected data snapshots. ```python theme={null} from brightdata import SyncBrightDataClient with SyncBrightDataClient() as client: # 1. Request snapshot with filters print("Requesting snapshot...") snapshot_id = client.datasets.imdb_movies( filter={"name": "year", "operator": "=", "value": 2024}, records_limit=10 ) # 2. Download (SDK polls automatically) print(f"Snapshot {snapshot_id} ready. Downloading...") data = client.datasets.imdb_movies.download(snapshot_id) print(f"Downloaded {len(data)} records.") ``` In your IDE, hover over the `BrightDataClient` class or **any of its methods** to view available parameters, type hints, and usage examples. The SDK provides full IntelliSense support. ## Use dataclass payloads for type safety The SDK includes dataclass payloads with runtime validation and helper properties. ```python theme={null} from brightdata import BrightDataClient from brightdata.payloads import ( AmazonProductPayload, LinkedInJobSearchPayload, ChatGPTPromptPayload ) client = BrightDataClient() # Amazon product with validation amazon_payload = AmazonProductPayload( url="https://amazon.com/dp/B123456789", reviews_count=50 # Runtime validated. ) print(f"ASIN: {amazon_payload.asin}") # Helper property print(f"Domain: {amazon_payload.domain}") # LinkedIn job search linkedin_payload = LinkedInJobSearchPayload( keyword="python developer", location="San Francisco", remote=True ) print(f"Remote search: {linkedin_payload.is_remote_search}") # Use with client result = client.scrape.amazon.products(**amazon_payload.to_dict()) ``` ## Connect to scraping browser Connect Playwright to Bright Data's cloud browser via the Browser API. ```python theme={null} from brightdata import BrightDataClient from playwright.async_api import async_playwright client = BrightDataClient( browser_username="brd-customer-xxxx-zone-scraping_browser1", browser_password="YOUR_ZONE_PASSWORD", ) async with async_playwright() as pw: browser = await pw.chromium.connect_over_cdp(client.browser.get_connect_url()) page = await browser.new_page() await page.goto("https://example.com", timeout=120000) print(await page.content()) await browser.close() ``` ## Use the CLI tool The SDK includes a command-line interface for terminal usage. ```bash theme={null} # Search operations brightdata search google "python tutorial" --location "United States" brightdata search linkedin jobs --keyword "python developer" --remote # Scrape operations brightdata scrape amazon products "https://amazon.com/dp/B123" brightdata scrape linkedin profiles "https://linkedin.com/in/johndoe" # Generic web scraping brightdata scrape generic "https://example.com" --output-format pretty # Save results to file brightdata search google "AI news" --output-file results.json ``` ## Async usage for better performance For concurrent operations, pass a list of URLs to `scrape_url` and run inside an async context manager. ```python theme={null} import asyncio from brightdata import BrightDataClient async def main(): async with BrightDataClient() as client: results = await client.scrape_url([ "https://example1.com", "https://example2.com", "https://example3.com", ], mode="async", poll_timeout=180) for result in results: print(result.data) asyncio.run(main()) ``` ## Resources View source code, examples, and contribute 10+ working examples for all features Package listing and release history ## What's new | Feature | Description | | - | - | | **Scraper Studio** | `client.scraper_studio.run/trigger/status()`. Trigger and fetch your custom Scraper Studio scrapers. Available on both sync and async clients. | | **Browser API** | `client.browser.get_connect_url()`. Connect Playwright/Puppeteer to Bright Data's cloud browser. Replaces `client.connect_browser()`. | | **New scrapers** | TikTok, YouTube, Reddit, Pinterest, Digikey, Perplexity added to `client.scrape.*` | | **Platform search** | `client.search.linkedin` / `amazon` / `instagram` / `tiktok` / `youtube` / `pinterest` / `chatgpt`. Discover content by parameters, not just URLs. | | **126+ Datasets** | Full catalog via `client.datasets.*` with `.sample()` and `.download()` | | **Full sync parity** | `SyncBrightDataClient` now covers all features: scrapers, search, discover, scraper studio, browser | # JavaScript SDK Source: https://docs.brightdata.com/api-reference/SDK-JS Install and use the Bright Data JavaScript SDK (thousands of scrapers) to scrape URLs, search Google/Bing/Yandex and connect to the Browser API from Node.js. This guide shows how to install the Bright Data JavaScript SDK, scrape URLs, run searches, call platform-specific scrapers (LinkedIn, Amazon, Instagram, and more), and connect to the Browser API and Scraper Studio from Node.js. ## Install the package Open the terminal and run: ```bash theme={null} npm install @brightdata/sdk ``` In your code file, import the package and launch your first requests: ```javascript theme={null} import { bdclient } from '@brightdata/sdk'; const client = new bdclient({ apiKey: 'YOUR_API_KEY' }); const html = await client.scrapeUrl('https://example.com'); const google = await client.search.google('pizza restaurants'); console.log(google); await client.close(); ``` ## Launch scrapes and web searches ```javascript search(...) theme={null} import { bdclient } from '@brightdata/sdk'; const client = new bdclient({ apiKey: 'YOUR_API_KEY' }); const google = await client.search.google('best laptops 2026'); const bing = await client.search.bing('python tutorial'); const yandex = await client.search.yandex('AI news'); console.log({ google, bing, yandex }); await client.close(); ``` ```javascript scrape(...) theme={null} import { bdclient } from '@brightdata/sdk'; const client = new bdclient({ apiKey: 'YOUR_API_KEY' }); const html = await client.scrapeUrl('https://example.com'); const md = await client.scrapeUrl('https://example.com', { dataFormat: 'markdown', country: 'us', }); console.log({ html, md }); await client.close(); ``` ## Platform scrapers and datasets Collect data from popular platforms by URL, or discover content by parameters. ```javascript LinkedIn theme={null} import { bdclient } from '@brightdata/sdk'; const client = new bdclient({ apiKey: 'YOUR_API_KEY' }); // Collect by URL const profiles = await client.scrape.linkedin.collectProfiles([ 'https://www.linkedin.com/in/satyanadella/', ]); const companies = await client.scrape.linkedin.collectCompanies([ 'https://www.linkedin.com/company/bright-data', ]); const jobs = await client.scrape.linkedin.collectJobs([ 'https://www.linkedin.com/jobs/view/123456', ]); const posts = await client.scrape.linkedin.collectPosts([ 'https://www.linkedin.com/feed/update/urn:li:activity:123', ]); // Discover by parameters const byKeyword = await client.scrape.linkedin.discoverJobs( [{ keyword: 'python developer', location: 'New York', remote: true }], {} ); const byName = await client.scrape.linkedin.discoverProfiles( [{ firstName: 'John', lastName: 'Doe' }], {} ); const compPosts = await client.scrape.linkedin.discoverCompanyPosts( [{ url: 'https://www.linkedin.com/company/bright-data' }], {} ); const userPosts = await client.scrape.linkedin.discoverUserPosts( [{ url: 'https://www.linkedin.com/in/satyanadella' }], {} ); ``` ```javascript Amazon theme={null} import { bdclient } from '@brightdata/sdk'; const client = new bdclient({ apiKey: 'YOUR_API_KEY' }); // Collect by URL const products = await client.scrape.amazon.collectProducts( ['https://www.amazon.com/dp/B0D77BX8Y4'], {} ); const reviews = await client.scrape.amazon.collectReviews( ['https://www.amazon.com/dp/B0D77BX8Y4'], {} ); const sellers = await client.scrape.amazon.collectSellers( ['https://www.amazon.com/sp?seller=AXXX'], {} ); // Discover by parameters const byKeyword = await client.scrape.amazon.discoverProductsByKeyword( [{ keyword: 'wireless headphones' }], {} ); const byCategory = await client.scrape.amazon.discoverProductsByCategoryURL( ['https://amazon.com/s?category=electronics'], {} ); const byBSUrl = await client.scrape.amazon.discoverProductsByBestSellerURL( ['https://amazon.com/gp/bestsellers/electronics'], {} ); // One-call orchestrated (trigger, poll, and return) const result = await client.scrape.amazon.products([ 'https://www.amazon.com/dp/B0D77BX8Y4', ]); ``` ```javascript Instagram theme={null} const profiles = await client.scrape.instagram.collectProfiles([ 'https://www.instagram.com/natgeo/', ]); const posts = await client.scrape.instagram.collectPosts([ 'https://www.instagram.com/p/ABC123/', ]); const reels = await client.scrape.instagram.collectReels([ 'https://www.instagram.com/reel/XYZ789/', ]); const comments = await client.scrape.instagram.collectComments([ 'https://www.instagram.com/p/ABC123/', ]); ``` ```javascript Facebook theme={null} const profilePosts = await client.scrape.facebook.collectPostsByProfile([ 'https://www.facebook.com/nasa', ]); const groupPosts = await client.scrape.facebook.collectPostsByGroup([ 'https://www.facebook.com/groups/example', ]); const postByUrl = await client.scrape.facebook.collectPostsByUrl([ 'https://www.facebook.com/post/123456', ]); const comments = await client.scrape.facebook.collectComments([ 'https://www.facebook.com/post/123456', ]); const reels = await client.scrape.facebook.collectReels([ 'https://www.facebook.com/nasa', ]); ``` ```javascript TikTok theme={null} const profiles = await client.scrape.tiktok.collectProfiles([ 'https://www.tiktok.com/@tiktok', ]); const posts = await client.scrape.tiktok.collectPosts([ 'https://www.tiktok.com/@user/video/7433494424040017194', ]); const comments = await client.scrape.tiktok.collectComments([ 'https://www.tiktok.com/@user/video/7216019547806092550', ]); ``` ```javascript YouTube theme={null} const videos = await client.scrape.youtube.collectVideos([ 'https://www.youtube.com/watch?v=dQw4w9WgXcQ', ]); ``` ```javascript Reddit theme={null} const posts = await client.scrape.reddit.collectPosts([ 'https://www.reddit.com/r/technology/top/', ]); ``` ```javascript ChatGPT / Perplexity / Digikey / Pinterest theme={null} const chatgpt = await client.scrape.chatGPT.search([ 'What are the latest trends in AI?', ]); const perplexity = await client.scrape.perplexity.search([ 'Latest breakthroughs in battery tech?', ]); const parts = await client.scrape.digikey.collectProducts([ 'https://www.digikey.com/en/products/detail/abc', ]); const pins = await client.scrape.pinterest.collectPosts([ 'https://www.pinterest.com/pin/123456', ]); ``` ## Scraper Studio Run your custom Scraper Studio collectors from the SDK. ```javascript theme={null} // Run and get results in one call const results = await client.scraperStudio.run('c_your_collector_id', { input: { url: 'https://example.com/product/1' }, }); // results is [{ input, data, error, responseId, elapsedMs }] // Manual: trigger, wait, and fetch const job = await client.scraperStudio.trigger('c_your_collector_id', { url: 'https://example.com/product/1', }); const data = await job.waitAndFetch(); // Check status const status = await client.scraperStudio.status('j_abc123'); // status.status is 'queued' | 'running' | 'done' | 'failed' ``` ## Browser API Connect Playwright to Bright Data's cloud browser. ```javascript theme={null} import { bdclient } from '@brightdata/sdk'; import { chromium } from 'playwright'; const client = new bdclient({ apiKey: 'YOUR_API_KEY', browserUsername: 'brd-customer-xxxx-zone-scraping_browser1', browserPassword: 'YOUR_ZONE_PASSWORD', }); const browser = await chromium.connectOverCDP(client.browser.getConnectUrl()); const page = await browser.newPage(); await page.goto('https://example.com'); await browser.close(); await client.close(); ``` ## Datasets API Query and download from any of the 126+ Bright Data datasets. ```javascript theme={null} const ds = client.datasets; // Query, then download const snapshotId = await ds.instagramProfiles.query( { url: 'https://www.instagram.com/natgeo/' }, { records_limit: 10 } ); const rows = await ds.instagramProfiles.download(snapshotId); // Same pattern across the full catalog await ds.amazonProducts.query({ url: 'https://amazon.com/dp/B123' }); await ds.linkedinProfiles.query({ url: 'https://linkedin.com/in/johndoe' }); await ds.imdbMovies.query({}, { records_limit: 50 }); await ds.redditPosts.query({ subreddit: 'technology' }); await ds.glassdoorReviews.query({ company: 'Bright Data' }); // Get field metadata const meta = await ds.instagramProfiles.getMetadata(); ``` ## Client and method parameters | Parameter | Type | Description | Default | | - | - | - | - | | `apiKey` | `string` | Your API key (or `BRIGHTDATA_API_KEY` env var) | . | | `autoCreateZones` | `boolean` | Auto-create zones if they do not exist | `true` | | `webUnlockerZone` | `string` | Custom Web Unlocker zone name | . | | `serpZone` | `string` | Custom SERP zone name | . | | `browserUsername` | `string` | Browser API username (or `BRIGHTDATA_BROWSERAPI_USERNAME` env var) | . | | `browserPassword` | `string` | Browser API password (or `BRIGHTDATA_BROWSERAPI_PASSWORD` env var) | . | | `logLevel` | `string` | Log level | `'INFO'` | | `structuredLogging` | `boolean` | Use structured JSON logging | `true` | | `verbose` | `boolean` | Enable verbose logging | `false` | ```javascript theme={null} const client = new bdclient({ apiKey: 'YOUR_API_KEY', autoCreateZones: true, webUnlockerZone: 'unlocker_zone1', serpZone: 'serp_zone1', browserUsername: 'brd-customer-xxxx-zone-scraping_browser1', browserPassword: 'YOUR_ZONE_PASSWORD', logLevel: 'INFO', structuredLogging: true, verbose: false, }); ``` Search the web using the SERP API. | Name | Type | Description | Default | | - | - | - | - | | `query` | `string \| string[]` | Search query string or array of queries | . | | `options.searchEngine` | `'google' \| 'bing' \| 'yandex'` | Search engine | `'google'` | | `options.zone` | `string` | Zone identifier (auto-configured if null) | . | | `options.format` | `'json' \| 'raw'` | Response format | `'raw'` | | `options.method` | `string` | HTTP method | `'GET'` | | `options.country` | `string` | Two-letter country code | `''` | | `options.dataFormat` | `'markdown' \| 'screenshot' \| 'html'` | Returned content format | `'html'` | | `options.concurrency` | `number` | Max parallel workers | `10` | | `options.timeout` | `number (ms)` | Request timeout | `30000` | Scrape a single URL or list of URLs using the Web Unlocker API. | Name | Type | Description | Default | | - | - | - | - | | `url` | `string \| string[]` | Single URL string or array of URLs | . | | `options.zone` | `string` | Zone identifier (auto-configured if null) | . | | `options.format` | `'json' \| 'raw'` | Response format | `'raw'` | | `options.method` | `string` | HTTP method | `'GET'` | | `options.country` | `string` | Two-letter country code | `''` | | `options.dataFormat` | `'markdown' \| 'screenshot' \| 'html'` | Returned content format | `'html'` | | `options.concurrency` | `number` | Max parallel workers | `10` | | `options.timeout` | `number (ms)` | Request timeout | `30000` | | Name | Type | Description | Default | | - | - | - | - | | `content` | `any` | Content to save | . | | `options.filename` | `string` | Output filename (auto-generated if null) | . | | `options.format` | `'json' \| 'csv' \| 'txt'` | File format | . | ## Error handling Enable `VERBOSE` in the Client for advanced logging (see Client parameters). Use the `listZones()` function to check available zones. Create a [Bright Data](https://brightdata.com/) account and copy your API key. Go to [account settings](https://brightdata.com/cp/setting/users) and make sure your API key has **admin permissions**. ## Resources Visit the Bright Data SDK GitHub repository # Choose a delivery type on request level Source: https://docs.brightdata.com/api-reference/scraper-studio-api/Choose_a_delivery_type_on_request_level Pick a Bright Data Scraper Studio API collection mode: batch, async real-time or sync real-time, and match delivery preferences to each endpoint. Instead of creating duplicate scrapers for each delivery type, you can choose a delivery type per job using API. hero-image.png Scraper Studio supports three API collection modes: * **Batch collection**: runs one or more inputs in the background and returns a collection\_id. * **Asynchronous real-time collection**: runs one input in the background and returns a response\_id. * **Synchronous real-time collection**: runs one input and returns the collected data directly in the same API response, if the request completes within the timeout. The scraper’s **Delivery preferences** in the control panel affect which API modes can be used. A scraper configured for Batch delivery can still be triggered with the real-time API endpoints. However, a scraper configured for Real-time delivery cannot be triggered with the batch endpoint. Delivery preferences tab in the Scraper Studio control panel, showing the "Choose when to get the data" options: "On a job completion (batch)" selected, "Realtime (single request)" and "Split delivery to small batches" ## **Delivery preference compatibility** Before triggering a scraper by API, check its Delivery preferences in the control panel. | Delivery preference in control panel | Batch endpoint | Real-time endpoints | | :- | :- | :- | | Batch / On job completion | Supported | Supported | | Real-time | Not supported | Supported | If you call the batch endpoint for a scraper configured as real-time, the API returns an error: ```json theme={null} "error": "Cannot trigger a batch job with a real-time scraper. Use /trigger_immediate endpoint instead" ``` ## **Choose a collection mode** | Mode | Endpoint | Input shape | Response | Use when | | :- | :- | :- | :- | :- | | Batch | `POST /dca/trigger` | JSON array of input objects | collection\_id | You need to process one or more inputs in a background job. | | Async real-time | `POST /dca/trigger_immediate` | Single JSON input object | response\_id | You need to run one input in the background and retrieve results later. | | Sync real-time | `POST /dca/crawl` | Single JSON input object | Collected data, or 202 if timeout is reached | You need the result in the same API request. | ## **Batch collection** Use batch collection when you want to run one or more inputs in the background. Batch collection is best for: * Multiple URLs, keywords or input rows * Large jobs * Scheduled or recurring data collection * Jobs where results can be retrieved later Batch collection is available only when the scraper is not configured as Real-time in Delivery preferences. See [Trigger batch collection](/api-reference/scraper-studio-api/Trigger_a_scraper_for_batch_collection_method) for request parameters and examples. ## **Asynchronous real-time collection** Use asynchronous real-time collection when you want to run a single input in the background and retrieve the result later. Async real-time is best for: * One input per request * Low-latency workflows where the client should not keep the HTTP request open * Applications that prefer to trigger now and retrieve results later A scraper configured for Batch delivery in the control panel can still be triggered with the async real-time endpoint. See [Trigger async real-time collection](/api-reference/scraper-studio-api/initiate-a-realtime-job/async-realtime-job) for request parameters and examples. ## **Synchronous real-time collection** Use synchronous real-time collection when you want the API request to wait for the scraper result and return the collected data directly. Sync real-time is best for: * One input per request * Applications that need an immediate response * Scrapers that usually complete within the configured timeout A scraper configured for Batch delivery in the control panel can still be triggered with the sync real-time endpoint. Synchronous real-time requests use the timeout query parameter. The timeout must be between 25s and 50s. If the scraper finishes within the timeout, the API returns 200 OK with the collected data. If the scraper is still running when the timeout is reached, the API may return 202 Accepted with a response\_id. Use this response\_id with the real-time data endpoint to retrieve the result asynchronously. See [Trigger sync real-time collection](/api-reference/scraper-studio-api/initiate-a-realtime-job/sync-realtime-job) for request parameters and examples. ## **Delivery behavior** Collection mode controls how the scraper runs. Delivery preferences control default delivery behavior configured in the control panel. Depending on your scraper configuration, results can be retrieved by API or delivered to a configured destination, such as: * API download * Webhook * Amazon S3 * Google Cloud Storage * Azure Blob Storage * SFTP / FTP * Snowflake * Email To override the configured destination for a single batch request, pass the `deliver` query parameter on `POST /dca/trigger`, and use the `notify` parameter to send a notification when the job or delivery completes. See [request-level delivery and notifications](/api-reference/scraper-studio-api/Trigger_a_scraper_for_batch_collection_method#override-delivery-settings-for-one-request). **Important behavior:** * A scraper configured for Batch delivery can be triggered by either batch or real-time API endpoints. * A scraper configured for Real-time delivery can be triggered only by real-time API endpoints. * Batch endpoint calls return a collection\_id. * Async real-time endpoint calls return a response\_id. * Sync real-time endpoint calls return collected data directly, unless the timeout is reached. Batch responses begin with `j_****` and real-time responses begin with `d****` # Understanding the AI Scraper Studio APIs Source: https://docs.brightdata.com/api-reference/scraper-studio-api/ai-flow/overview Use Bright Data Scraper Studio AI Flow APIs to create scrapers with AI, monitor job progress and update existing scrapers with the Self-Healing tool. The AI Scraper Studio API is a set of endpoints that let you create Scraper Studio scrapers with AI and update existing scrapers using Self-Healing tool, without using the UI. This page explains which endpoint to call and in what order. Both flows are asynchronous: you start an AI job, then check its progress until it completes. ## Workflow 1: Create a new scraper with AI Use these endpoints when you want AI to generate a scraper (schema + code) from scratch for a target site. 1. Create the scraper entity
→ Create scraper template 2. Start the AI generation job (schema + code)
→ Trigger AI Flow to create code 3. Poll until the AI job finishes and returns the result
→ AI job progress ## Workflow 2: Update an existing scraper with Self-Healing Use these endpoints when you already have a scraper and want to fix or modify it using a prompt. 1. Start a Self-Healing refactor job
→ Trigger Self-Healing 2. Poll until the refactor job finishes or pauses for user input
→ Self-Healing job progress 3. If progress returns `status: "pending_answer"`, approve or reject the proposed diff
→ Resume Self-Healing job 4. After the scraper is ready, initiate collection
→ Initiate collection Once created or updated, run it using the standard Scraper Studio API initiation endpoints (manual trigger, schedule, queue, etc.). ## Workflow 3: Heal a scraper automatically with Auto Self-Healing Use this endpoint when you want Scraper Studio to start a healing job on its own whenever a scraper's success rate drops below a threshold you set. 1. Enable Auto Self-Healing and set the trigger rules
→ Enable Auto Self-Healing For what each trigger rule does, see [Scraper Studio Auto Self-Healing](/products/scraper-studio/auto-self-healing). **Not sure which to use?** * No scraper yet → [Workflow 1](#workflow-1-create-a-new-scraper-with-ai) * Scraper needs changes → [Workflow 2](#workflow-2-update-an-existing-scraper-with-self-healing) * Scraper should fix itself when it fails → [Workflow 3](#workflow-3-heal-a-scraper-automatically-with-auto-self-healing) **New to the API?** See: Getting started with the API # Ethical web scraping best practices Source: https://docs.brightdata.com/concepts/ethical-web-scraping 5 core principles of ethical web data collection, the legal risks of non-compliant scraping and a practical checklist for sustainable data pipelines. How do you build a data pipeline that won't get shut down by a lawsuit, a regulatory change, or a blocked IP range? As scraping lawsuits multiply and AI-specific regulation accelerates, how you collect data matters as much as what you collect. This guide covers the risks of non-compliant scraping, five core principles for ethical data collection, and a practical checklist you can apply today.