# Python SDK
Source: https://docs.brightdata.com/api-reference/SDK
Install the Bright Data Python SDK (thousands of scrapers) to scrape URLs, search the web, run platform scrapers and connect Playwright to the Browser API.
This guide shows how to install the Bright Data Python SDK and call every feature it exposes: URL scraping, search engines, platform-specific scrapers (LinkedIn, Amazon, Instagram, TikTok, YouTube, Reddit, Pinterest, ChatGPT, Perplexity, Digikey), Scraper Studio, datasets, and the Browser API.
## Installation and setup
Install the package via pip:
```bash theme={null}
pip install brightdata-sdk
```
### Configuration
You must provide your API key. You can find it in your [Bright Data Control Panel](https://brightdata.com/cp/api_keys).
**Option 1: Environment variable (recommended)**
```bash theme={null}
export BRIGHTDATA_API_KEY="YOUR_API_KEY"
```
Scraper Studio supports three API collection modes:
* **Batch collection**: runs one or more inputs in the background and returns a collection\_id.
* **Asynchronous real-time collection**: runs one input in the background and returns a response\_id.
* **Synchronous real-time collection**: runs one input and returns the collected data directly in the same API response, if the request completes within the timeout.
The scraper’s **Delivery preferences** in the control panel affect which API modes can be used. A scraper configured for Batch delivery can still be triggered with the real-time API endpoints. However, a scraper configured for Real-time delivery cannot be triggered with the batch endpoint.
## **Delivery preference compatibility**
Before triggering a scraper by API, check its Delivery preferences in the control panel.
| Delivery preference in control panel | Batch endpoint | Real-time endpoints |
| :- | :- | :- |
| Batch / On job completion | Supported | Supported |
| Real-time | Not supported | Supported |
If you call the batch endpoint for a scraper configured as real-time, the API returns an error:
```json theme={null}
"error": "Cannot trigger a batch job with a real-time scraper. Use /trigger_immediate endpoint instead"
```
## **Choose a collection mode**
| Mode | Endpoint | Input shape | Response | Use when |
| :- | :- | :- | :- | :- |
| Batch | `POST /dca/trigger` | JSON array of input objects | collection\_id | You need to process one or more inputs in a background job. |
| Async real-time | `POST /dca/trigger_immediate` | Single JSON input object | response\_id | You need to run one input in the background and retrieve results later. |
| Sync real-time | `POST /dca/crawl` | Single JSON input object | Collected data, or 202 if timeout is reached | You need the result in the same API request. |
## **Batch collection**
Use batch collection when you want to run one or more inputs in the background.
Batch collection is best for:
* Multiple URLs, keywords or input rows
* Large jobs
* Scheduled or recurring data collection
* Jobs where results can be retrieved later
Batch collection is available only when the scraper is not configured as Real-time in Delivery preferences.
See [Trigger batch collection](/api-reference/scraper-studio-api/Trigger_a_scraper_for_batch_collection_method) for request parameters and examples.
## **Asynchronous real-time collection**
Use asynchronous real-time collection when you want to run a single input in the background and retrieve the result later.
Async real-time is best for:
* One input per request
* Low-latency workflows where the client should not keep the HTTP request open
* Applications that prefer to trigger now and retrieve results later
A scraper configured for Batch delivery in the control panel can still be triggered with the async real-time endpoint.
See [Trigger async real-time collection](/api-reference/scraper-studio-api/initiate-a-realtime-job/async-realtime-job) for request parameters and examples.
## **Synchronous real-time collection**
Use synchronous real-time collection when you want the API request to wait for the scraper result and return the collected data directly.
Sync real-time is best for:
* One input per request
* Applications that need an immediate response
* Scrapers that usually complete within the configured timeout
A scraper configured for Batch delivery in the control panel can still be triggered with the sync real-time endpoint.
Synchronous real-time requests use the timeout query parameter. The timeout must be between 25s and 50s.
If the scraper finishes within the timeout, the API returns 200 OK with the collected data. If the scraper is still running when the timeout is reached, the API may return 202 Accepted with a response\_id. Use this response\_id with the real-time data endpoint to retrieve the result asynchronously.
See [Trigger sync real-time collection](/api-reference/scraper-studio-api/initiate-a-realtime-job/sync-realtime-job) for request parameters and examples.
## **Delivery behavior**
Collection mode controls how the scraper runs. Delivery preferences control default delivery behavior configured in the control panel.
Depending on your scraper configuration, results can be retrieved by API or delivered to a configured destination, such as:
* API download
* Webhook
* Amazon S3
* Google Cloud Storage
* Azure Blob Storage
* SFTP / FTP
* Snowflake
* Email
To override the configured destination for a single batch request, pass the `deliver` query parameter on `POST /dca/trigger`, and use the `notify` parameter to send a notification when the job or delivery completes. See [request-level delivery and notifications](/api-reference/scraper-studio-api/Trigger_a_scraper_for_batch_collection_method#override-delivery-settings-for-one-request).
**Important behavior:**
* A scraper configured for Batch delivery can be triggered by either batch or real-time API endpoints.
* A scraper configured for Real-time delivery can be triggered only by real-time API endpoints.
* Batch endpoint calls return a collection\_id.
* Async real-time endpoint calls return a response\_id.
* Sync real-time endpoint calls return collected data directly, unless the timeout is reached.
→
→
→
→
→
→
→
→
You are unable to access g2.com
``` ## Why maintaining anti-blocking yourself is costly Each detection layer requires a different countermeasure, and each countermeasure requires ongoing maintenance. IP rotation means sourcing, validating, and retiring proxies. TLS fingerprinting means patching your HTTP client to randomize handshake signatures, something most libraries don't support natively. Browser fingerprinting means keeping headless browser patches current as detection systems update weekly. CAPTCHA solving means integrating third-party solvers and handling failures. In practice, teams that build this in-house find that anti-blocking maintenance consumes a significant share of their engineering effort, often more time than building the actual data pipeline. When a target site updates its defenses, your scraper fails silently, returning empty fields or stale data while appearing to work normally. By the time you notice, your downstream systems have been ingesting bad data for hours or days. Every Bright Data scraping product shares the same anti-blocking engine. The difference between products is what you get back and how much control you keep. ## How Bright Data handles each detection layer **IP reputation.** Requests are routed through residential IPs from real ISPs across 195+ countries. Anti-bot systems treat these as regular household traffic. IPs are automatically rotated per request and flagged addresses are retired from the pool. **TLS fingerprinting.** When your HTTP client opens a connection, it sends a TLS Client Hello that anti-bot systems hash into a JA3 fingerprint. Python `requests`, for example, always produces the same hash, which is a known bot signature. Bright Data generates a unique TLS fingerprint per request, matching the diversity of real browser populations. **Browser fingerprinting.** Sites collect Canvas renders, WebGL data, installed fonts, screen dimensions, and audio context to build a device profile. Bright Data emulates complete, consistent browser environments that pass checks from Cloudflare Turnstile, Akamai Bot Manager, and other major detection systems. **Behavioral analysis.** Anti-bot systems track request timing, navigation sequences, and interaction patterns. Bright Data varies request timing, simulates realistic navigation patterns, and manages session state to match human browsing behavior. **CAPTCHA solving.** CAPTCHAs are solved automatically, including reCAPTCHA, hCaptcha, Cloudflare Turnstile, and others. You never see the challenge. The response arrives as if no CAPTCHA existed. ## Which product fits your situation Each product inherits the anti-blocking capabilities described above. The difference is what you get back and how much control you need. **I want full control over my HTTP client and just need IPs that won't get flagged.** Use [Bright Data Proxies](/left-over/proxy-networks-introduction). Four types are available: residential (highest trust, best for protected sites), datacenter (fastest, best for unprotected sites), ISP (residential trust at datacenter speed), and mobile (highest trust, best for heavily protected targets). Proxies solve IP reputation but leave the other four detection layers to you. **I need raw HTML from protected sites without building anti-blocking myself.** Use [Web Unlocker](/products/web-unlocker/introduction). Send a URL, get back clean HTML. All five detection layers are handled in a single API call: IP rotation, TLS fingerprinting, browser fingerprinting, behavioral emulation, and CAPTCHA solving. **I need to navigate to click through pagination, or interact with JavaScript-heavy pages.** Use [Bright Data Browser API](/products/scraping-browser/introduction). Full cloud-hosted browsers controlled through Puppeteer or Playwright with all anti-blocking built in. Unlike Web Unlocker, Browser API runs a real GUI browser (not headless), which produces authentic rendering artifacts that detection systems are far less likely to flag. **I need search engine results from Google, Bing, or other engines without getting blocked.** Use [SERP API](/products/serp-api/introduction). Search engines are among the most aggressively protected targets. SERP API handles all the anti-blocking and returns structured JSON with organic results, ads, featured snippets, and knowledge panels already parsed. **I want structured data from popular websites without writing or maintaining parsers.** Use [Web Scraper API](/products/scrapers/overview). Thousands of pre-built scrapers for sites like Amazon, LinkedIn, Instagram, YouTube, TikTok, and Google Maps. You get clean JSON with an average of 220+ data fields per scraper. When target sites change, Bright Data updates the scrapers. ## Common misconceptions **"Residential proxies alone solve blocking."** They handle IP reputation, but modern systems also check TLS fingerprints, browser fingerprints, and behavioral patterns. Proxies are one layer out of five. **"Stealth plugins make headless browsers undetectable."** Cloudflare Turnstile and Akamai Bot Manager detect patched headless browsers even with Playwright Stealth applied. Browser API solves this by running a real GUI browser rather than a patched headless one. **"Slowing down requests prevents blocks."** Rate limiting helps avoid basic IP bans, but sites like LinkedIn, Instagram, and Amazon use session-level fingerprinting that detects automation regardless of speed. **"Proxies and Web Unlocker do the same thing."** Proxies route your requests through different IPs only. Web Unlocker also manages TLS fingerprints, solves CAPTCHAs, emulates browser behavior, handles JavaScript rendering, and retries with fresh fingerprints on failure. ## FAQsScrape any site, buy ready-made datasets. Thousands of pre-built scrapers, one API key.
**5,000 free credits per month**, no credit card required. See the [free tier](/general/account/billing-and-pricing/free-tier).
Extract structured data, unblock pages and collect search results.
Search records and the live web by API.
Media and archived pages for model training.
Five network types for routing traffic at scale.
Tools that connect agents to the live web.
` | ISO country code for geo-targeting (e.g. `us`, `de`, `jp`) |
| `--zone ` | Web Unlocker zone name |
| `--mobile` | Use a mobile user agent |
| `--async` | Submit async, return a snapshot ID |
| `-o, --output ` | Write output to file |
| `--json` | Force JSON output |
| `--pretty` | Pretty-print JSON output |
```bash Markdown (default) theme={null}
brightdata scrape https://news.ycombinator.com
```
```bash HTML theme={null}
brightdata scrape https://example.com -f html
```
```bash JSON with geo-targeting theme={null}
brightdata scrape https://amazon.com -f json --country us -o product.json
```
```bash Screenshot theme={null}
brightdata scrape https://example.com -f screenshot -o page.png
```
```bash Async mode theme={null}
brightdata scrape https://example.com --async
```
```bash Pipe to markdown reader theme={null}
brightdata scrape https://docs.github.com | glow -
```
***
## `brightdata search `
Search Google, Bing, or Yandex via Bright Data's SERP API. Google returns structured JSON with organic results, ads, People Also Ask, and related searches. Bing and Yandex return markdown by default.
| Flag | Description |
| - | - |
| `--engine ` | `google` (default), `bing`, `yandex` |
| `--country ` | Localized results (e.g. `us`, `de`) |
| `--language ` | Language code (e.g. `en`, `fr`) |
| `--page ` | Page number, 0-indexed (default: `0`) |
| `--type ` | `web` (default), `news`, `images`, `shopping` |
| `--device ` | `desktop`, `mobile` |
| `--zone ` | SERP zone name |
| `-o, --output ` | Write output to file |
| `--json` | Force JSON output |
| `--pretty` | Pretty-print JSON output |
```bash Basic search theme={null}
brightdata search "typescript best practices"
```
```bash Localized search theme={null}
brightdata search "restaurants berlin" --country de --language de
```
```bash News search theme={null}
brightdata search "AI regulation" --type news
```
```bash Pagination theme={null}
brightdata search "web scraping" --page 1
```
```bash Extract URLs with jq theme={null}
brightdata search "open source scraping" --json | jq -r '.organic[].link'
```
```bash Bing search theme={null}
brightdata search "bright data pricing" --engine bing
```
***
## `brightdata pipelines [params...] [options]`
Extract structured data from 40+ platforms. Triggers an async collection job, polls until results are ready, and returns the data.
| Flag | Description |
| - | - |
| `--format ` | `json` (default), `csv`, `ndjson`, `jsonl` |
| `--timeout ` | Polling timeout (default: `600`) |
| `-o, --output ` | Write output to file |
| `--json` | Force JSON output |
| `--pretty` | Pretty-print JSON output |
```bash theme={null}
# List all available pipeline types
brightdata pipelines list
```
```bash LinkedIn profile theme={null}
brightdata pipelines linkedin_person_profile "https://linkedin.com/in/username"
```
```bash Amazon product theme={null}
brightdata pipelines amazon_product "https://amazon.com/dp/B09V3KXJPB"
```
```bash Amazon search theme={null}
brightdata pipelines amazon_product_search "laptop" "https://amazon.com"
```
```bash Instagram profile theme={null}
brightdata pipelines instagram_profiles "https://instagram.com/username"
```
```bash YouTube comments theme={null}
brightdata pipelines youtube_comments "https://youtube.com/watch?v=..." 50
```
```bash CSV export theme={null}
brightdata pipelines amazon_product "https://amazon.com/dp/..." --format csv -o product.csv
```
### Supported platforms
| Type | Platform | Parameters |
| - | - | - |
| `amazon_product` | Amazon product page | `` |
| `amazon_product_reviews` | Amazon reviews | `` |
| `amazon_product_search` | Amazon search results | ` ` |
| `walmart_product` | Walmart product page | `` |
| `walmart_seller` | Walmart seller profile | `` |
| `ebay_product` | eBay listing | `` |
| `bestbuy_products` | Best Buy | `` |
| `etsy_products` | Etsy | `` |
| `homedepot_products` | Home Depot | `` |
| `zara_products` | Zara | `` |
| `google_shopping` | Google Shopping | `` |
| Type | Platform | Parameters |
| - | - | - |
| `linkedin_person_profile` | LinkedIn person | `` |
| `linkedin_company_profile` | LinkedIn company | `` |
| `linkedin_job_listings` | LinkedIn jobs | `` |
| `linkedin_posts` | LinkedIn posts | `` |
| `linkedin_people_search` | LinkedIn people search | ` ` |
| `crunchbase_company` | Crunchbase | `` |
| `zoominfo_company_profile` | ZoomInfo | `` |
| Type | Platform | Parameters |
| - | - | - |
| `instagram_profiles` | Instagram profiles | `` |
| `instagram_posts` | Instagram posts | `` |
| `instagram_reels` | Instagram reels | `` |
| `instagram_comments` | Instagram comments | `` |
| `facebook_posts` | Facebook posts | `` |
| `facebook_marketplace_listings` | Facebook Marketplace | `` |
| `facebook_company_reviews` | Facebook reviews | ` [num_reviews]` |
| `facebook_events` | Facebook events | `` |
| `tiktok_profiles` | TikTok profiles | `` |
| `tiktok_posts` | TikTok posts | `` |
| `tiktok_shop` | TikTok shop | `` |
| `tiktok_comments` | TikTok comments | `` |
| `x_posts` | X (Twitter) posts | `` |
| `youtube_profiles` | YouTube channels | `` |
| `youtube_videos` | YouTube videos | `` |
| `youtube_comments` | YouTube comments | ` [num_comments]` |
| `reddit_posts` | Reddit posts | `` |
| Type | Platform | Parameters |
| - | - | - |
| `google_maps_reviews` | Google Maps reviews | ` [days_limit]` |
| `google_play_store` | Google Play | `` |
| `apple_app_store` | Apple App Store | `` |
| `github_repository_file` | GitHub repository files | `` |
| `yahoo_finance_business` | Yahoo Finance | `` |
| `zillow_properties_listing` | Zillow | `` |
| `booking_hotel_listings` | Booking.com | `` |
Run `brightdata pipelines list` in your terminal to see all available types at any time.
***
## `brightdata scraper`
Build, run, and maintain custom [Bright Data Scraper Studio](/products/scraper-studio/build-with-the-cli) scrapers from the terminal. Each scraper is identified by a Collector ID (a `c_*` string) that stays stable across runs and self-healing.
### `brightdata scraper create `
Build a scraper from a natural-language description using Bright Data's AI Agent. Returns a Collector ID.
AI generation typically takes 5 to 15 minutes and can run up to 25 minutes on complex targets.
| Flag | Description |
| - | - |
| `--name ` | Scraper template name (default: `cli-scraper-`) |
| `--deliver-webhook ` | Webhook URL for the deliver stub (default: `https://example.com/webhook`) |
| `--timeout ` | Polling timeout (default: `600`) |
| `--max-retries ` | Max retries on the AI-Flow concurrent-job 429 cap (default: `4`) |
| `--no-retry` | Fail immediately on 429 instead of waiting through the cap. Same as `--max-retries 0` |
| `--legacy-output` | Emit the bare AI-progress payload used before v0.3 instead of the envelope below |
| `-o, --output ` | Write output to file |
| `--json` / `--pretty` | JSON output (raw / indented) |
| `--timing` | Show request timing |
```bash theme={null}
# Minimal: a URL plus a description of the data you want
brightdata scraper create https://news.ycombinator.com \
"Extract top stories: title, url, points, author, comment count"
# Named scraper, with the JSON envelope saved for inspection
brightdata scraper create https://www.ycombinator.com/companies?batch=W26 \
"For each company card, extract name, vertical, tagline, link" \
--name yc-w26 --pretty -o create.json
```
#### JSON output from `scraper create`
With `--json`, `--pretty`, `-o` or any non-TTY stream, `scraper create` returns an envelope instead of the human summary:
| Field | Description |
| - | - |
| `collector_id` | Collector ID of the new scraper |
| `name` | Scraper template name |
| `status` | Job status when the response was written |
| `completed_steps` | AI generation steps finished so far |
| `view_url` | Scraper Studio URL for the new collector |
| `created_at` | Creation timestamp, when the API returns one |
| `error` | Failure reason, present only on failure |
`--legacy-output` replaces the envelope with the bare AI-progress payload used before v0.3. Use it only while migrating a script to the envelope.
#### Concurrent job cap on `scraper create`
Bright Data's AI Flow caps how many generation jobs run at once. When your account is at the cap, the API returns HTTP 429 and the CLI waits and retries, four times by default. Each wait grows exponentially with jitter, up to about four minutes between attempts. Raise or lower the count with `--max-retries`, or fail on the first 429 with `--no-retry`.
### `brightdata scraper run [url]`
Run a scraper on one or more URLs and return the data. The CLI tries real-time mode first and falls back to batch mode automatically when a run exceeds the real-time page-load limit.
| Flag | Description |
| - | - |
| `--urls ` | Comma-separated list of URLs, sent to `/dca/trigger` as a single batch |
| `--input-file ` | File of URLs: one per line, or a JSON array of strings, or a JSON array of `{"url": "..."}` objects. Blank lines and `#` comments are skipped |
| `--sync` | Use the synchronous `/dca/crawl` endpoint, which has a server-side 25 to 50 second cap. Single URL only |
| `--sync-timeout ` | Sync-mode server timeout, 25 to 50 (default: `50`) |
| `--timeout ` | Polling timeout in async mode (default: `600`; batch mode: `3600`) |
| `--name ` | Human-readable job name |
| `--version ` | Scraper version, for example `dev` |
| `-o, --output ` | Write output to file |
| `--json` / `--pretty` | JSON output (raw / indented) |
| `--timing` | Show request timing |
Pass the URL as an argument for a single page, or use `--urls` or `--input-file` to run a batch. Omit the URL argument when you use either batch flag.
```bash theme={null}
# One URL
brightdata scraper run c_mpohus372o5tmid1jk https://news.ycombinator.com --pretty
# Several URLs as one batch job
brightdata scraper run c_mpohus372o5tmid1jk \
--urls https://news.ycombinator.com,https://news.ycombinator.com/newest
# URLs from a file, one per line
brightdata scraper run c_mpohus372o5tmid1jk --input-file urls.txt
# Sync mode for a small, fast page
brightdata scraper run c_mpohus372o5tmid1jk https://news.ycombinator.com --sync
```
`--sync` cannot be combined with `--urls` or `--input-file`. The `/dca/crawl` endpoint accepts one URL at a time, so drop `--sync` to run a batch.
### `brightdata scraper heal `
Fix an existing scraper in place via AI self-healing. The Collector ID does not change. By default, `heal` stops at an approval gate and returns `status: "awaiting_approval"` with a `preview_result`.
| Flag | Description |
| - | - |
| `--url ` | Verify target woven into the next-step hint. Not sent to the heal call |
| `--auto-approve` | Approve automatically at the gate and poll through to `done` |
| `--timeout ` | Polling timeout (default: `600`) |
| `--max-retries ` | Max retries on the AI-Flow concurrent-job 429 cap (default: `4`) |
| `--no-retry` | Fail immediately on 429 instead of waiting through the cap |
| `-o, --output ` | Write output to file |
| `--json` / `--pretty` | JSON output (raw / indented) |
```bash theme={null}
brightdata scraper heal c_mpohus372o5tmid1jk \
"The price field returns null since the redesign. Re-capture price and currency." \
--url https://example.com/product/1
```
### `brightdata scraper approve `
Commit a heal that is awaiting approval, or reject it with `--reject`.
| Flag | Description |
| - | - |
| `--reject` | Reject the proposed fix instead of approving it |
| `--url ` | Verify target woven into the next-step hint on success |
| `--timeout ` | Polling timeout (default: `600`) |
| `-o, --output ` | Write output to file |
| `--json` / `--pretty` | JSON output |
```bash theme={null}
brightdata scraper approve c_mpohus372o5tmid1jk --url https://example.com/product/1
brightdata scraper approve c_mpohus372o5tmid1jk --reject
```
**Approving is not the same as saving.** A successful `approve` can return exit code 0 and a success message while the healed template is still a development draft, so the next live run serves the pre-heal version. Confirm the fix reached production before you rely on it: re-run the collector and check that the healed field is present in the output, or open the scraper in the Scraper Studio IDE and check it shows **Active** or **Ready** rather than **Draft**. See [Initiate collection and delivery](/products/scraper-studio/initiate-collection-and-delivery-options#how-do-i-save-a-scraper-to-production).
### Heal statuses
`heal` and `approve` report the job through these statuses. Poll until you reach `done` or `error`.
| Status | Meaning | What to do |
| - | - | - |
| `awaiting_approval` | The fix is generated and previewed, waiting on you | Inspect `preview_result`, then run `approve` |
| `pending_answer` | The job is parked on a human decision and the collector keeps serving the old template | Same as `awaiting_approval`: approve or reject. The job does not advance on its own |
| `done` | The heal completed | Verify a live run returns the healed field |
| `error` | The heal failed | Read the failure reason and retry with a more specific prompt |
A heal that reaches `done` confirms a value came back, not that it is the right value. Check the healed field against the live page before trusting it downstream.
The self-healing flow is run, inspect, `heal`, `approve`, re-run. See the [Scraper Studio CLI quickstart](/products/scraper-studio/build-with-the-cli) for a full walkthrough. `scraper heal` and `scraper approve` were added in CLI v0.3.1.
***
## `brightdata status `
Check the status of an async snapshot job (from `--async` scrapes or pipeline collections).
| Flag | Description |
| - | - |
| `--wait` | Poll until the job completes |
| `--timeout ` | Polling timeout (default: `600`) |
| `-o, --output ` | Write output to file |
| `--json` / `--pretty` | JSON output |
```bash theme={null}
brightdata status s_abc123xyz
brightdata status s_abc123xyz --wait --pretty
brightdata status s_abc123xyz --wait --timeout 300
```
***
## `brightdata browser`
Control a real browser session powered by Bright Data's [Scraping Browser](/products/scraping-browser/introduction). A lightweight local daemon holds the browser connection open between commands, giving you persistent state without reconnecting on every call.
```bash theme={null}
brightdata browser [options]
```
### Global flags
These flags work with every `browser` subcommand:
| Flag | Description |
| - | - |
| `--session ` | Session name for running multiple isolated sessions in parallel (default: `default`) |
| `--country ` | Geo-target by ISO country code. On `open`, changing country reconnects the browser |
| `--zone ` | Scraping Browser zone (default: `cli_browser`) |
| `--timeout ` | IPC command timeout in milliseconds (default: `30000`) |
| `--idle-timeout ` | Daemon auto-shutdown after idle (default: `600000` / 10 min) |
| `--json` / `--pretty` | JSON output |
| `-o, --output ` | Write output to file |
### Subcommands
Navigate to a URL. Starts the daemon and browser session automatically if not already running.
```bash theme={null}
brightdata browser open
brightdata browser open https://amazon.com --country us --session shop
```
| Flag | Description |
| - | - |
| `--country ` | Geo-targeting. Reconnects the browser if the country changes on an existing session |
| `--zone ` | Browser zone name |
| `--idle-timeout ` | Daemon idle timeout for this session |
Capture the page as a text accessibility tree. This is the primary way AI agents read page content - far more token-efficient than raw HTML.
Each interactive element gets a `ref` (e.g. `e1`, `e2`) that you pass to `click`, `type`, `fill`, and other interaction commands.
```bash theme={null}
brightdata browser snapshot
brightdata browser snapshot --compact # Interactive elements + ancestors only
brightdata browser snapshot --interactive # Interactive elements as a flat list
brightdata browser snapshot --depth 3 # Limit tree depth
brightdata browser snapshot --selector "main" # Scope to a CSS subtree
```
**Example output:**
```text theme={null}
Page: Example Domain
URL: https://example.com
- heading "Example Domain" [level=1]
- paragraph "This domain is for use in illustrative examples."
- link "More information..." [ref=e1]
```
| Flag | Description |
| - | - |
| `--compact` | Only interactive elements and their ancestors (70-90% fewer tokens) |
| `--interactive` | Only interactive elements, as a flat list |
| `--depth ` | Limit tree depth |
| `--selector ` | Scope snapshot to elements matching a CSS selector |
| `--wrap` | Wrap output in content boundaries (useful for AI agent prompt injection safety) |
Capture a PNG screenshot of the current viewport.
```bash theme={null}
brightdata browser screenshot
brightdata browser screenshot ./result.png
brightdata browser screenshot --full-page -o page.png
brightdata browser screenshot --base64
```
| Flag | Description |
| - | - |
| `[path]` | Where to save the PNG (default: temp directory) |
| `--full-page` | Capture the full scrollable page, not just the viewport |
| `--base64` | Output base64-encoded PNG data instead of saving to a file |
Interact with elements using their snapshot `ref` values.
```bash theme={null}
# Click an element
brightdata browser click e3
# Type text into a field (clears first by default)
brightdata browser type e5 "search query"
brightdata browser type e5 " more text" --append # Append to existing value
brightdata browser type e5 "search query" --submit # Press Enter after typing
# Fill a form field directly (no keyboard simulation)
brightdata browser fill e2 "user@example.com"
# Select a dropdown option by visible label
brightdata browser select e4 "United States"
# Check / uncheck a checkbox or radio button
brightdata browser check e7
brightdata browser uncheck e7
# Hover over an element
brightdata browser hover e2
```
| Flag (for `type`) | Description |
| - | - |
| `--append` | Append to existing value using key-by-key simulation |
| `--submit` | Press Enter after typing |
Scroll the viewport or scroll an element into view.
```bash theme={null}
brightdata browser scroll # Scroll down 300px (default)
brightdata browser scroll --direction up
brightdata browser scroll --direction down --distance 600
brightdata browser scroll --ref e10 # Scroll element into view
```
| Flag | Description |
| - | - |
| `--direction ` | `up`, `down`, `left`, `right` (default: `down`) |
| `--distance ` | Pixels to scroll (default: `300`) |
| `--ref ` | Scroll this element into view instead of the viewport |
Get text or HTML content from the page or a specific element.
```bash theme={null}
# Text content
brightdata browser get text # Full page text
brightdata browser get text "h1" # Text of the first h1
brightdata browser get text "#price" # Text inside #price
# HTML content
brightdata browser get html # Full page outer HTML
brightdata browser get html ".product" # innerHTML of .product
```
Inspect session state.
```bash theme={null}
# HTTP requests captured since last navigation
brightdata browser network
# Cookies for the active session
brightdata browser cookies
# Current session state
brightdata browser status
brightdata browser status --session shop --pretty
# List all active sessions
brightdata browser sessions
```
Navigation controls.
```bash theme={null}
brightdata browser back
brightdata browser forward
brightdata browser reload
```
Close a session and stop its daemon.
```bash theme={null}
brightdata browser close # Close the default session
brightdata browser close --session shop # Close a named session
brightdata browser close --all # Close all active sessions
```
Element `ref` values (e.g. `e1`, `e3`) are re-assigned on every `snapshot` call. After navigating or clicking, take a fresh snapshot before using refs again.
***
## `brightdata zones`
List and inspect Bright Data proxy zones.
```bash theme={null}
brightdata zones # List all active zones
brightdata zones info # Full details for a zone
brightdata zones --json -o zones.json # Export as JSON
brightdata zones info my_zone --pretty # Pretty-print zone info
```
***
## `brightdata budget`
View account balance and per-zone cost/bandwidth. Read-only.
| Subcommand | Description |
| - | - |
| *(none)* | Quick account balance |
| `balance` | Balance + pending charges |
| `zones` | Cost & bandwidth table for all zones |
| `zone ` | Detailed cost & bandwidth for one zone |
| Flag | Description |
| - | - |
| `--from ` | Start of date range (e.g. `2024-01-01T00:00:00`) |
| `--to ` | End of date range |
| `--json` / `--pretty` | JSON output |
```bash theme={null}
brightdata budget
brightdata budget balance
brightdata budget zones
brightdata budget zone my_zone
brightdata budget zones --from 2024-01-01T00:00:00 --to 2024-02-01T00:00:00
```
***
## `brightdata config`
View and manage CLI configuration.
| Subcommand | Description |
| - | - |
| *(none)* | Show all config |
| `get ` | Get a single value |
| `set ` | Set a value |
| Config Key | Description |
| - | - |
| `default_zone_unlocker` | Default zone for `scrape` and `search` |
| `default_zone_serp` | Override zone for `search` only |
| `default_format` | Default output format: `markdown` or `json` |
| `api_url` | Override API base URL |
```bash theme={null}
brightdata config
brightdata config set default_zone_unlocker my_zone
brightdata config set default_format json
brightdata config get default_zone_unlocker
```
***
## `brightdata init`
Interactive setup wizard. Walks through authentication, zone selection, and default configuration.
| Flag | Description |
| - | - |
| `--skip-auth` | Skip the authentication step |
| `-k, --api-key ` | Provide API key directly |
```bash theme={null}
brightdata init
```
***
## `brightdata skill`
Install Bright Data AI agent skills into coding agents (Claude Code, Cursor, Copilot, etc.).
| Subcommand | Description |
| - | - |
| `add` | Interactive picker - choose skills + target agents |
| `add ` | Install a specific skill directly |
| `list` | List all available skills |
Available skills: `search`, `scrape`, `data-feeds`, `bright-data-mcp`, `bright-data-best-practices`
```bash theme={null}
brightdata skill add # Interactive picker
brightdata skill add scrape # Direct install
brightdata skill list # See what's available
```
***
## `brightdata add mcp`
Add the Bright Data MCP server to Claude Code, Cursor, or Codex. Uses the API key stored by `brightdata login`.
```bash theme={null}
brightdata add mcp # Interactive agent + scope prompts
brightdata add mcp --agent claude-code --global
brightdata add mcp --agent claude-code,cursor --project
brightdata add mcp --agent codex --global
```
| Flag | Description |
| - | - |
| `--agent ` | Comma-separated targets: `claude-code`, `cursor`, `codex` |
| `--global` | Install to the agent's global config file |
| `--project` | Install to the current project's config file |
### Config file locations
| Agent | Global path | Project path |
| - | - | - |
| Claude Code | `~/.claude.json` | `.claude/settings.json` |
| Cursor | `~/.cursor/mcp.json` | `.cursor/mcp.json` |
| Codex | `$CODEX_HOME/mcp.json` or `~/.codex/mcp.json` | Not supported |
The command writes the MCP server entry under `mcpServers["bright-data"]`. Existing config is preserved - only the `bright-data` key is added or replaced.
`brightdata add mcp` uses the API key stored by `brightdata login`. It does not read `BRIGHTDATA_API_KEY` or the `--api-key` flag, so run `brightdata login` first.
# Usage examples
Source: https://docs.brightdata.com/products/cli/examples
Real-world workflows and recipes for the Bright Data CLI, from 1-line scrapes returning clean markdown to scheduled pipelines feeding LLMs.
## How to scrape with the CLI
### Get clean content from any website
```bash theme={null}
# Clean markdown - great for reading or feeding to LLMs
brightdata scrape https://news.ycombinator.com
# Save to file
brightdata scrape https://docs.python.org/3/tutorial/ -o python-tutorial.md
# Get raw HTML for parsing
brightdata scrape https://example.com -f html -o page.html
```
### How to scrape with geo-targeting
```bash theme={null}
# See Amazon prices as a US customer
brightdata scrape https://amazon.com/dp/B09V3KXJPB --country us
# Scrape a German news site from Germany
brightdata scrape https://spiegel.de --country de
# Mobile user agent for mobile-optimized pages
brightdata scrape https://example.com --mobile
```
### How to take screenshots
```bash theme={null}
# Full-page screenshot
brightdata scrape https://example.com -f screenshot -o homepage.png
# Screenshot from a specific country
brightdata scrape https://amazon.co.uk -f screenshot --country gb -o uk-amazon.png
```
***
## How to run search workflows
### Basic web search
```bash theme={null}
# Google search with formatted table output
brightdata search "best web scraping tools 2025"
# Get raw JSON for processing
brightdata search "typescript best practices" --json
# Pretty-print for inspection
brightdata search "AI startups" --pretty
```
### Localized and specialized search
```bash theme={null}
# Local restaurant search from Germany, in German
brightdata search "restaurants berlin" --country de --language de
# News-only results
brightdata search "AI regulation 2025" --type news
# Shopping results
brightdata search "wireless headphones" --type shopping
# Image search
brightdata search "mountain landscape wallpaper" --type images
```
### Pagination
```bash theme={null}
# First page (default)
brightdata search "web scraping tutorials"
# Second page
brightdata search "web scraping tutorials" --page 1
# Third page
brightdata search "web scraping tutorials" --page 2
```
***
## How to run browser workflows
### Navigate and read pages
```bash theme={null}
# Open a page and read its content
brightdata browser open https://example.com
brightdata browser snapshot
# Compact snapshot for AI agents (70-90% fewer tokens)
brightdata browser snapshot --compact
# Scope to a specific section
brightdata browser snapshot --selector "main"
```
### Interact with pages
```bash theme={null}
# Open a page
brightdata browser open https://example.com
# Read the page structure
brightdata browser snapshot --compact
# Click, type, and submit using refs from the snapshot
brightdata browser click e3
brightdata browser type e5 "search query" --submit
# Get updated snapshot after interaction
brightdata browser snapshot --compact
# Take a screenshot for visual verification
brightdata browser screenshot ./result.png
# Clean up
brightdata browser close
```
### How to compare multiple sessions
```bash theme={null}
# Open the same page from different countries
brightdata browser open https://amazon.com --session us --country us
brightdata browser open https://amazon.com --session de --country de
# Capture both snapshots
brightdata browser snapshot --session us --json > us.json
brightdata browser snapshot --session de --json > de.json
# Close all sessions
brightdata browser close --all
```
### Extract content from dynamic pages
```bash theme={null}
# Navigate to a page that requires JavaScript
brightdata browser open https://example.com/dashboard
# Get text content from specific elements
brightdata browser get text "#total-revenue"
brightdata browser get text ".summary-table"
# Get HTML for parsing
brightdata browser get html ".product-grid"
# Check network requests
brightdata browser network
```
***
## Structured data extraction
### How to extract e-commerce data
```bash theme={null}
# Amazon product details
brightdata pipelines amazon_product "https://amazon.com/dp/B09V3KXJPB"
# Amazon product reviews
brightdata pipelines amazon_product_reviews "https://amazon.com/dp/B09V3KXJPB"
# Amazon search - requires keyword + domain
brightdata pipelines amazon_product_search "wireless headphones" "https://amazon.com"
# Walmart product
brightdata pipelines walmart_product "https://walmart.com/ip/123456"
# Export as CSV
brightdata pipelines amazon_product "https://amazon.com/dp/B09V3KXJPB" --format csv -o product.csv
```
### Social media profiles
```bash theme={null}
# LinkedIn person
brightdata pipelines linkedin_person_profile "https://linkedin.com/in/username"
# LinkedIn company
brightdata pipelines linkedin_company_profile "https://linkedin.com/company/bright-data"
# Instagram profile
brightdata pipelines instagram_profiles "https://instagram.com/username"
# TikTok profile
brightdata pipelines tiktok_profiles "https://tiktok.com/@username"
```
### Reviews and comments
```bash theme={null}
# Google Maps reviews - last 7 days
brightdata pipelines google_maps_reviews "https://maps.google.com/maps/place/..." 7
# YouTube comments - top 50
brightdata pipelines youtube_comments "https://youtube.com/watch?v=dQw4w9WgXcQ" 50
# Facebook company reviews - 25 reviews
brightdata pipelines facebook_company_reviews "https://facebook.com/company" 25
# Instagram comments
brightdata pipelines instagram_comments "https://instagram.com/p/ABC123"
```
***
## Piping and automation
The CLI is designed to be pipe-friendly. When stdout is not a TTY, colors and spinners are automatically disabled.
### Chain search → scrape
```bash theme={null}
# Search Google, extract the first URL, then scrape it
brightdata search "best python frameworks 2025" --json \
| jq -r '.organic[0].link' \
| xargs brightdata scrape
```
### Scrape and read in terminal
```bash theme={null}
# Pipe markdown output to a terminal reader
brightdata scrape https://docs.github.com | glow -
# Or use less
brightdata scrape https://docs.github.com | less
```
### Export to CSV for analysis
```bash theme={null}
# Amazon product data to CSV
brightdata pipelines amazon_product "https://amazon.com/dp/B09V3KXJPB" --format csv > product.csv
# LinkedIn jobs to CSV
brightdata pipelines linkedin_job_listings "https://linkedin.com/jobs/view/123" --format csv -o jobs.csv
```
### Extract specific fields with jq
```bash theme={null}
# Get just the titles and prices from Amazon search
brightdata pipelines amazon_product_search "laptop" "https://amazon.com" \
| jq '[.[] | {title, price: .final_price}]'
# Get just organic result URLs from search
brightdata search "web scraping" --json | jq -r '.organic[].link'
```
### Async jobs for heavy workloads
```bash theme={null}
# Submit an async scrape
JOB_ID=$(brightdata scrape https://heavy-page.com --async --json | jq -r '.snapshot_id')
# Do other work...
# Check back later
brightdata status $JOB_ID --wait --pretty
```
***
## How to manage your account
### Monitor costs
```bash theme={null}
# Quick balance check
brightdata budget
# Detailed balance with pending charges
brightdata budget balance
# Cost breakdown by zone
brightdata budget zones
# Specific zone in a date range
brightdata budget zone cli_unlocker --from 2024-01-01T00:00:00 --to 2024-02-01T00:00:00
```
### Manage configuration
```bash theme={null}
# View current config
brightdata config
# Set default output to JSON
brightdata config set default_format json
# Use a custom zone for scraping
brightdata config set default_zone_unlocker my_custom_zone
# Override SERP zone
brightdata config set default_zone_serp my_serp_zone
```
***
## AI agent integration
### Add the MCP server to your coding agent
```bash theme={null}
# Interactive - choose agent and scope
brightdata add mcp
# Add to Claude Code globally
brightdata add mcp --agent claude-code --global
# Add to multiple agents at once
brightdata add mcp --agent claude-code,cursor --project
# Add to Codex
brightdata add mcp --agent codex --global
```
### Install skills into coding agents
```bash theme={null}
# Interactive picker - choose skills and target agents
brightdata skill add
# Install the scraping skill into your agent
brightdata skill add scrape
# Install search capabilities
brightdata skill add search
# See all available skills
brightdata skill list
```
Skills are pre-packaged bundles of prompts and configuration that teach AI coding agents how to use Bright Data effectively. See [For your coding agent](/quickstart-coding-agent) and [github.com/brightdata/skills](https://github.com/brightdata/skills) for more details.
***
## Supported environment variables
Override any stored configuration with environment variables:
| Variable | Purpose |
| - | - |
| `BRIGHTDATA_API_KEY` | API key (skips login entirely) |
| `BRIGHTDATA_UNLOCKER_ZONE` | Default Web Unlocker zone |
| `BRIGHTDATA_SERP_ZONE` | Default SERP zone |
| `BRIGHTDATA_POLLING_TIMEOUT` | Polling timeout in seconds |
| `BRIGHTDATA_BROWSER_ZONE` | Default Scraping Browser zone (default: `cli_browser`) |
| `BRIGHTDATA_DAEMON_DIR` | Directory for the browser daemon socket and state. Defaults to the CLI config directory for your OS |
```bash theme={null}
# Use in CI/CD without login
BRIGHTDATA_API_KEY=your_key brightdata scrape https://example.com
# Override timeout for large pipeline jobs
BRIGHTDATA_POLLING_TIMEOUT=1200 brightdata pipelines amazon_product "https://amazon.com/dp/..."
```
# Bright Data CLI FAQs
Source: https://docs.brightdata.com/products/cli/faqs
FAQs and troubleshooting for the Bright Data CLI: brightdata and bdata aliases, authentication, installation issues and common errors.
## Frequently asked general questions
They are identical - `bdata` is a shorthand alias installed alongside `brightdata`. Use whichever you prefer.
Yes. The CLI connects to Bright Data's infrastructure to handle web requests. You can [sign up for free](https://brightdata.com/?hs_signup=1\&utm_source=docs) and get started with the free tier.
The CLI supports 40+ platforms through the `pipelines` command, including Amazon, LinkedIn, Instagram, TikTok, YouTube, Facebook, Reddit, Google Maps, Walmart, eBay, and many more. Run `brightdata pipelines list` to see the full list.
* **Scrape:** `markdown` (default), `html`, `json`, `screenshot`
* **Search:** Formatted table (default), `json`, `pretty`
* **Pipelines:** `json` (default), `csv`, `ndjson`, `jsonl`
All commands support `-o ` to write output to a file.
Yes. The CLI is fully pipe-friendly. When stdout is not a TTY, colors and spinners are automatically disabled. Use `--json` for machine-readable output and `BRIGHTDATA_API_KEY` environment variable for non-interactive authentication.
## Authentication
Credentials are stored locally on your machine:
| OS | Path |
| - | - |
| macOS | `~/Library/Application Support/brightdata-cli/credentials.json` |
| Linux | `~/.config/brightdata-cli/credentials.json` |
| Windows | `%APPDATA%\brightdata-cli\credentials.json` |
The file is set to mode `0o600` (owner read/write only).
Use the device flow:
```bash theme={null}
brightdata login --device
```
This prints a URL and a verification code. Open the URL on any device with a browser, enter the code, and authentication completes on the server.
Run `brightdata logout` followed by `brightdata login` with the new account. Or pass a different API key directly:
```bash theme={null}
brightdata login --api-key
```
## Troubleshooting
This means no default zone is configured. Fix it by either:
```bash theme={null}
# Re-run login (auto-creates zones)
brightdata login
# Or set a zone manually
brightdata config set default_zone_unlocker
```
Your stored API key is no longer valid. Re-authenticate:
```bash theme={null}
brightdata login
```
Your API key does not have permission for the requested zone or operation. Check zone permissions in the [Bright Data control panel](https://brightdata.com/cp).
You've hit the rate limit for your zone. Options:
* Wait a moment and retry
* Use `--async` for large jobs to avoid blocking
* Contact your account manager to increase limits
The default polling timeout is 600 seconds (10 minutes). For large datasets, increase it:
```bash theme={null}
# Per-command
brightdata pipelines amazon_product "" --timeout 1200
# Via environment variable
export BRIGHTDATA_POLLING_TIMEOUT=1200
```
The CLI auto-detects TTY support. If detection fails, pipe through `cat` to force plain output:
```bash theme={null}
brightdata scrape https://example.com | cat
```
Or use `--json` for clean, parseable output.
## Frequently asked configuration questions
```bash theme={null}
brightdata config set default_format json
```
Valid values: `markdown`, `json`.
Settings are resolved in this order (highest priority first):
1. **CLI flags** - e.g., `--zone my_zone`
2. **Environment variables** - e.g., `BRIGHTDATA_UNLOCKER_ZONE`
3. **config.json** - e.g., `default_zone_unlocker`
4. **Defaults** - built-in fallbacks
Delete the configuration directory:
```bash theme={null}
# Linux
rm -rf ~/.config/brightdata-cli/
# macOS
rm -rf ~/Library/Application\ Support/brightdata-cli/
```
Then run `brightdata login` to start fresh.
# Install the Bright Data CLI
Source: https://docs.brightdata.com/products/cli/installation
Install the Bright Data CLI via npm or curl and authenticate your account with an API key in 1 minute. Includes setup verification commands.
## Install the CLI
The fastest way to run the Bright Data CLI is `npx`, which executes the latest version with no global install:
```bash theme={null}
npx -p @brightdata/cli brightdata --version
```
Prefix any command in these docs with `npx -p @brightdata/cli` to run it the same way, with no global dependency to maintain. The Bright Data CLI requires Node.js 20 or later.
For a persistent `brightdata` command and faster startup, install the CLI globally instead:
```bash npm (recommended) theme={null}
npm install -g @brightdata/cli
```
```bash yarn theme={null}
yarn global add @brightdata/cli
```
```bash pnpm theme={null}
pnpm add -g @brightdata/cli
```
On macOS and Linux, an install script does the same global install and then starts the login flow:
```bash theme={null}
curl -fsSL cli.brightdata.com/install.sh | sh
```
The script checks for Node.js 20 or later, runs `npm install -g @brightdata/cli`, then calls `brightdata login`. It needs npm on your PATH, so it is a shortcut rather than a Node-free install.
Verify a global install:
```bash theme={null}
brightdata --version
```
The shorthand alias `bdata` is also available, both with `npx -p @brightdata/cli bdata` and after a global install - use whichever you prefer.
## Windows
Running the CLI through `npx` authenticates your session but does not put a `brightdata` or `bdata` binary on your PATH. Install globally if you want a persistent command:
```bash theme={null}
npm install -g @brightdata/cli
```
Calling the CLI from a Python subprocess on Windows fails with `[WinError 2] The system cannot find the file specified`, or `[WinError 193]`, because npm installs a `.cmd` shim that Python cannot execute directly. Three ways round it:
```python theme={null}
import subprocess
# 1. Call the .cmd shim by name
subprocess.run(["bdata.cmd", "scraper", "run", "c_YOUR_COLLECTOR_ID"])
# 2. Route through the shell
subprocess.run("bdata scraper run c_YOUR_COLLECTOR_ID", shell=True)
# 3. Go through cmd /c
subprocess.run(["cmd", "/c", "bdata", "scraper", "run", "c_YOUR_COLLECTOR_ID"])
```
Two more Windows notes:
* PowerShell blocks the npm shim under the default execution policy. Run `Get-ExecutionPolicy` and, if it returns `Restricted`, allow local scripts with `Set-ExecutionPolicy -Scope CurrentUser RemoteSigned`.
* The default console code page is cp1252, which raises `UnicodeEncodeError` when scraped pages contain UTF-8. Set `PYTHONUTF8=1`, or run `chcp 65001` before the command.
## Update the CLI
Upgrade to the latest version with the same package manager you installed with:
```bash npm theme={null}
npm install -g @brightdata/cli@latest
```
```bash yarn theme={null}
yarn global add @brightdata/cli@latest
```
```bash pnpm theme={null}
pnpm add -g @brightdata/cli@latest
```
Check your installed version against the latest published version:
```bash theme={null}
brightdata --version # your installed version
npm view @brightdata/cli version # latest version on npm
```
New commands ship in CLI releases. For example, `scraper heal` and `scraper approve` were added in v0.3.1. Run the update command above to pick them up. See the [release notes](https://github.com/brightdata/cli/releases) for what changed in each version.
## Authenticate
Run the login command to connect your Bright Data account:
```bash theme={null}
brightdata login
```
This opens your browser for secure OAuth authentication. Once complete, the CLI:
1. Validates and stores your API key locally
2. Auto-creates required proxy zones (`cli_unlocker`, `cli_browser`)
3. Sets sensible defaults so you can start immediately
You only need to log in **once**. All subsequent commands authenticate automatically.
### Alternative authentication methods
When no browser is available, use the device flow:
```bash theme={null}
brightdata login --device
```
This prints a URL and a code. Open the URL on any device, enter the code, and the CLI completes authentication.
For CI/CD pipelines or non-interactive environments, pass your API key directly:
```bash theme={null}
brightdata login --api-key YOUR_API_KEY
```
You can find your API key in the [Bright Data control panel](https://brightdata.com/cp/setting).
Set the `BRIGHTDATA_API_KEY` environment variable to skip login entirely:
```bash theme={null}
export BRIGHTDATA_API_KEY=YOUR_API_KEY
```
This is useful for Docker containers, GitHub Actions, and other automated environments.
The [Bright Data Python SDK](/api-reference/SDK) 2.5.0 and later also reads `BRIGHTDATA_API_KEY`, so one export covers the CLI and the SDK. Older SDK versions read only `BRIGHTDATA_API_TOKEN`.
## Interactive setup wizard
For a guided first-time experience, use the init command:
```bash theme={null}
brightdata init
```
This walks you through authentication, zone selection, and default configuration step by step.
| Flag | Description |
| - | - |
| `--skip-auth` | Skip the authentication step (if already logged in) |
| `-k, --api-key ` | Provide API key directly |
## Verify your setup
After logging in, confirm everything is working:
```bash theme={null}
# Check your configuration
brightdata config
# Verify API connectivity
brightdata budget
# Try a quick scrape
brightdata scrape https://example.com
```
## Where configuration is stored
The CLI stores credentials and configuration locally:
| OS | Path |
| - | - |
| macOS | `~/Library/Application Support/brightdata-cli/` |
| Linux | `~/.config/brightdata-cli/` |
| Windows | `%APPDATA%\brightdata-cli\` |
Two files are created:
| File | Purpose | Permissions |
| - | - | - |
| `credentials.json` | API key | `0o600` (owner-only) |
| `config.json` | Zones, output format, preferences | Standard |
**Priority order for configuration:** CLI flags → Environment variables → `config.json` → Defaults. You can always override any setting on a per-command basis.
## Next steps
Explore the full command reference.
Jump into real-world workflows and recipes.
# Bright Data CLI
Source: https://docs.brightdata.com/products/cli/overview
Scrape websites, search the web, extract structured data from 40+ platforms, and manage your Bright Data account from the terminal.
Install the CLI, log in once, and start scraping - CAPTCHAs, anti-bot protections, and JavaScript rendering are handled automatically.
## What can you do with the CLI?
Get clean markdown, HTML, JSON, or screenshots from any URL - anti-bot bypass and JS rendering included.
Query Google, Bing, or Yandex and get structured results with organic listings, ads, and People Also Ask.
Pull product details, profiles, reviews, and more from 40+ platforms like Amazon, LinkedIn, Instagram, and TikTok.
Navigate, click, type, screenshot, and read pages using a real remote browser session with persistent state.
Install skills or add the Bright Data MCP server into Claude Code, Cursor, Codex, and other AI coding agents.
List proxy zones, inspect configurations, and monitor account balance and per-zone costs.
JSON output, file export, and pipe-friendly design make it easy to chain commands and build workflows.
## What the CLI looks like
```bash theme={null}
# Scrape a page as clean markdown
brightdata scrape https://news.ycombinator.com
# Search Google and get structured results
brightdata search "best web scraping tools 2025"
# Extract an Amazon product as JSON
brightdata pipelines amazon_product "https://amazon.com/dp/B09V3KXJPB" --format json
# Control a remote browser session
brightdata browser open https://example.com
brightdata browser snapshot --compact
# Add Bright Data MCP server to your coding agent
brightdata add mcp
```
The CLI alias `bdata` is available as a shorthand - e.g., `bdata scrape https://example.com`.
## How it works
The Bright Data CLI wraps the full Bright Data web data platform into simple terminal commands. Under the hood, it:
1. **Authenticates once** via OAuth, device flow, or API key - credentials are stored locally and never need to be entered again
2. **Auto-provisions zones** (`cli_unlocker`, `cli_browser`) on first login so you can start immediately
3. **Routes requests** through Bright Data's infrastructure, handling CAPTCHAs, bot detection, IP rotation, and JavaScript rendering
4. **Returns clean output** - formatted tables in the terminal, or structured JSON/CSV/markdown for automation
## Explore CLI resources
Install the CLI and authenticate with Bright Data.
Full reference for every command, flag, and option.
Real-world workflows and recipes for common tasks.
Answers to common questions and troubleshooting tips.
**Zero configuration required.** After a one-time `brightdata login`, every command works out of the box - no tokens to manage, no zones to create, no proxies to configure.
# Customization and filtering
Source: https://docs.brightdata.com/products/marketplace/customization-and-filtering
Customize fields and apply filters to Bright Data Marketplace datasets (250+ domains) using the UI or API to get the data subset you need.
Narrow down Bright Data Marketplace datasets (250+ domains) to the exact records and fields you need using the UI or API by creating filtered subsets and selecting the fields to include.
**What you can do:**
* **Customize fields:** choose which columns (fields) to include in your dataset view or export.
* **Filter a dataset:** create a saved subset using filter rules.
**Permissions:** You need access to the dataset and permission to create subsets.
## How to customize fields
Choose which fields (columns) appear in your Bright Data Marketplace dataset view and exports.
1. Navigate to the dataset you want to work with.
2. On the Data sample tab, click Edit table.
3. Select the fields you want to include in the view.
4. Click Apply, and save the view if prompted.
**Expected result:** The dataset preview and any export from this view include only the selected fields.
## How to filter a dataset
Create a saved subset of a Bright Data Marketplace dataset by applying filter rules in the UI or through the API.
### Filter a dataset using the UI
1. Navigate to the dataset you want to filter.
2. Click the Filter icon (top right).
3. Enter a name so you can find this subset later.
4. Under Include filters, add one or more filters (for example: country, job title, date).
5. Click Create subset.
**Expected result:** A new subset appears with only the records that match your filters.
### Filter operators reference
#### Select
Match one or more exact values from a predefined list (for example: countries or regions).
#### Boolean (true/false)
Filter fields that can only be true or false (for example: a field like `verified`).
#### Date
Filter records within a specific date range (start date and end date).
#### Number (operators)
* **Is:** match an exact numeric value.
* **Not:** exclude a numeric value.
* **Exists:** include only records where the field is not empty.
* **List (exact match):** match any value in a provided list.
* **Lower than / Lower or equal to:** match values below (or up to) a threshold.
* **Greater than / Greater or equal to:** match values above (or at) a threshold.
#### String
Filter text fields using the match types available in the UI (for example: exact match or contains).
#### Array
Use Array includes to match records where a multi-value (array) field contains a specific value (for example: categories, attributes or labels).
#### Upload a CSV list
If you need to match many values, use the CSV upload option in the value input. Upload a CSV with one value per row. After upload, the filter matches records where the array contains any value from the uploaded list.
#### CSV upload limitations
* **CSV format:** upload a `.csv` file with one value per row (single column).
* **Maximum 10,000 values per list:** to match more, split the values across multiple CSV lists, up to 100,000 values total, and apply them as filters in separate subsets.
* **No empty or whitespace-only lines:** lines that contain only spaces are rejected.
* **Single column only:** the CSV must contain only one column. If you include a header, it must be a single column.
* **One value per row:** put exactly one filter value on each row. Do not comma-separate multiple values on the same row.
### Includes vs List (exact match)
#### Includes
Use Includes to match records where the field contains the value you enter (partial match).
Example: if the field is `name` and you filter with Includes = `john`, you match values like `John Smith` and `Johnson`.
#### List (exact match)
Use List (exact match) to match records where the field value is exactly one of the values in your list (no partial matches).
Example: List (exact match) = `John Smith`, `Jane Doe` matches only those exact values.
### Group filters (rule-based filters)
1. Click + Add filter.
2. Select Add group.
3. Define your group rules (for example: category is "Electronics" and brand is "Dell" or "Apple").
**Expected result:** The subset includes only records that match your group logic.
### Limitations
* Filter groups can be nested up to 3 levels deep, with no limit on the number of sibling groups. See the [filter syntax reference](/api-reference/marketplace-dataset-api/filter-syntax).
* A maximum of four inputs per filter group is allowed.
* To filter by more values, use a CSV list upload option (if available) or contact your account manager.
* For more complex queries, contact your account manager.
## Troubleshooting
* **I can't see the Filter icon.** Make sure you're viewing the dataset table and that your account has permission to create subsets.
* **My subset returns no results.** Remove filters one by one to identify the restrictive condition, then reapply the correct values and operators.
## Next steps
* [Export or download a dataset subset](/products/marketplace/data-delivery-and-export)
* [Filter a dataset by API](/products/marketplace/filter-dataset-by-api)
* [Bright Data Marketplace datasets overview](/left-over/marketplace-overview)
# Dataset view
Source: https://docs.brightdata.com/products/marketplace/dataset-view
Browse dataset details in the Bright Data Marketplace (250+ domains), including data fields, freshness, sample downloads and purchase options.
## How to preview a data sample
Some values in the data sample preview may be partially masked with asterisks `*`). This means the value is hidden in the preview and does not indicate that the field is missing. Full field availability depends on the dataset, selected filters, and purchase options.
* Data-fields
* Records
* Freshness
* Min order
* Download data sample: JSON, CSV, number records (display 30 records but you get 1,000 records)
* CSV/JSON
* Create a subset
* Talk to a data expert
* Purchase options
### How to customize a dataset
* Customize - Create a custom subset by hiding/showing and renaming fields and filter the dataset according to your requirements.
* Customize Fields
* Filter
## Where to view the data dictionary
* Search attribute
* Table
* Column name
* Description
* Data type
## When to use this dataset
* Case study
* Related subset
* What category it’s related to
## What this dataset covers
* Description about the dataset
* What domain we collect from
* Use case categories
* Customize dataset CTA
* Delivery options
* FAQ
## Initiate by API
* Ability to submit a request through the API or the UI to collect fresh data based on your own URLs (inputs)
* +Add input
* Upload CSV
* Creating API key to initiate a collection
* Setting the expected records
* Data Collection Choices
* Discover new records
* Specific URL collection
* Purchase options
# Marketplace fill rates
Source: https://docs.brightdata.com/products/marketplace/fill-rates-and-statistics
Review per-field fill rates, record counts and data freshness on the Statistics tab of any Bright Data Marketplace dataset (250+ domains) before purchase.
Every Bright Data Marketplace dataset page exposes field-level fill rates in the **Fill rate** column of the **Dictionary** tab, alongside the total record count and a freshness indicator; the **Statistics** tab holds dataset-level charts. Review these before purchase to confirm the dataset meets your requirements.
## What are fill rates?
A fill rate is the percentage of records in a dataset where a specific field is populated. A fill rate of 100 percent means every record contains a value for that field. A lower fill rate means some records are missing values for that field.
Fill rates vary depending on:
* The dataset type (social media, B2B, eCommerce)
* The source quality and what is publicly available
* The specific field being measured (email, phone number, location)
## Where to find fill rates
Fill rates and statistics are shown on each dataset's page in the Marketplace.
In the Bright Data Control Panel, go to **Dataset Marketplace**.
Open the dataset whose coverage you want to review.
Navigate to the **Dictionary** tab and review the **Fill rate** column in the dictionary table to view field-level fill rates. The **Statistics** tab shows dataset-level charts.
## What statistics are available?
For each dataset you can review:
* **Fill rate per field**: percentage of records with a value
* **Record count**: total number of available records
* **Freshness**: how recently the data was collected
## Why do some fields have low fill rates?
Some fields have lower fill rates because of limitations or gaps in the publicly available source data. For example, phone numbers and email addresses are not always publicly listed, so contact fields commonly show lower fill rates than name or location fields.
Always review fill rates before purchasing to confirm the data meets your specific requirements.
# Pricing
Source: https://docs.brightdata.com/products/marketplace/pricing
Understand how Bright Data Marketplace datasets are priced, including per-record costs and compute fees for pre-collected and fresh records.
* Pre-collected records
* Fresh records
* Pricing is determined based on compute cost and record cost to reach and collect the data.
For delivery methods and output formats, see [Data delivery and export](/products/marketplace/data-delivery-and-export).
# Purchase options
Source: https://docs.brightdata.com/products/marketplace/purchase-options
Compare 2 Bright Data Marketplace purchase options: one-time or subscription, pre-collected or fresh data, with supported delivery destinations.
* Purchase options → Flexibility
* One-time purchase
* Subscription
* Select data type
* Pre-collected data - Access data that was collected recently and is ready for use.
* Fresh data - Access data that's up-to-date and fresh, available immediately after it's collected.
* Data distribution - shows how many records are available for the dataset
* Specific time range - Access data from a specific time range.(back from today)
* Available only for pre-collected.
* Select Period
* Collect fresh data for the rest of the records
* Records outside of the specified time range will be scraped when the request is processed and will incur an additional fee. The refresh time depends on the size of the dataset. You'll get an email when your snapshot is ready.
* As the data changes, these records may no longer match your filters and may be excluded from your dataset. If these records later do not meet your filters, they will be removed, but you will still be charged for the initial scrape.
* Set record limit - Enter the number of records you would like. Records are selected randomly. Notice that the price will change according to the record count.
* Order
* Dataset name - “LinkedIn people profile dataset”
* Total amount of records
* “Records” - cost of total records
* Number of records
* Cost per record - \$0.0006
* Subtotal - (before VAT or salestax)
* Total
* Price adjustment (max snapshot cost)
* Small dataset fee (min snapshot cost)
# Bright Data MCP FAQs
Source: https://docs.brightdata.com/products/mcp-server/faqs
FAQs about the Bright Data MCP server (60+ tools): setup, hosted versus self-hosted modes, available tools, token usage and billing for AI agents.
You might be eligible for our Startup Program. Get fully funded access to the infrastructure you're reading about right now (up to \$20K value).
Bright Data MCP (Model Context Protocol) is a server that enables LLMs, AI agents, and applications to access, discover, and extract web data in real-time. \
It allows MCP clients like Claude Desktop, Cursor, and Windsurf to search the web, navigate websites, take actions, and retrieve data without getting blocked, making it perfect for web scraping tasks.\\
* New MCP users get a **free tier** with 5,000 requests per month
The MCP server provides advanced capabilities, including:
* Bypassing geo-restrictions to access content regardless of location
* Navigating websites with bot detection protection using unlocker technology
* Structured data extraction from platforms like Amazon, LinkedIn, Instagram, and many other data feeds
* Remote browser automation for complex web interactions
* Access to a global IP network to avoid blocking or rate limiting
**Get started with Bright Data MCP:**
1. [**Explore the MCP GitHub repository**](https://github.com/brightdata/brightdata-mcp) to review the source code, contribute, or deploy your own instance.
2. [**Read the MCP documentation**](/products/mcp-server/overview) to understand the API, request structure, and supported features.
3. [Get your API key](https://brightdata.com/cp/setting/users)
4. [**Try the Bright Data AI Playground**](https://brightdata.com/ai/playground-chat) to test and experiment with live requests in a no-code environment.
Yes, the free tier includes up to 5,000 requests per month for web search and web page scraping via Web Unlocker.
You can get your API token from the [user settings page](https://brightdata.com/cp/setting/users) in your Bright Data account. Make sure you have an account on [brightdata.com](https://brightdata.com) - new users get free credit for testing, and pay-as-you-go options are available after that.
No. The Bright Data MCP server only requires an API token, for both the remote (hosted) and local (self-hosted) setup. Usernames (format: `brd-customer-{customer_id}-zone-{zone_name}`) are only used for native proxy access and are not needed for MCP, SDK or API integrations.
If a third-party tool or integration asks for a username, you can leave it empty or check whether it is actually configuring proxy access.
Bright Data MCP (Model Context Protocol) is a server that allows AI agents, LLMs, and dev tools to access and extract real-time web data using Bright Data’s infrastructure. It is available as both a [remote (hosted)](/products/mcp-server/remote/quickstart) and [local (self-hosted)](/products/mcp-server/local/quickstart) server.
To get started: [**BrightData MCP Overview**](/products/mcp-server/overview)
**Helpful links:**
* [MCP GitHub Repository](https://github.com/brightdata/brightdata-mcp)
* [MCP Documentation](/products/mcp-server/overview)
* [Try MCP in the AI Playground](https://brightdata.com/ai/playground-chat)
Bright Data MCP allows you to integrate real-time web access into your AI agents or dev tools. It supports:
**Search & Scraping:**
* `search_engine`: Search Google, Bing, Yandex
* `scrape_as_markdown`, `scrape_as_html`: Get content in clean formats
* `session_stats`: Monitor tool usage
**Structured Data Extraction:**
* Amazon: `web_data_amazon_product`, `web_data_amazon_product_reviews`
* LinkedIn: `web_data_linkedin_person_profile`, `web_data_linkedin_company_profile`
* Instagram: `web_data_instagram_profiles`, `web_data_instagram_posts`, `web_data_instagram_reels`
* Facebook: `web_data_facebook_posts`, `web_data_facebook_marketplace_listings`
* Others: `web_data_x_posts`, `web_data_youtube_videos`, `web_data_zillow_properties_listing`
**Browser Automation:**
* `scraping_browser_navigate`, `scraping_browser_click_ref`, `scraping_browser_type_ref`
* `scraping_browser_get_html`, `scraping_browser_screenshot`, `scraping_browser_wait_for_ref`
**🔗 Full list of tools:** [MCP Tools on GitHub](https://github.com/brightdata/brightdata-mcp/blob/main/assets/Tools.md)
Bright Data MCP is designed to work with a wide range of AI tools and agents, including:
* **Claude Desktop** (Anthropic)
* **Cursor IDE**
* **Windsurf**
* **LangChain**
* **AutoGPT / AgentGPT**
* **Smithery**
* **Custom LLMs or AI agents** that support MCP
These tools can use MCP to:
* Perform real-time web searches
* Scrape structured data
* Automate browser interactions
* Access geo-restricted or protected content
Try it live: [Bright Data AI Playground](https://brightdata.com/ai/playground-chat)
All you need is your Bright Data API key, the MCP server automatically creates and configures the required browser zone for you.
To access browser tools, enable them using one of these methods:
* **Pro mode:** Set `&pro=1` (remote) or `PRO_MODE=true` (self-hosted) to enable all tools, including browser automation
* **Tool groups:** Set `&groups=browser` (remote) or `GROUPS=browser` (self-hosted) to enable just the browser automation tools
For example, with the remote MCP:
```http theme={null}
https://mcp.brightdata.com/sse?token=YOUR_API_TOKEN&groups=browser
```
See the [Remote Advanced Configuration](/products/mcp-server/remote/advanced) or [Local Advanced Configuration](/products/mcp-server/local/advanced) for more details.
For the full list of tools, please refer to our GitHub repo: [MCP Tools](https://github.com/brightdata/brightdata-mcp/blob/main/assets/Tools.md).
Bright Data MCP offers a comprehensive set of tools, including:
**Search and basic scraping:**
* `search_engine`: Scrape search results from Google, Bing, or Yandex
* `scrape_as_markdown`: Scrape a webpage and get results in Markdown format
* `scrape_as_html`: Scrape a webpage and get results in HTML format
* `session_stats`: View tool usage during the current session
**Structured data extraction:**
* Amazon: `web_data_amazon_product`, `web_data_amazon_product_reviews`
* LinkedIn: `web_data_linkedin_person_profile`, `web_data_linkedin_company_profile`
* ZoomInfo: `web_data_zoominfo_company_profile`
* Instagram: `web_data_instagram_profiles`, `web_data_instagram_posts`, `web_data_instagram_reels`, `web_data_instagram_comments`
* Facebook: `web_data_facebook_posts`, `web_data_facebook_marketplace_listings`, `web_data_facebook_company_reviews`
* Various others: `web_data_x_posts`, `web_data_zillow_properties_listing`, `web_data_booking_hotel_listings`, `web_data_youtube_videos`
**Browser automation:**
* `scraping_browser_navigate`: Navigate to a URL
* `scraping_browser_go_back`/`scraping_browser_go_forward`: Navigate browser history
* `scraping_browser_click_ref`: Click on an element
* `scraping_browser_type_ref`: Type text into an element
* `scraping_browser_wait_for_ref`: Wait for an element to appear
* `scraping_browser_screenshot`: Take a screenshot
* `scraping_browser_get_html`/`scraping_browser_get_text`: Get page content
Some web data tools can take longer to execute, especially when dealing with complex websites. To ensure your agent can consume the data:
1. Set a high enough timeout in your agent settings (180 seconds is recommended)
2. For particularly slow sites, you may need to increase this value further
3. Use the specialized `web_data_*` tools when available, as they're often faster than general scraping
4. For browser automation sequences, keep operations close together in time
Always treat scraped web content as untrusted data. Never use raw scraped content directly in LLM prompts to avoid potential prompt injection risks. Instead:
* Filter and validate all web data before passing it to the LLM
* Use structured data extraction rather than raw text when possible
* Be cautious with executing JavaScript from scraped content
This error occurs when your system cannot find the `npx` command. To fix it:
1. Find your Node.js path:
* On macOS: Run `which node` in Terminal
* On Windows: Run `where node` in Command Prompt
2. Update your MCP configuration to use the full path instead of `npx`:
```json theme={null}
{
"mcpServers": {
"Bright Data": {
"command": "/usr/local/bin/node", // Replace with your actual Node.js path
"args": ["node_modules/@brightdata/mcp/index.js"],
"env": {
"API_TOKEN": "YOUR_API_KEY"
}
}
}
}
```
You can use the `session_stats` tool to check your usage during the current session. For comprehensive usage tracking and billing information, log in to your Bright Data account dashboard. The `session_stats` tool will show you information about:
* Number of requests made during the current session
* Tools used and their frequency
Yes, you can try Bright Data MCP without any setup using the [Bright Data AI Playground](https://brightdata.com/ai/playground-chat). It provides an easy way to explore the capabilities of Bright Data MCP without any local setup. Just sign in and start experimenting with web data collection!
You can connect your AI agent with Bright Data by following these steps:
1. Install [Node.js](https://nodejs.org/en/download) (only required for the self-hosted option)
2. Create a [Bright Data account](https://brightdata.com/?hs_signup=1\&utm_source=docs) (new users get free credit for testing)
3. Copy your API key from the [user settings page](https://brightdata.com/cp/setting/users) (new users receive an **API key** in the welcome email)
4. Configure your agent with your API key. For more detailed examples, visit us on [GitHub](https://github.com/brightdata/brightdata-mcp?tab=readme-ov-file#-usage-examples) or check our [MCP documentation](/products/mcp-server/overview#configuration).
No, zones are created automatically for you. All you need is Node.js (for self-hosted) and your Bright Data API key. If you want to change your zone name to something specific, you can always do so through the [control panel](https://brightdata.com/cp/zones).
Yes! Bright Data provides a free tier of **5,000 requests/month** to new MCP users. The free tier includes web search and web page scraping via Web Unlocker. See [What is the MCP free tier?](#) below for more details.
The MCP free tier allows you to connect your AI agent via MCP (Model Context Protocol) to Bright Data services for free.
You get up to 5,000 requests per month for web search and scraping any public webpage via Web Unlocker at no cost. This is ideal for testing, development, or light usage.
The MCP free tier is not a separate grant. It is the same account-level [free tier](/general/account/billing-and-pricing/free-tier) of 5,000 credits per month, drawn from a single shared pool, because the MCP server runs on the Web Unlocker API.
For more information, see the [Remote MCP Quickstart](/products/mcp-server/remote/quickstart).
When you exceed the 5,000 monthly request limit on the free tier, requests will start failing. To continue using the MCP server, you need to create a new Web Unlocker zone and configure the MCP to use it:
1. Go to [brightdata.com/cp](https://brightdata.com/cp) and click **Add** (top-right)
2. Select **Unlocker zone** and create it with any name (e.g., `my_unlocker`)
3. Update your MCP configuration to use the new zone:
* **Remote MCP:** Add `&unlocker=ZONE_NAME` to your MCP URL
* **Self-hosted MCP:** Set the `WEB_UNLOCKER_ZONE=ZONE_NAME` environment variable
4. Restart your MCP client (e.g., Claude Desktop, Cursor) after the configuration change
Yes, you can run multiple MCP instances for parallelized workloads. However, if you are using the self-hosted (local) MCP with `npx`, each instance downloads and caches the package separately, which can consume significant disk space.
For parallel use cases, we recommend using the **Remote MCP** instead. It requires no local installation, runs entirely in the cloud, and eliminates disk space concerns. Simply point each instance to the remote endpoint:
```http theme={null}
https://mcp.brightdata.com/sse?token=YOUR_API_TOKEN
```
See the [Remote MCP Quickstart](/products/mcp-server/remote/quickstart) for setup instructions.
# Local MCP server advanced configuration
Source: https://docs.brightdata.com/products/mcp-server/local/advanced
Customize the local Bright Data MCP server (60+ tools): enable Browser API, configure proxies, set rate limits and pass environment variables.
You might be eligible for our Startup Program. Get fully funded access to the infrastructure you're reading about right now (up to \$20K value).
## MCP Server Modes
Bright Data's MCP server offers two modes to suit different needs:
* **Rapid (Free)** - Quickly scrape search results and unlock any public webpage as clean Markdown.
* **Pro** - Access advanced scraping, structured data from top platforms (Amazon, LinkedIn, X, Instagram, etc.), and full browser automation. Built for dynamic and large-scale use cases.
* **Groups** - Unlock targeted capabilities with pre-built bundles. Whether you're monitoring competitors, analyzing social trends, or automating browsers, select the exact tool groups your project requires.
* **Tools** - Fine-tune your toolkit. Cherry-pick individual tools to supplement your Groups or build a completely custom setup for specialized use cases.
**Rapid (Free)** mode is enabled by default and is **recommended** for everyday browsing and data needs.
***
## How to configure the local MCP server
When using the MCP server locally, you can set environment variables for finer control:
* `RATE_LIMIT` - Set the rate limit for requests (*default:* `100/1h`).
* `WEB_UNLOCKER_ZONE` - Override the default web unlocker zone (*default:* `mcp_unlocker`).
* `BROWSER_ZONE` - Override the default browser zone (*default:* `mcp_browser`).
* `PRO_MODE` - Set to true/false to enable/disable Pro tools (*default:* `false`).
* `GROUPS` - Comma-separated list of tool group IDs to enable (e.g., `browser,ecommerce,social`). Available groups: `ecommerce`, `social`, `browser`, `finance`, `business`, `research`, `app_stores`, `travel`, `advanced_scraping`, `geo`, `code`.
* `TOOLS` - Comma-separated list of individual tool names to add on top of selected groups (e.g., `web_data_linkedin_person_profile,scrape_as_html`). Use this to fine-tune your toolkit with specific capabilities.
> **Note:** `PRO_MODE=true` enables all tools. If you use `GROUPS` or `TOOLS`, they take priority and override Pro mode.
Use `PRO_MODE=true` to enable **Pro mode**.
### Example configuration
```json theme={null}
{
"mcpServers":{
"Bright Data":{
"command":"npx",
"args":["@brightdata/mcp"],
"env":{
"API_TOKEN":"YOUR_API_KEY",
"WEB_UNLOCKER_ZONE": "my_zone_name",
"BROWSER_ZONE": "my_browser_zone",
"PRO_MODE": "true",
"GROUPS": "browser,advanced_scraping",
"TOOLS": "web_data_linkedin_person_profile,web_data_amazon_product"
}
}
}
}
```
# How to set up local (self-hosted) MCP
Source: https://docs.brightdata.com/products/mcp-server/local/quickstart
Set up a self-hosted Bright Data MCP server: step-by-step installation, configuration and connection to AI agents like Claude or Cursor in under 10 minutes.
You might be eligible for our Startup Program. Get fully funded access to the infrastructure you're reading about right now (up to \$20K value).
Before you begin, make sure you have the following:
*Sign up for free and receive **5,000 requests** per month at no cost.*
*New users receive their API key in the welcome email.*
*Ensure Node.js is installed and up to date.*
No username is required. The self-hosted MCP server authenticates with your `API_TOKEN` only. Usernames (format: `brd-customer-{customer_id}-zone-{zone_name}`) are for native proxy access, not MCP. If a third-party tool asks for a username, leave it empty.
Select and configure your preferred MCP client:
* Claude
* n8n
Try these example prompts with your MCP server:
“Extract all flight times departing from JFK Airport to Heathrow in the next 24 hours.”
“Extract the Bright Data overview section from LinkedIn.”
“Go to ebay.com, click the ‘Shop by category’ button in the navigation bar, and extract all categories.”
See the Advanced Configuration page for more details.
# MCP server overview
Source: https://docs.brightdata.com/products/mcp-server/overview
Use the Bright Data MCP server to give AI agents real-time access to public web data with 5,000 free requests per month and zero infrastructure setup.
You might be eligible for our Startup Program. Get fully funded access to the infrastructure you're reading about right now (up to \$20K value).
The **Bright Data MCP server** gives AI agents and assistants real-time access to public web data through the Model Context Protocol (MCP). Connect it to Claude, Cursor or any MCP-compatible client, and your agent can search, scrape and read live web pages without you building or maintaining data pipelines.
The MCP server runs on the Web Unlocker API, so every request handles proxy rotation, anti-bot challenges and CAPTCHA solving automatically. Every new account includes 5,000 free requests per month. You can run the server two ways: a fully managed remote server, or a self-hosted local instance.
Start using the Bright Data MCP server with **5,000 requests** per month, completely free.
## Choose your MCP server
Launch a fully managed MCP server in the cloud, no setup required.
Set up your own MCP server instance on-premise or in your private cloud.
**Optimize your token usage:** Select only the tools you need using [tool groups](/products/mcp-server/tools) (e.g., `groups=ecommerce,social`) or individual tools (e.g., `tools=web_data_amazon_product`). This reduces context size and maximizes your usage.
## Explore MCP server resources
Discover the built-in tools available for your MCP server.
Get quick answers to frequently asked MCP server questions.
Learn from practical examples to accelerate your implementation.
Connect the MCP server with leading AI platforms like Claude, Cursor and more.
Explore the MCP server source code on GitHub.
The MCP free tier is the same account-level [free tier](/general/account/billing-and-pricing/free-tier) of 5,000 credits per month, drawn from a single shared pool, because the MCP server runs on the Web Unlocker API. For team accounts, that free tier is shared across all users within the account. If you require additional credits, please contact your account manager.
# Remote MCP server advanced configuration
Source: https://docs.brightdata.com/products/mcp-server/remote/advanced
Customize the remote Bright Data MCP server (60+ tools): configure auth tokens, switch proxies, tune timeouts and enable advanced tools per session.
You might be eligible for our Startup Program. Get fully funded access to the infrastructure you're reading about right now (up to \$20K value).
## MCP Server Modes
Bright Data's MCP server offers two modes to suit different needs:
* **Rapid (Free)** - Quickly scrape search results and unlock any public webpage as clean Markdown.
* **Pro** - Access advanced scraping, structured data from top platforms (Amazon, LinkedIn, X, Instagram, etc.), and full browser automation. Built for dynamic and large-scale use cases.
* **Groups** - Unlock targeted capabilities with pre-built bundles. Whether you're monitoring competitors, analyzing social trends, or automating browsers, select the exact tool groups your project requires.
* **Tools** - Fine-tune your toolkit. Cherry-pick individual tools to supplement your Groups or build a completely custom setup for specialized use cases.
**Rapid (Free)** mode is enabled by default and is **recommended** for everyday browsing and data needs.
***
## How to configure the remote MCP server
When using a remote MCP server, you can configure advanced options by passing query parameters to your endpoint:
Use a custom zone name for web scraping.
> **For example:** `&unlocker=my_zone_name`
Use a custom zone name for browser automation.
> **For example:** `&browser=my_browser_zone`
Enable advanced (Pro) processing.
> **For example:** `&pro=1`
Use `&pro=1` to enable **Pro mode**.
Select pre-built tool bundles tailored to your domain. Comma-separated list of group IDs.
> **For example:** `&groups=browser,ecommerce,social`
Available groups: `ecommerce`, `social`, `browser`, `finance`, `business`, `research`, `app_stores`, `travel`, `advanced_scraping`, `geo`, `code`
Handpick individual tools to complement your groups selection. Comma-separated list of tool names.
> **For example:** `&tools=scrape_as_markdown,web_data_linkedin_person_profile`
Using `groups` or `tools` overrides `pro=1`. Tool selection takes priority over Pro mode.
### Example configurations
```sh SSE - Pro Mode (All Tools) theme={null}
https://mcp.brightdata.com/sse?token=YOUR_API_TOKEN&pro=1
```
```sh SSE - Tool Groups theme={null}
https://mcp.brightdata.com/sse?token=YOUR_API_TOKEN&groups=browser,ecommerce,social
```
```sh SSE - Custom Tools theme={null}
https://mcp.brightdata.com/sse?token=YOUR_API_TOKEN&groups=browser&tools=web_data_linkedin_person_profile,scrape_as_html
```
```sh HTTP - Pro Mode (All Tools) theme={null}
https://mcp.brightdata.com/mcp?token=YOUR_API_TOKEN&pro=1
```
```sh HTTP - Tool Groups theme={null}
https://mcp.brightdata.com/mcp?token=YOUR_API_TOKEN&groups=browser,ecommerce,social
```
```sh HTTP - Custom Tools theme={null}
https://mcp.brightdata.com/mcp?token=YOUR_API_TOKEN&groups=browser&tools=web_data_linkedin_person_profile,scrape_as_html
```
***
# How to get started with remote MCP server
Source: https://docs.brightdata.com/products/mcp-server/remote/quickstart
Connect AI agents to the fully managed Bright Data Remote MCP server (60+ tools) with no setup: authentication, tool selection and a working sample request.
You might be eligible for our Startup Program. Get fully funded access to the infrastructure you're reading about right now (up to \$20K value).
Before getting started, make sure you have the following:
*Sign up for free and receive **5,000 requests** per month at no cost.*
*New users receive their API key via the welcome email.*
Use one of the following endpoints to connect to the Remote MCP Server:
```sh SSE (Server-Sent Events) endpoint: theme={null}
https://mcp.brightdata.com/sse?token=YOUR_API_TOKEN
```
```sh Streamable HTTP endpoint: theme={null}
https://mcp.brightdata.com/mcp?token=YOUR_API_TOKEN
```
Replace `YOUR_API_TOKEN` with your actual Bright Data API token.
No username is required. The Remote MCP server authenticates with your API token only. Usernames (format: `brd-customer-{customer_id}-zone-{zone_name}`) are for native proxy access, not MCP. If a third-party tool asks for a username, leave it empty.
Select and configure your preferred client:
* Claude
* Cursor
* n8n
Try these sample prompts with your MCP Server:
“Extract all flight times departing from JFK Airport to Heathrow in the next 24 hours.”
“Extract the Bright Data overview section from LinkedIn.”
“Go to ebay.com, click the ‘Shop by category’ button in the navigation bar, and extract all categories.”
See the Advanced Configuration page for more details.
# Bright Data MCP tools
Source: https://docs.brightdata.com/products/mcp-server/tools
Reference for the Bright Data MCP server tools (60+ tools): web search, scraping, structured data and platform extractors for ecommerce and social workflows.
You might be eligible for our Startup Program. Get fully funded access to the infrastructure you're reading about right now (up to \$20K value).
### Modes: Rapid (Free), Pro and 11 tool groups
Bright Data's MCP server offers two modes to suit different needs:
* **Rapid (Free)** - Quickly scrape search results and unlock any public webpage as clean Markdown.
* **Pro** - Access advanced scraping, structured data from top platforms (Amazon, LinkedIn, X, Instagram, etc.), and full browser automation. Built for dynamic and large-scale use cases.
* **Groups** - Pre-configured tool collections for specific use cases including E-commerce, Social Media, Browser Automation, Business Intelligence, Finance, Research, App Stores, Travel, Advanced Scraping, GEO & LLM Visibility, and Code. Each group bundles relevant tools and data feeds for its domain.
***
To use **Pro mode**:
* In Remote MCP, request Pro features by setting `&pro=1`.
* In Local MCP, enable Pro features with `PRO_MODE=true`.
To use **Groups:**
* In Remote MCP, request the Groups by setting `&groups=`.
* In Local MCP, enable Groups with `GROUPS=`.
**Rapid (Free)** mode is enabled by default and is **recommended** for everyday browsing and data needs.
## Bright Data MCP Tools Reference
| Mode | Tool Name | Description | Group |
| :-: | :- | :- | - |
| | `search_engine` | Scrape search results from Google, Bing, or Yandex. Returns SERP results in JSON for Google and Markdown for Bing/Yandex; supports pagination with the cursor parameter. | `all` |
| | `scrape_as_markdown` | Scrape a single webpage with advanced extraction and return Markdown. Uses Bright Data's unlocker to handle bot protection and CAPTCHA. | `all` |
| | `search_engine_batch` | Run up to 10 search queries in parallel. Returns JSON for Google results and Markdown for Bing/Yandex. | `advanced_scraping` |
| | `scrape_batch` | Scrape up to 10 webpages in one request and return an array of URL/content pairs in Markdown format. | `advanced_scraping` |
| | `scrape_as_html` | Scrape a single webpage with advanced extraction and return the HTML response body. Handles sites protected by bot detection or CAPTCHA. | `advanced_scraping` |
| | `extract` | Scrape a webpage as Markdown and convert it to structured JSON using AI sampling, with an optional custom extraction prompt. | `advanced_scraping` |
| | `session_stats` | Report how many times each tool has been called during the current MCP session. | `advanced_scraping` |
| | `web_data_amazon_product` | Quickly read structured Amazon product data. Requires a valid product URL containing /dp/. Often faster and more reliable than scraping. | `ecommerce` |
| | `web_data_amazon_product_reviews` | Quickly read structured Amazon product review data. Requires a valid product URL containing /dp/. Often faster and more reliable than scraping. | `ecommerce` |
| | `web_data_amazon_product_search` | Retrieve structured Amazon search results. Requires a search keyword and Amazon domain URL; limited to the first page of results. | `ecommerce` |
| | `web_data_walmart_product` | Quickly read structured Walmart product data. Requires a product URL containing /ip/. Often faster and more reliable than scraping. | `ecommerce` |
| | `web_data_walmart_seller` | Quickly read structured Walmart seller data. Requires a valid Walmart seller URL. Often faster and more reliable than scraping. | `ecommerce` |
| | `web_data_ebay_product` | Quickly read structured eBay product data. Requires a valid eBay product URL. Often faster and more reliable than scraping. | `ecommerce` |
| | `web_data_homedepot_products` | Quickly read structured Home Depot product data. Requires a valid homedepot.com product URL. Often faster and more reliable than scraping. | `ecommerce` |
| | `web_data_zara_products` | Quickly read structured Zara product data. Requires a valid Zara product URL. Often faster and more reliable than scraping. | `ecommerce` |
| | `web_data_etsy_products` | Quickly read structured Etsy product data. Requires a valid Etsy product URL. Often faster and more reliable than scraping. | `ecommerce` |
| | `web_data_bestbuy_products` | Quickly read structured Best Buy product data. Requires a valid Best Buy product URL. Often faster and more reliable than scraping. | `ecommerce` |
| | `web_data_linkedin_person_profile` | Quickly read structured LinkedIn people profile data. Requires a valid LinkedIn profile URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_linkedin_company_profile` | Quickly read structured LinkedIn company profile data. Requires a valid LinkedIn company URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_linkedin_job_listings` | Quickly read structured LinkedIn job listings data. Requires a valid LinkedIn jobs URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_linkedin_posts` | Quickly read structured LinkedIn posts data. Requires a valid LinkedIn post URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_linkedin_people_search` | Quickly read structured LinkedIn people search data. Requires a LinkedIn people search URL. Often faster and more reliable than scraping. | `social` |
| | `list_dataset_fields` | List the filterable fields of a searchable dataset with each field's name, type and description. Call before `search_dataset` to learn which fields you can filter on. | `social`, `business` |
| | `search_dataset` | Search a Bright Data dataset by a filter and get matching records back directly, with no snapshot step. Finds many records by criteria, unlike `web_data_*` tools that fetch one record by URL. Returns up to 10 records per call with `search_after` pagination. | `social`, `business` |
| | `web_data_crunchbase_company` | Quickly read structured Crunchbase company data. Requires a valid Crunchbase company URL. Often faster and more reliable than scraping. | `business` |
| | `web_data_zoominfo_company_profile` | Quickly read structured ZoomInfo company profile data. Requires a valid ZoomInfo company URL. Often faster and more reliable than scraping. | `business` |
| | `web_data_instagram_profiles` | Quickly read structured Instagram profile data. Requires a valid Instagram profile URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_instagram_posts` | Quickly read structured Instagram post data. Requires a valid Instagram post URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_instagram_reels` | Quickly read structured Instagram reel data. Requires a valid Instagram reel URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_instagram_comments` | Quickly read structured Instagram comments data. Requires a valid Instagram URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_facebook_posts` | Quickly read structured Facebook post data. Requires a valid Facebook post URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_facebook_marketplace_listings` | Quickly read structured Facebook Marketplace listing data. Requires a valid Marketplace listing URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_facebook_company_reviews` | Quickly read structured Facebook company reviews data. Requires a valid Facebook company URL and review count. Often faster and more reliable than scraping. | `social` |
| | `web_data_facebook_events` | Quickly read structured Facebook events data. Requires a valid Facebook event URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_tiktok_profiles` | Quickly read structured TikTok profile data. Requires a valid TikTok profile URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_tiktok_posts` | Quickly read structured TikTok post data. Requires a valid TikTok post URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_tiktok_shop` | Quickly read structured TikTok Shop product data. Requires a valid TikTok Shop product URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_tiktok_comments` | Quickly read structured TikTok comments data. Requires a valid TikTok video URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_google_maps_reviews` | Quickly read structured Google Maps reviews data. Requires a valid Google Maps URL and optional days\_limit (default 3). Often faster and more reliable than scraping. | `business` |
| | `web_data_google_shopping` | Quickly read structured Google Shopping product data. Requires a valid Google Shopping product URL. Often faster and more reliable than scraping. | `ecommerce` |
| | `web_data_google_play_store` | Quickly read structured Google Play Store app data. Requires a valid Play Store app URL. Often faster and more reliable than scraping. | `app_stores` |
| | `web_data_apple_app_store` | Quickly read structured Apple App Store app data. Requires a valid App Store app URL. Often faster and more reliable than scraping. | `app_stores` |
| | `web_data_github_repository_file` | Quickly read structured GitHub repository file data. Requires a valid GitHub file URL. Often faster and more reliable than scraping. | `research` |
| | `web_data_yahoo_finance_business` | Quickly read structured Yahoo Finance company profile data. Requires a valid Yahoo Finance business URL. Often faster and more reliable than scraping. | `finance` |
| | `web_data_x_posts` | Quickly read structured X (Twitter) post data. Requires a valid X post URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_x_profile_posts` | Quickly read structured X posts from a profile. Requires a valid X profile URL. Returns the most recent posts, with optional date range filtering. | `social` |
| | `web_data_zillow_properties_listing` | Quickly read structured Zillow property listing data. Requires a valid Zillow listing URL. Often faster and more reliable than scraping. | `business` |
| | `web_data_booking_hotel_listings` | Quickly read structured Booking.com hotel listing data. Requires a valid Booking.com listing URL. Often faster and more reliable than scraping. | `travel` |
| | `web_data_youtube_videos` | Quickly read structured YouTube video metadata. Requires a valid YouTube video URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_youtube_profiles` | Quickly read structured YouTube channel profile data. Requires a valid YouTube channel URL. Often faster and more reliable than scraping. | `social` |
| | `web_data_youtube_comments` | Quickly read structured YouTube comments data. Requires a valid YouTube video URL and optional num\_of\_comments (default 10). Often faster and more reliable than scraping. | `social` |
| | `web_data_reddit_posts` | Quickly read structured Reddit post data. Requires a valid Reddit post URL. Often faster and more reliable than scraping. | `social` |
| | `scraping_browser_navigate` | Open or reuse a scraping-browser session and navigate to the provided URL, resetting tracked network requests. | `browser` |
| | `scraping_browser_go_back` | Navigate the active scraping-browser session back to the previous page and report the new URL and title. | `browser` |
| | `scraping_browser_go_forward` | Navigate the active scraping-browser session forward to the next page and report the new URL and title. | `browser` |
| | `scraping_browser_snapshot` | Capture an ARIA snapshot of the current page listing interactive elements and their refs for later ref-based actions. | `browser` |
| | `scraping_browser_click_ref` | Click an element using its ref from the latest ARIA snapshot; requires a ref and human-readable element description. | `browser` |
| | `scraping_browser_type_ref` | Fill an element identified by ref from the ARIA snapshot, optionally pressing Enter to submit after typing. | `browser` |
| | `scraping_browser_screenshot` | Capture a screenshot of the current page; supports optional full\_page mode for full-length images. | `browser` |
| | `scraping_browser_network_requests` | List the network requests recorded since page load with HTTP method, URL, and response status for debugging. | `browser` |
| | `scraping_browser_wait_for_ref` | Wait until an element identified by ARIA ref becomes visible, with an optional timeout in milliseconds. | `browser` |
| | `scraping_browser_get_text` | Return the text content of the current page's body element. | `browser` |
| | `scraping_browser_get_html` | Return the HTML content of the current page; avoid the full\_page option unless head or script tags are required. | `browser` |
| | `scraping_browser_scroll` | Scroll to the bottom of the current page in the scraping-browser session. | `browser` |
| | `scraping_browser_scroll_to_ref` | Scroll the page until the element referenced in the ARIA snapshot is in view. | `browser` |
| | `web_data_chatgpt_ai_insights` | Send a prompt to ChatGPT and get back AI-generated insights. Returns structured answer text, citations, recommendations, and markdown. Useful for GEO and LLM as a judge. | `geo` |
| | `web_data_grok_ai_insights` | Send a prompt to Grok and get back AI-generated insights. Returns structured answer text in markdown format. Useful for GEO and LLM as a judge. | `geo` |
| | `web_data_perplexity_ai_insights` | Send a prompt to Perplexity and get back AI-generated insights. Returns structured answer text in markdown format. Useful for GEO and LLM as a judge. | `geo` |
| | `web_data_npm_package` | Quickly read structured npm package data. Requires a valid npm package name (e.g., `@brightdata/sdk`). Often faster and more reliable than scraping. | `code` |
| | `web_data_pypi_package` | Quickly read structured PyPI package data. Requires a valid PyPI package name (e.g., `langchain-brightdata`). Often faster and more reliable than scraping. | `code` |
## How do I search datasets by filter with the Bright Data MCP server?
Use `list_dataset_fields` and `search_dataset` to find many records that match your criteria, as opposed to the `web_data_*` tools, which fetch one record by URL. The search runs directly against the dataset with no snapshot step and returns up to 10 records per call. Under the hood, `list_dataset_fields` calls the [Get dataset metadata](/api-reference/marketplace-dataset-api/get-dataset-metadata) endpoint and `search_dataset` calls the [Search dataset](/api-reference/marketplace-dataset-api/search-dataset) endpoint of the Marketplace Dataset API.
Three datasets are searchable:
| `dataset_id` | Dataset |
| :- | :- |
| `gd_l1viktl72bvl7bjuj0` | LinkedIn people profiles |
| `gd_me5ppxjr2ge6icjuh0` | LinkedIn people profiles (contact-enriched) |
| `gd_l1vikfnt1wgvvqz95w` | LinkedIn company information |
The workflow is two calls:
1. Call `list_dataset_fields` with a `dataset_id`. It returns each filterable field's name, type and description.
2. Call `search_dataset` with the same `dataset_id` and a `filter`.
A filter is a tree with a maximum nesting depth of three. Each node is either a group, `{"operator": "and" | "or", "filters": [...]}`, or a leaf, `{"name": ..., "operator": ..., "value": ...}`. See [filter syntax](/api-reference/marketplace-dataset-api/filter-syntax) for the full list of leaf operators and their behavior.
For example, this filter matches profiles in the United States whose position mentions data engineering:
```json theme={null}
{
"operator": "and",
"filters": [
{ "name": "country_code", "operator": "=", "value": "US" },
{ "name": "position", "operator": "includes", "value": "data engineer" }
]
}
```
The response contains `hits` (the matching records), `total_hits` and `took` (search time in milliseconds). To page through more than 10 results, set `sort` to `"default"` or a custom sort like `[{"timestamp": "asc"}]`, then pass the response's `search_after` cursor into the next call. Set `sort` to `"random"` to sample records instead.
# TOON format for Bright Data MCP
Source: https://docs.brightdata.com/products/mcp-server/toon
Use Token-Oriented Object Notation (TOON) with the Bright Data MCP server (60+ tools) to cut token usage on large web data payloads without losing structure.
You might be eligible for our Startup Program. Get fully funded access to the infrastructure you're reading about right now (up to \$20K value).
## What is TOON?
**Token-Oriented Object Notation (TOON)** is a compact, human-readable encoding of the JSON data model designed specifically for LLM input. It minimizes token usage while maintaining full data fidelity, making it an excellent choice for reducing costs when processing large amounts of structured data.
TOON combines YAML's indentation-based structure with a CSV-style tabular layout for uniform arrays, achieving **30-60% token reduction** compared to standard JSON.
TOON is a **translation layer** - use JSON programmatically in your code, and encode it as TOON when passing data to LLMs.
***
## How TOON Works
### The Problem with JSON
Standard JSON repeats field names for every record, which is token-expensive:
```json theme={null}
{
"products": [
{ "id": 1, "name": "Laptop", "price": 999 },
{ "id": 2, "name": "Mouse", "price": 29 },
{ "id": 3, "name": "Keyboard", "price": 79 }
]
}
```
### The TOON Solution
TOON declares fields once and streams data as compact rows:
```yaml theme={null}
products[3]{id,name,price}:
1,Laptop,999
2,Mouse,29
3,Keyboard,79
```
* `[3]` declares the array length (helps detect truncation)
* `{id,name,price}` declares field names once
* Each row contains comma-separated values
***
## Token Savings Example
**235 tokens**
Verbose with repeated keys
**106 tokens**
\~55% reduction
```json JSON (235 tokens) theme={null}
{
"context": {
"task": "Product catalog",
"category": "Electronics"
},
"items": [
{ "id": 1, "name": "Laptop", "price": 999, "stock": 50, "active": true },
{ "id": 2, "name": "Mouse", "price": 29, "stock": 200, "active": true },
{ "id": 3, "name": "Keyboard", "price": 79, "stock": 150, "active": false }
]
}
```
```yaml TOON (106 tokens) theme={null}
context:
task: Product catalog
category: Electronics
items[3]{id,name,price,stock,active}:
1,Laptop,999,50,true
2,Mouse,29,200,true
3,Keyboard,79,150,false
```
***
## When to Use TOON
**TOON excels with flat, uniform arrays of objects** - data where items share the same structure across all records.
### Ideal Use Cases
| Data Type | Example | Token Savings |
| - | - | - |
| Product listings | E-commerce catalogs | 40-60% |
| User records | Flat profile data | 35-55% |
| Log entries | Structured logs | 45-60% |
| Search results | SERP data (flat) | 40-55% |
| Simple tables | Spreadsheet-like data | 50-65% |
***
## When NOT to Use TOON
**TOON performs poorly on deeply nested or non-uniform data structures.** This is particularly relevant for Bright Data MCP responses.
### Avoid TOON When:
Complex objects with multiple nesting levels don't benefit from TOON's tabular format. JSON-compact often uses fewer tokens for such structures.
**Example:** LinkedIn profiles with nested experience, education, and skills objects.
When array items have different fields or optional properties, TOON's header-based approach breaks down.
**Example:** Mixed product types with varying attributes.
Most Bright Data web data endpoints return **richly nested JSON** with hierarchical relationships. TOON won't provide meaningful savings here.
**Example:** `web_data_amazon_product`, `web_data_linkedin_person_profile`, etc.
***
## TOON with Bright Data MCP
### Understanding the Limitation
Bright Data's MCP server returns structured data from platforms like Amazon, LinkedIn, Instagram, and more. This data is typically **nested and hierarchical**, making it unsuitable for TOON optimization.
Connect via Server-Sent Events:
```http theme={null}
https://mcp.brightdata.com/mcp?token=YOUR_API_TOKEN
```
Configure in your MCP client:
```json theme={null}
{
"mcpServers": {
"Bright Data": {
"command": "npx",
"args": ["@brightdata/mcp"],
"env": {
"API_TOKEN": "YOUR_API_KEY",
"WEB_UNLOCKER_ZONE": "my_zone_name",
"BROWSER_ZONE": "my_browser_zone",
"PRO_MODE": "true",
"GROUPS": "browser,advanced_scraping",
"TOOLS": "web_data_linkedin_person_profile,web_data_amazon_product"
}
}
}
}
```
### When TOON Can Help
While most Bright Data endpoints return nested data, there are scenarios where TOON can be beneficial:
Post-process MCP responses to extract flat arrays before encoding to TOON.
When using `scrape_batch` or `search_engine_batch`, the URL/content pairs can be flattened.
Use the `extract` tool with a prompt that requests flat, tabular data.
***
## Quick Start with TOON
### How to install TOON
```bash npm theme={null}
npm install @toon-format/toon
```
```bash pnpm theme={null}
pnpm add @toon-format/toon
```
```bash yarn theme={null}
yarn add @toon-format/toon
```
### How to encode and decode data
```typescript theme={null}
import { encode, decode } from '@toon-format/toon'
// Your flat data (ideal for TOON)
const data = {
products: [
{ id: 1, name: 'Laptop', price: 999 },
{ id: 2, name: 'Mouse', price: 29 },
{ id: 3, name: 'Keyboard', price: 79 }
]
}
// Encode to TOON for LLM input
const toonData = encode(data)
console.log(toonData)
// Output:
// products[3]{id,name,price}:
// 1,Laptop,999
// 2,Mouse,29
// 3,Keyboard,79
// Decode back to JSON (lossless round-trip)
const jsonData = decode(toonData)
```
### How to use the CLI
```bash theme={null}
# Convert JSON to TOON
npx @toon-format/cli input.json -o output.toon
# Convert TOON back to JSON
npx @toon-format/cli input.toon -o output.json
```
***
## Practical Example: Using TOON with MCP Client
This example demonstrates a complete workflow: connecting to Bright Data's MCP server, fetching data, flattening the nested response, and converting it to TOON format for token-efficient LLM processing.
### Full MCP Client Example
```typescript theme={null}
import { Client } from '@modelcontextprotocol/sdk/client/index.js'
import { SSEClientTransport } from '@modelcontextprotocol/sdk/client/sse.js'
import { encode } from '@toon-format/toon'
// Initialize MCP client with Bright Data (ecommerce group for Amazon tools)
const transport = new SSEClientTransport(
new URL('https://mcp.brightdata.com/sse?token=YOUR_API_TOKEN&groups=ecommerce')
)
const client = new Client({
name: 'toon-example-client',
version: '1.0.0'
})
await client.connect(transport)
// Fetch Amazon product data using MCP (with timeout for slow responses)
const result = await client.callTool(
{
name: 'web_data_amazon_product',
arguments: {
url: 'https://www.amazon.com/dp/B0EXAMPLE123'
}
},
undefined,
{ timeout: 120000 } // 2 minutes timeout
)
// The MCP response is an array with nested product data
const mcpResponse = JSON.parse(result.content[0].text)
console.log('Raw MCP Response:', JSON.stringify(mcpResponse, null, 2))
// [
// {
// "title": "Ceramic Vase Set of 3...",
// "final_price": 29.99,
// "initial_price": 29.99,
// "currency": "USD",
// "rating": 4.5,
// "reviews_count": 1250,
// "seller_name": "HomeDecorStore",
// "brand": "Lyeec",
// "is_available": true,
// "asin": "B0G13HFJNB",
// ...
// }
// ]
// Flatten the nested data for TOON optimization
// Map actual MCP field names to flat structure
function flattenProduct(product: any) {
return {
title: product.title,
price: product.final_price,
original_price: product.initial_price,
currency: product.currency,
rating: product.rating,
reviews: product.reviews_count,
seller: product.seller_name,
brand: product.brand,
in_stock: product.is_available
}
}
// Response is an array - flatten all products
const flatProducts = mcpResponse.map(flattenProduct)
// Encode to TOON - now it's efficient!
const toonOutput = encode({ products: flatProducts })
console.log('TOON Output:', toonOutput)
// products[1]{title,price,original_price,currency,rating,reviews,seller,brand,in_stock}:
// Ceramic Vase Set of 3...,29.99,29.99,USD,4.5,1250,HomeDecorStore,Lyeec,true
// Use the TOON-encoded data in your LLM prompt
const prompt = `Analyze this product data and suggest pricing strategy:\n\n${toonOutput}`
```
### Batch Processing Multiple Products
For larger datasets, the token savings become even more significant:
```typescript theme={null}
import { Client } from '@modelcontextprotocol/sdk/client/index.js'
import { SSEClientTransport } from '@modelcontextprotocol/sdk/client/sse.js'
import { encode } from '@toon-format/toon'
const transport = new SSEClientTransport(
new URL('https://mcp.brightdata.com/sse?token=YOUR_API_TOKEN&groups=ecommerce')
)
const client = new Client({ name: 'batch-example', version: '1.0.0' })
await client.connect(transport)
// Fetch multiple products
const productUrls = [
'https://www.amazon.com/dp/B0EXAMPLE1',
'https://www.amazon.com/dp/B0EXAMPLE2',
'https://www.amazon.com/dp/B0EXAMPLE3'
]
const results = await Promise.all(
productUrls.map(async (url) => {
const result = await client.callTool(
{ name: 'web_data_amazon_product', arguments: { url } },
undefined,
{ timeout: 120000 }
)
// MCP returns an array, get the first item
return JSON.parse(result.content[0].text)[0]
})
)
// Flatten all products for TOON using actual MCP field names
const flattenedProducts = results.map(p => ({
title: p.title,
price: p.final_price,
currency: p.currency,
rating: p.rating,
reviews: p.reviews_count,
seller: p.seller_name,
brand: p.brand,
in_stock: p.is_available
}))
// Convert to TOON - massive token savings with multiple items!
const toonData = encode({ products: flattenedProducts })
console.log(toonData)
// products[3]{title,price,currency,rating,reviews,seller,brand,in_stock}:
// Wireless Mouse,29.99,USD,4.5,1250,TechGear,Logitech,true
// Gaming Keyboard,89.99,USD,4.7,890,KeyMaster,Razer,true
// USB Hub,19.99,USD,4.2,2100,PortPlus,Anker,false
// Token comparison for 3 products:
// - JSON: ~180 tokens
// - TOON: ~65 tokens (64% savings!)
```
### Creating a Reusable Flattener
For production use, create a reusable utility with the correct MCP field mappings:
```typescript theme={null}
import { encode } from '@toon-format/toon'
// Generic flattener for common MCP data structures
// These map the actual field names returned by Bright Data MCP
const flatteners = {
amazon_product: (p: any) => ({
title: p.title,
price: p.final_price,
original_price: p.initial_price,
currency: p.currency,
rating: p.rating,
reviews: p.reviews_count,
seller: p.seller_name,
brand: p.brand,
in_stock: p.is_available
}),
linkedin_profile: (p: any) => ({
name: p.full_name,
headline: p.headline,
location: p.city,
country: p.country,
connections: p.connections_count,
followers: p.followers_count,
company: p.current_company_name,
position: p.position
}),
instagram_post: (p: any) => ({
caption: p.description?.substring(0, 100),
likes: p.likes,
comments: p.comments,
posted: p.upload_date,
author: p.channel
})
}
// Helper to process MCP response and convert to TOON
function mcpToToon(
data: T | T[],
flattener: (item: T) => Record,
key: string = 'items'
): string {
const items = Array.isArray(data) ? data : [data]
const flattened = items.map(flattener)
return encode({ [key]: flattened })
}
// Usage example
const result = await client.callTool(
{ name: 'web_data_amazon_product', arguments: { url: '...' } },
undefined,
{ timeout: 120000 }
)
const mcpData = JSON.parse(result.content[0].text)
const toonOutput = mcpToToon(mcpData, flatteners.amazon_product, 'products')
```
***
## Summary
* Flat, uniform arrays
* Tabular data structures
* Large datasets with repeated schemas
* Post-processed, flattened MCP data
* Deeply nested objects
* Non-uniform data structures
* Direct MCP endpoint responses
* Complex hierarchical data
**Bottom Line:** TOON is a powerful tool for token optimization, but it's designed for flat, uniform data. Bright Data MCP responses are typically nested, so apply TOON only after flattening the data to a tabular structure.
***
## Where to learn more
Official format specification
NPM package documentation
Interactive playground and docs
# MCP usage examples
Source: https://docs.brightdata.com/products/mcp-server/usage-examples
Browse community demos and reference projects built on the Bright Data MCP server (60+ tools), covering AI agents, scraping pipelines and integrations.
You might be eligible for our Startup Program. Get fully funded access to the infrastructure you're reading about right now (up to \$20K value).
1. AI voice agent that closed 4 deals & made \$596 overnight 🤑
* [GitHub Repo](https://github.com/llSourcell/my_ai_intern)
* [YouTube Demo](https://www.youtube.com/watch?v=YGzT3sVdwdY)
***
2. LangGraph with mcp-adapters demo
* [GitHub Repo](https://github.com/techwithtim/BrightDataMCPServerAgent)
* [YouTube Demo](https://www.youtube.com/watch?v=6DXuadyaJ4g)
***
3. Researcher Agent built with Google ADK that is connected to Bright Data's MCP to fetch real-time data
* [GitHub Repo](https://github.com/MeirKaD/MCP_ADK)
* [YouTube Demo](https://www.youtube.com/watch?v=r7WG6dXWdUI)
***
4. Multi-Agent job finder using Bright Data MCP and TypeScript from SCRATCH
* [GitHub Repo](https://github.com/bitswired/jobwizard)
* [YouTube Demo](https://www.youtube.com/watch?v=45OtteCGFiI)
***
5. Replacing 3 MCP servers with our MCP server to avoid getting blocked 🤯
* [YouTube Demo](https://www.youtube.com/watch?v=0xmE0OJrNmg)
***
6. Scrape ANY Website In Realtime With This Powerful AI MCP Server
* [YouTube Demo](https://www.youtube.com/watch?v=bL5JIeGL3J0)
***
7. Usage example with Gemini CLI
* [YouTube Demo](https://www.youtube.com/watch?v=FE1LChbgFEw)
# Build a scraper with the AI Agent
Source: https://docs.brightdata.com/products/scraper-studio/ai-agent
Use Bright Data Scraper Studio's AI Agent to build a custom web scraper from a natural-language prompt in about 10 minutes, no coding required.
Bright Data Scraper Studio's AI Agent builds a custom web scraper from a natural-language description. Give it a target URL and describe the data you want; the agent generates an output schema, writes the scraper code, and hands you a runnable scraper you can trigger by API, manually, or on a schedule. This tutorial walks through the full flow.
**Time to complete:** about 10 minutes
Prefer the terminal? Use the [Bright Data CLI](/products/scraper-studio/build-with-the-cli) to build the same scraper from any shell or coding-agent embedded terminal (Claude Code, Cursor, Codex).
## Prerequisites
* A Bright Data account ([sign up free](https://brightdata.com/?hs_signup=1\&utm_source=docs))
* The URL of the website you want to scrape
## Build your first scraper with the AI Agent
In the Bright Data control panel, click **Scrapers** in the left menu and open **Scraper Studio**.
Paste the URL of the page you want to scrape into the chat input. Along with the URL, add any context that helps the AI build a more accurate scraper on the first try. The more context you provide, the better the generated code.
Useful context to include:
* **Specific fields** you need: "I need price, title, and stock status"
* **Where the data lives** on the page: "prices are in the product detail panel, not the listing page"
* **Actions required** to reach the data: "click 'Show more' to load full descriptions"
* **CSS selectors**, if you know them: `.product-price span.amount`
* **Page load behavior**, if the site is slow or lazy-loads content: "results load dynamically, give it extra time"
> **Expected result:** the AI Agent acknowledges the URL and may ask one or two clarifying questions about the data you want.
Respond in plain language.
> **Expected result:** the AI Agent generates a schema, a structured list of fields with data types that will become your scraper's output.
Read through the generated schema. You have four options:
* **Approve:** click Approve to accept the schema as-is
* **Decline:** type feedback in the chat (for example, "Remove the image field and add a rating field") and the AI regenerates the schema
* **Edit inline:** modify the schema directly without going back to the chat
* **Upload your own schema:** bring your own schema file; download the example file to see the correct format
Inline editing options:
* **Edit a field** (pencil icon): change a field's name or data type
* **Delete a field** (trash icon): remove fields you do not need
* **Add a field** (plus button): add new fields to the schema
Additional controls:
* **Start from scratch:** clears every field so you can build the schema manually from an empty state
* **Reset the schema:** discards inline changes and returns to the original AI-generated schema
Click **Approve** when you are ready.
> **Expected result:** once approved, the AI Agent starts generating the scraper code.
The AI writes the full scraper, including extraction logic, navigation handling, data validation, and error handling. This takes a few minutes.
> **Expected result:** a confirmation popup appears indicating your scraper is ready.
Click **Try it out** to open the Initiate Manually page. Review the collection settings and click **Start** to begin data collection.
You can also choose an alternative initiation method:
* **[Initiate by API](/api-reference/scraper-studio-api/Trigger_a_scraper_for_batch_collection_method):** trigger the scraper programmatically without opening the control panel
* **Schedule:** run the scraper on an hourly, daily, weekly, or custom interval
> **Expected result:** the scraper collects data. Monitor progress from the Runs dashboard and download results in JSON, NDJSON, CSV, or XLSX once the job completes.
After the scraper is ready, use the [Self-Healing tool](/products/scraper-studio/self-healing-tool) to make plain-language changes, or open the code in the Bright Data Scraper Studio IDE to edit it directly.
## **Resume a previous AI chat**
Use **My AI Chats** on the **Scraper Studio** chat page to reopen AI Agent conversations that are still waiting for your input or have finished creating a scraper.
**My AI Chats** can show chats with different statuses:
* **Pending answer**: the AI Agent is waiting for your response. Open the chat to continue the scraper creation flow.
* **Done**: the scraper is ready. Open the chat to access the created scraper. After you open it, the chat is removed from **My AI Chats**.
Once a scraper is created, you cannot continue editing it from the same AI chat. To change an existing scraper, use the [**Self-Healing**](https://docs.brightdata.com/datasets/scraper-studio/self-healing-tool) tool.
## What can the AI Agent build?
Reach for the Bright Data Scraper Studio AI Agent when the data you need isn't already in the [Datasets Marketplace](/datasets/marketplace). For sites Bright Data already covers (Amazon, Walmart, LinkedIn, eBay), the marketplace ships pre-built scrapers that are faster and cheaper than building your own. The AI Agent shines on regional ecommerce, B2B catalogs, niche verticals and any site where no marketplace scraper exists.
The AI Agent builds one of five scraper types. Pick the one that matches what you have and what you need.
| Scraper type | What you provide | What you get | Page visits per input |
| - | - | - | - |
| **PDP** | List of product URLs | Full per-product detail | 1 |
| **Discovery** | Category / listing URL | Listing-level rows (title, price, rank) | 1 |
| **Discovery + PDP** | Category / listing URL | Full per-product detail for every item | 1 + N |
| **Search** | Keyword (+ optional country) | Either Discovery or Discovery + PDP shape | 1 + M |
| **Sitemap** | Domain or `sitemap.xml` URL | Full per-page detail for every URL in the sitemap | 1 + N |
**Not a crawler.** Don't pass a homepage and ask for "everything." Each scraper is scoped to one data shape.
### When do I use a PDP scraper?
Use a PDP scraper when you have a list of specific product URLs and need full per-product detail. Each input URL produces one row.
**Example:** 100 product URLs from a regional retailer like [dm.de](https://www.dm.de) → 100 rows with title, price, availability, images and any other field you defined in the schema.
Paste this into the AI Agent chat:
```text Prompt theme={null}
Build a PDP scraper for dm.de. For each product URL I provide, extract the product title, price, availability, brand, rating and all product image URLs. Return one row per input URL.
```
### When do I use a Discovery scraper?
Use a Discovery scraper when you need an overview of items on a single listing page and can live without per-product detail. Each input URL produces N rows where N equals the number of items in the listing.
**Example:** A category page like `https://www.dm.de/baby-und-kind` → 30 rows with title, price, rating and listing position. No description, no full image gallery.
Paste this into the AI Agent chat:
```text Prompt theme={null}
Build a Discovery scraper for the dm.de category page https://www.dm.de/baby-und-kind. Return one row per item shown in the listing with title, price, rating and listing position. Do not open the individual product pages.
```
### When do I use a Discovery + PDP scraper?
Use a Discovery + PDP scraper when you need full per-product detail for an entire category. Each input URL produces N rows with the full PDP shape.
This is the most expensive type. One category URL with 200 products costs roughly 200 times a single PDP scrape.
**Example:** A category page on a regional retailer like [decathlon.fr](https://www.decathlon.fr) → full per-product detail for every item in that category.
Paste this into the AI Agent chat:
```text Prompt theme={null}
Build a Discovery + PDP scraper for a category page on decathlon.fr (for example a sport category like running shoes). Find every product in the category, open each product page, and extract full detail: title, price, availability, description, rating and image URLs. Return one row per product.
```
### When do I use a Search scraper?
Use a Search scraper when you do not have specific URLs. Provide a keyword and optionally a country; the AI Agent picks Discovery or Discovery + PDP shape based on whether you asked for listing-level fields or full PDP detail.
**Example:** keyword "brake pads" on a B2B parts site like [autodoc.de](https://www.autodoc.de) → matching product results in either listing-level or full PDP shape.
Paste this into the AI Agent chat:
```text Prompt theme={null}
Build a Search scraper for autodoc.de. For the keyword "brake pads" in Germany, return matching products with title, price, brand and product URL. No need to open each product page.
```
### When do I use a Sitemap scraper?
Use a Sitemap scraper when you need data from many pages on a site and the site publishes those page URLs in an XML sitemap. The sitemap lets the scraper discover URLs directly instead of clicking through category pages, pagination or search results, so it fits large ecommerce sites, marketplaces, catalogs, blogs and documentation sites.
Provide a domain or a sitemap URL. Each sitemap produces N discovered page URLs, and the scraper visits each relevant page to collect the fields you defined in the schema.
**Example:** a domain like [dm.de](https://www.dm.de) or its sitemap at `https://www.dm.de/sitemap.xml` → discover every product URL in the sitemap → visit each product page → collect full product detail.
Paste this into the AI Agent chat:
```text Prompt theme={null}
Build a Sitemap scraper for dm.de. Collect all product URLs from the sitemap, visit each product page, and extract product name, price, SKU, description, image URL and availability. Return one row per product.
```
If you already have the sitemap URL, point the AI Agent straight at it:
```text Prompt theme={null}
Build a scraper using this sitemap: https://www.dm.de/sitemap.xml. Extract all product page URLs, visit each product page, and collect the product detail fields.
```
## Frequently asked questions
Yes. Every scraper the AI Agent generates can be opened in the Bright Data Scraper Studio IDE and edited directly. If you prefer not to write code, use the [Self-Healing tool](/products/scraper-studio/self-healing-tool) to request changes in plain language.
The AI Agent generates scrapers that run on Bright Data's proxy and unblocking infrastructure, which handles most anti-bot defenses. For sites that require a logged-in session, build the scraper in the IDE and use `set_session_cookie()` or the authentication pattern that matches the target site.
The AI Agent relies on the context you give it. If the output is off, decline the schema and add more specifics about field names, selectors, or the exact page section where the data lives. You can also use the [Self-Healing tool](/products/scraper-studio/self-healing-tool) to refine a generated scraper after the fact.
## Related
Build the same scraper from the terminal or any coding agent
Build a scraper by writing JavaScript directly
Update a generated scraper with plain-language prompts
# Scraper Studio Auto Self-Healing
Source: https://docs.brightdata.com/products/scraper-studio/auto-self-healing
Configure Auto Self-Healing in Bright Data Scraper Studio to fix a scraper automatically when its success rate drops or 1 of 6 supported errors occurs.
Use the **Auto Self-Healing** tab to let Bright Data Scraper Studio try to fix a scraper on its own when collection quality drops or a supported scraper error occurs. Auto Self-Healing jobs do not cost credits.
Auto Self-Healing is configured per scraper. Once enabled, Bright Data Scraper Studio monitors that scraper's runs and starts a healing job whenever the trigger rules you set are met.
Auto Self-Healing runs without you asking. To fix a scraper on demand from a plain-language prompt instead, use the [Self-Healing tool](/products/scraper-studio/self-healing-tool) in the Scraper Studio IDE.
## How do I open Auto Self-Healing settings?
1. Open **My Scrapers** in Bright Data Scraper Studio.
2. Select the scraper you want to configure.
3. Open the **Auto Self-Healing** tab.
4. Configure the general settings, trigger rules and notifications.
5. Click **Save**.
Saved settings apply to future runs of that scraper. Runs that already completed are not reprocessed.
## Which general settings control Auto Self-Healing?
| Setting | Options | What it does |
| - | - | - |
| **Enable Auto self healing** | On, Off | Turns Auto Self-Healing on or off for this scraper. When On, Bright Data Scraper Studio can start healing jobs as soon as the trigger rules are met. |
| **Auto save code** | On, Off | Controls whether Bright Data Scraper Studio saves the code the healing process generates. When On, any successful healing job saves the newly extracted code to production and updates the collector's template version. When Off, the healing job still runs but the generated code is not saved, and the users you select under **Notifications** get an email so they can review the suggested fix before saving it. |
**Auto save code** writes straight to production. A successful healing job saves the new code and bumps the collector's template version with no human review, so the next run collects with AI-written code. Turn it on only when you accept that. Leave it Off to keep a review step before any fix reaches production.
## Which trigger rules start a healing job?
Trigger rules decide when Auto Self-Healing starts for the scraper you are configuring. **Success Rate Threshold** and **Minimum Inputs Threshold** are required fields.
| Trigger rule | Default | What it controls |
| - | - | - |
| **Success Rate Threshold (%)** | `40` | The success rate below which Auto Self-Healing triggers. At `40`, a healing job can start once the scraper's success rate falls below 40 percent. |
| **Trigger Frequency** | Per Day | Whether Bright Data Scraper Studio checks once per day (**Per Day**) or on every job run (**Per Job**). |
| **Minimum Inputs Threshold** | `10` | The minimum number of job inputs required before Auto Self-Healing triggers. A run smaller than this never starts a healing job, so a handful of failures in a tiny run does not count as a quality drop. |
| **Cooldown Period (hours)** | `3` | The minimum hours to wait between consecutive Auto Self-Healing triggers for this scraper. |
| **Max Auto-Healing Jobs Per Day** | `8` | The maximum number of Auto Self-Healing jobs allowed for this scraper in one day. |
A healing job that fails still counts against **Max Auto-Healing Jobs Per Day** and still starts the **Cooldown Period**.
## Which error codes trigger Auto Self-Healing?
Auto Self-Healing starts only for these six scraper error codes:
* `crawl_error`
* `parse_error`
* `bad_cmd_arg`
* `dead_page`
* `bad_input`
* `bad_navigate`
A scraper that fails with any other error code does not start Auto Self-Healing, even when every trigger rule is met. For what each code means and how to resolve it by hand, see [Scraper Studio error codes](/products/scraper-studio/error-codes).
## Where do I see status and notifications?
The **Status** section of the **Auto Self-Healing** tab reports this scraper's healing-job activity. Before any healing job has run it reads `No auto healing jobs were triggered`.
The **Notifications** section has a **Select users to notify** picker listing the users on your account. Selected users get email when Auto Self-Healing is triggered, when it completes and when it needs attention. Add at least one user when **Auto save code** is Off, because the review email is the only way a suggested fix reaches a person.
## FAQ
### How is Auto Self-Healing different from the Self-Healing tool?
Auto Self-Healing runs on its own against trigger rules you configure per scraper. The [Self-Healing tool](/products/scraper-studio/self-healing-tool) is manual: you open a scraper in the Scraper Studio IDE, describe the fix in plain language and review the diff before accepting it. Auto Self-Healing needs no prompt.
### Does Auto Self-Healing change my scraper without asking?
Only when **Auto save code** is On. With **Auto save code** Off, Bright Data Scraper Studio still runs the healing job but does not save the generated code, and emails the users selected under **Notifications** to review the suggested fix first.
### Why did Auto Self-Healing not start after my scraper failed?
Check the failure's error code first. Auto Self-Healing starts only for the six error codes listed on this page, so a scraper that fails with any other code never triggers it. If the code is supported, check the other trigger rules set on the scraper: the run may have had fewer inputs than **Minimum Inputs Threshold**, or the scraper may still be inside its **Cooldown Period** or have already hit **Max Auto-Healing Jobs Per Day**. Failed healing jobs count toward both limits.
### Can I configure Auto Self-Healing through the API?
You can turn it on. Send the scraper ID and trigger rules to the [Enable Auto Self-Healing](/api-reference/scraper-studio-api/ai-flow/enable-auto-self-healing) endpoint of the [AI Flow API](/api-reference/scraper-studio-api/ai-flow/overview). The endpoint takes the same settings as the **Auto Self-Healing** tab, including `sr_threshold`, `trigger_frequency`, `min_inputs_threshold`, `auto_save`, `cooldown_period`, `max_jobs_per_day` and `notification_emails`.
Changing those settings afterward is control panel work. The API creates them once per scraper and has no endpoint to read, update, delete or disable them. To start a self-healing job on demand instead, use [Trigger Self-Healing](/api-reference/scraper-studio-api/ai-flow/trigger-self-healing).
### Do the settings affect runs that already finished?
No. Auto Self-Healing settings apply to future runs of the scraper. Runs that already completed are not changed or reprocessed.
## Related
Fix a scraper on demand from a plain-language prompt
What each scraper error code means and how to resolve it
Start a self-healing job from the AI Flow API
Run a production scraper and configure delivery
# Web scraping basics in Scraper Studio
Source: https://docs.brightdata.com/products/scraper-studio/basics-of-web-scraping
Understand how Scraper Studio IDE scrapers are structured, from inputs and interaction code to parsing, stages, workers, and output records.
Bright Data Scraper Studio IDE scrapers are built from a few core parts: **inputs**, **interaction code**, **parser code**, **stages**, **workers**, and **output records**.
This page explains how those parts work together so you can understand, edit, and debug scraper code with confidence.
For a step-by-step build walkthrough, see [**Develop a scraper with the IDE**](/products/scraper-studio/develop-a-scraper).
## Prerequisites
* Basic JavaScript familiarity (variables, functions, async control flow)
* An active [Bright Data account](https://brightdata.com/)
## **Basic scraper flow**
A Scraper Studio IDE scraper usually follows this flow:
1. The scraper receives an **input**, such as a URL, keyword, location, or custom value.
2. **Interaction code** uses that input to open a page, send a request, click, scroll, paginate, or create the next stage.
3. **Parser code** extracts structured data from the loaded page or response.
4. The scraper saves records to the output dataset with `collect()` or `set_lines()`.
5. The final output is delivered according to the scraper’s delivery preferences.
## **Inputs**
Inputs are the values passed into a scraper run.
A scraper might use:
* `url`, for page-based scraping
* `keyword`, for search or discovery flows
* `location`, for location-based searches
* `country`, `date`, `category`, or any other custom field
The available input fields are defined in the scraper’s [**input schema**](/products/scraper-studio/input-and-output-schema#what-is-the-input-schema).
Interaction code reads these values from the `input` object:
```js theme={null}
navigate(input.url);
```
For a keyword-based scraper:
```js theme={null}
navigate(`https://example.com/search?q=${input.keyword}`);
```
A scraper does not always need external input. For example, a scraper can use a hardcoded URL if it always collects from the same page.
## **Interaction code**
Interaction code controls how the scraper reaches the data.
It can:
* Navigate to a page
* Send HTTP requests
* Wait for page elements
* Click buttons
* Type into forms
* Scroll through a page
* Handle pagination
* Create additional crawl stages
A simple interaction flow looks like this:
```js theme={null}
navigate(input.url);
wait('.product-title');
const data = parse();
collect(data);
```
In this example:
* `navigate(input.url)` opens the page from the input.
* `wait('.product-title')` waits until the expected page element appears.
* `parse()` runs the parser code.
* `collect(data)` saves the extracted record.
Interaction code is responsible for **getting to the right page or response**. It should not contain most of the extraction logic. That belongs in parser code.
## **Parser code**
Parser code extracts structured fields from the page HTML or response.
Parser code commonly uses Cheerio, a jQuery-like API, to read page elements:
```js theme={null}
return {
title: $('h1').text_sane(),
price: $('.price').text_sane(),
availability: $('.stock-status').text_sane(),
};
```
The parser returns a JavaScript object. That object becomes the structured data your scraper can collect.
For example, a parser can return product data, profile data, listings, article content, or links that the interaction code will use in the next stage.
## `parse() `**and** `collect()`
`parse()` and `collect()` connect interaction code, parser code, and final output.
### `parse()`
`parse()` runs the parser code for the current page or response and returns its result.
Example:
```js theme={null}
let data = parse();
```
### `collect()`
`collect()` appends one record to the output dataset.
Example:
```js theme={null}
collect({
title: data.title,
price: data.price,
url: location.href,
});
```
## **Multi-stage scrapers**
Some scrapers need more than one step to reach the final data.
For example:
1. Start from a search page.
2. Discover result pages.
3. Open each result page.
4. Extract final details.
Scraper Studio supports this pattern with **stages**.
A stage is a separate crawl step. `next_stage()` sends new input values to the next stage.
Example:
```js theme={null}
navigate(`https://example.com/search?q=${input.keyword}`);
const results = parse().results;
for (const result of results) {
next_stage({ url: result.url });
}
```
Then the next stage can use the new `url` input:
```js theme={null}
navigate(input.url);
const data = parse();
collect(data);
```
Multi-stage scrapers are useful for:
* Search results → detail pages
* Category pages → product pages
* Listing pages → profile pages
* Pagination flows
* Discovery workflows where one page creates many child pages
Scraper Studio can run stages across workers, so this pattern is usually more scalable than processing every page serially in one long script.
## **Parent and child crawls**
When a scraper uses `next_stage()`, it creates a parent-child relationship between crawls.
For example:
* The search page is the **parent** crawl.
* Each result page created from it is a **child** crawl.
This relationship is useful when debugging. In the crawl inspector, you can see which input or page created another page, inspect child pages, and trace where failures happened in a multi-stage flow.
## **Code workers and Browser workers**
Scraper Studio supports two worker types.
### **Code worker**
A **Code worker** uses HTTP requests and raw responses.
Use a Code worker when:
* The data is available in the raw HTML
* The data is available from a public JSON endpoint
* The page does not require browser interaction
* You want faster and more cost-efficient scraping
Code workers cannot click, scroll, type, or run browser-only functions.
### **Browser worker**
A **Browser worker** uses a real headless browser.
Use a Browser worker when:
* The page renders data with JavaScript
* You need to click, scroll, type, or interact with the page
* You need to wait for elements to appear
* You need to capture browser network traffic
* The site requires browser-like behavior
Start with a Code worker when possible. Switch to a Browser worker when the target data is not available in the raw response or when browser interaction is required.
For the complete comparison, see [**Worker types**](/products/scraper-studio/worker-types).\\
## How does Scraper Studio handle blocking and CAPTCHAs?
Scraping at scale can trigger site defenses such as:
* IP blocking
* Rate limits
* CAPTCHAs
* Fingerprinting
* Bot detection
Scraper Studio runs on Bright Data’s proxy and unblocking infrastructure, so you do not need to manage proxy rotation, sessions, or retry logic yourself.
Depending on the scraper configuration and worker type, Scraper Studio can:
* Route requests through Bright Data proxy infrastructure
* Retry blocked requests
* Use browser-like fingerprints for Browser workers
* Support CAPTCHA-solving workflows with `solve_captcha()` when applicable
Your scraper code should focus on reaching the right pages and extracting the required data. Bright Data handles the access infrastructure behind the scenes.
## **How schema fits into scraper structure**
The scraper code and schema work together.
### **Input schema**
The input schema defines what values the scraper can receive.
Example input fields:
```js theme={null}
{
"url": "https://example.com/product/1",
"country": "US"
}
```
Interaction code reads those values from `input`.
### **Output schema**
The output schema defines what fields the scraper returns.
It is usually generated from the records passed to `collect()`.
Example collected record:
```js theme={null}
collect({
title: data.title,
price: data.price,
availability: data.availability,
});
```
Those fields become part of the output schema.
For more details, see [**Input and output schema**](/products/scraper-studio/input-and-output-schema).
## **Common scraper patterns**
### **Single-page scraper**
Use this pattern when each input maps to one final page.
Example:
```js theme={null}
navigate(input.url);
collect(parse());
```
Best for:
* Product pages
* Profile pages
* Article pages
* Detail pages
***
### **Search or discovery scraper**
Use this pattern when an input creates multiple result pages.
Example:
```js theme={null}
navigate(`https://example.com/search?q=${input.keyword}`);
const results = parse().results;
for (const result of results) {
next_stage({ url: result.url });
}
```
Best for:
* Search pages
* Category pages
* Directory listings
* Marketplace discovery
***
### **Multi-page detail scraper**
Use this pattern when a scraper must move from a list page to many detail pages.
Example flow:
**Input keyword → Search page → Result URLs → Detail pages → Output records**
Best for:
* Product discovery
* Lead generation
* Local business listings
* Review collection
* Job listings
***
## **Debugging mental model**
When a scraper does not return the expected data, check the flow in order:
1. **Input**, Did the scraper receive the expected input values?
2. **Interaction code**, Did it reach the correct page or response?
3. **Parser code**, Did the selectors extract the expected fields?
4. **Stages**, Did `next_stage()` create the expected child pages?
5. **Output records**, Did `collect()` emit the expected structure?
6. **Schema**, Does the output schema include the returned fields?
7. **Worker type**, Does the page require Browser worker behavior?
This flow helps isolate whether the issue is caused by input data, navigation, parsing, staging, schema, or worker configuration.
## Related
Choose between Browser worker and Code worker
Full reference for interaction and parser commands
Step-by-step walkthrough of building a scraper in the IDE
Recommended patterns for fast, reliable scrapers
# Scraper Studio IDE best practices
Source: https://docs.brightdata.com/products/scraper-studio/best-practices
Bright Data Scraper Studio IDE best practices for dead page detection, request batching, pagination, popups, tagged responses, timeouts and retries.
This guide shows the coding patterns the Bright Data Scraper Studio team recommends for writing fast, reliable scrapers in the IDE. Each section contrasts a common mistake with the preferred pattern and explains the reason.
## How do I detect dead pages reliably?
When using `navigate()`, add a `dead_page()` condition so the scraper does not retry pages that do not exist. Bright Data Scraper Studio automatically marks HTTP 404 responses as dead, but many sites return 200 with a "not found" template, so you must check for that yourself.
Do not wrap `wait()` in a `try/catch` and call `dead_page()` from the `catch` block. A thrown `wait()` only tells you the selector did not appear within the timeout, not that the page is actually dead.
```js Bad theme={null}
try {
// Waits 30 seconds for 'ok-selector' even if the page is already dead
wait('ok-selector');
} catch (e) {
// You cannot prove the page is dead from a wait timeout alone
dead_page("Page doesn't exist");
}
```
```js Good theme={null}
wait('ok-selector, 404-selector');
if (el_exists('404-selector'))
dead_page();
```
## How do I minimize requests to the browser?
Interaction commands like `click`, `type`, `el_exists`, `el_is_visible`, `wait`, and `wait_visible` each send a request to the browser. Combine selectors into a single call instead of chaining several calls.
```js Bad theme={null}
if (!(el_exists('#price1')) || el_exists('#price2')
|| el_exists('#price3') || el_exists('#discount'))
{
dead_page('No price found');
}
```
```js Good theme={null}
if (!el_exists('#price1, #price2, #price3, #discount'))
dead_page('No price found');
```
## How do I paginate without blocking parallelization?
When a site has paginated results and you want data from every page, call `rerun_stage()` once from the root page for every page you need. Do not call `rerun_stage()` from inside each page as you walk the pagination: that serializes the work and Bright Data Scraper Studio cannot parallelize the requests.
```js Bad theme={null}
navigate(input.url);
let $ = load_html(html());
let next_page_url = $('.next_page').attr('href');
rerun_stage({url: next_page_url});
```
```js Good theme={null}
let url = new URL(input.url);
if (input.page)
url.searchParams.set('page', input.page);
navigate(url);
// input.page only exists when this stage was re-run for a specific page.
// On the root page it's undefined, so we fall through to fan out.
if (input.page)
return;
let $ = load_html(html());
let total_products = +$('.total_pages').text();
let total_pages = Math.ceil(total_products / 20);
total_pages = Math.min(total_pages, 50);
for (let page = 2; page <= total_pages; page++)
rerun_stage({url: input.url, page});
```
Avoid collecting all pages in a single session. Each session has a 16MB result size limit, and that limit counts the total accumulated data in the session, including all collected lines, parsed results, children and parser metadata, not just the size of the rendered page. Calling `rerun_stage({url: input.url, page})` once per page processes one page per session and keeps each session within the limit.
## How do I close popups without waiting for them?
Use `close_popup('popup_selector', 'close_button_selector')` to register a background watcher that closes popups whenever they appear. Do not poll for a popup with `wait_visible()` before each interaction: popups can appear at any time, and explicit waits add latency on every step.
```js Bad theme={null}
navigate('https://example.com');
try {
wait_visible('.cky-btn-accept', {timeout: 5000});
click('.cky-btn-accept');
} catch (e) {
console.log('Accept cookies button does not exist, continue');
}
```
```js Good theme={null}
// Runs in the background with no per-step latency.
// The watcher checks for the popup before every interaction automatically.
close_popup('.cky-btn-accept', '.cky-btn-accept');
navigate('https://example.com');
click('.open-product-full-info');
```
## How do I wait for a tagged response before parsing?
When you use `tag_response()` to capture a background API call, follow it with `wait_for_parser_value()` to make sure the request has finished before you read `parser`. Without the wait, the parser may run before the response has arrived and `parser.` will be `undefined`.
```js Bad theme={null}
tag_response('product', /api\/product/);
navigate('https://example.com');
// Parser code:
// The request may not have finished yet; product could be undefined
let {product} = parser;
return product.data;
```
```js Good theme={null}
tag_response('product', /api\/product/);
navigate('https://example.com');
wait_for_parser_value('product');
// Parser code:
let {product} = parser;
return product.data;
```
```js Good (chained navigation) theme={null}
tag_response('product', /api\/product/);
navigate('https://example.com');
// wait_for_parser_value returns the value so you can use it in interaction code
let product = wait_for_parser_value('product');
navigate(product.reviews_url);
tag_html('reviews_html');
// Parser code:
let {product, reviews_html} = parser;
let $ = load_html(reviews_html);
let reviews = $('.review').toArray().map(v => $(v).text());
return {
...product.data,
reviews,
};
```
## Should I throw custom error messages?
No. Let built-in errors from Bright Data Scraper Studio bubble up. They include the selector, the timeout, and the stage, which is more useful than a hand-written "Page not loaded properly". Only throw a custom error when you are checking a domain-specific condition that Bright Data Scraper Studio cannot detect on its own, such as a missing product title.
```js Bad theme={null}
try {
wait('selector1');
// some code
wait('selector2');
// some code
} catch (e) {
throw "Page not loaded properly"
}
```
```js Good theme={null}
// Crawler error: waiting for selector "selector1" failed: timeout 30000ms exceeded
wait('selector1');
// some code
wait('selector2');
// some code
```
```js Good (domain-specific check) theme={null}
if (!el_exists('.product-title'))
throw new Error('Failed to load product page');
```
## How do I handle slow websites without over-extending timeouts?
Keep the default 30-second timeout for most waits. If a specific page is consistently slow, raise it to 45 or 60 seconds. Do not push beyond 60 seconds: a slower peer is usually the cause, and Bright Data Scraper Studio automatically retries with a fresh peer session when a page reports a timeout error.
```js Bad theme={null}
// 120 seconds is too long; the platform cannot recycle a stuck peer
wait('selector', {timeout: 120000});
```
```js Good theme={null}
wait('selector'); // default 30 seconds
wait('selector', {timeout: 45000}); // 45 seconds for slightly slow pages
wait('selector', {timeout: 60000}); // 60 seconds for consistently slow pages
```
## Should I build my own retry loop?
No. Bright Data Scraper Studio handles retries at the job level with a new peer session. A custom retry loop inside your scraper reuses the same session, which is the reason the first attempt failed. Report the error and let Bright Data Scraper Studio retry.
```js Bad theme={null}
let counter = input.counter || 5;
while (counter > 1) {
try {
wait('selector', {timeout: 500});
click('selector');
type('selector');
// some code
break;
} catch (e) {
// rerun_stage creates a new session but this pattern spends extra CPM
return rerun_stage({...input, counter: --counter});
}
}
```
```js Good theme={null}
navigate('https://example.com');
wait('h1');
```
## Should I wrap parser expressions in try/catch?
No. Use optional chaining (`?.`) and nullish coalescing (`??`) instead. A silent `try/catch` around a property access hides real bugs, and a `try/catch` around a `wait()` wastes browser time.
```js Bad theme={null}
try {
const example = obj.prop;
} catch (e) {}
```
```js Bad theme={null}
// Wasting browser time for no reason
try { wait('selector'); } catch (e) {}
try { wait_network_idle({timeout: 8000}); } catch (e) {}
try { wait_page_idle(); } catch (e) {}
```
```js Good theme={null}
const example = object?.prop;
const example2 = object.prop ?? undefined;
const example3 = object.prop ? object.prop : undefined;
```
## How do I extract values from a set of elements in parser code?
Use `toArray().map()` instead of `each()`. It is shorter, returns a real array, and reads as a single expression.
```js Bad theme={null}
const links = [];
$('.card.product-wrapper').each(function (i, el) {
links.push({url: $(this).find('h4 a').attr('href')});
})
return links;
```
```js Good theme={null}
const links = $('.card.product-wrapper').toArray().map(v => ({
url: $(v).find('h4 a').attr('href'),
}));
```
## How do I normalize text in parser code?
Call `$(selector).text_sane()`. Bright Data Scraper Studio adds this custom method to the Cheerio prototype: it collapses every run of whitespace to a single space and trims the result. For numeric extraction, strip non-digits with a regex.
```js Bad theme={null}
$.prototype.clearText = function () {
return this.text().replace(/\s+/g, ' ').trim();
}
```
```js Good theme={null}
let name = $('a').text_sane();
// For digits-only extraction:
let value = +$('a').text().replace(/\D+/g, '');
```
## Related
Full reference for Bright Data Scraper Studio interaction and parser commands
Choose between Browser worker and Code worker for your scraper
# Build a scraper with the Bright Data CLI
Source: https://docs.brightdata.com/products/scraper-studio/build-with-the-cli
Use the Bright Data CLI to create, run and self-heal a Scraper Studio scraper from your terminal or any coding agent like Claude Code or Cursor.
The Bright Data CLI builds Bright Data Scraper Studio scrapers from your terminal in three commands: log in with `bdata login`, then `bdata scraper create` and `bdata scraper run`. Run it on demand with `npx`, so there is nothing to install. This tutorial walks you through building a Hacker News top-stories scraper end to end. The CLI runs unchanged inside the embedded terminal of any coding agent like Claude Code, Cursor or Codex.
**Time to complete:** about 10 minutes (AI generation runs in the background)
## Prerequisites
* A Bright Data account ([sign up free](https://brightdata.com/?hs_signup=1\&utm_source=docs), no card required)
* A terminal. Any embedded terminal works too: Claude Code, Cursor, Codex, VS Code
## Run the Bright Data CLI with npx
You do not install anything. Run the CLI on demand with `npx`, which fetches the latest version each time:
```bash theme={null}
npx -p @brightdata/cli bdata --version
```
To keep the rest of this tutorial's commands short, alias `bdata` for your shell session. Otherwise, prefix any `bdata` command below with `npx -p @brightdata/cli`:
```bash theme={null}
alias bdata="npx -p @brightdata/cli bdata"
```
The CLI is published as `@brightdata/cli` on npm. The `bdata` and `brightdata` commands are interchangeable. Prefer a permanent command instead of npx? Install it globally with `npm install -g @brightdata/cli`.
## Build your first scraper from the terminal
Run `bdata login`. The CLI opens a browser tab so you can authorize it against your Bright Data account, then stores your API key locally. You do not paste or copy a key.
```bash theme={null}
bdata login
```
> **Expected result:**
>
> ```text theme={null}
> Opening browser for Bright Data authentication...
> Logged in successfully. Key: 2e75****12bf
> Checking for required zones...
> Zone "cli_unlocker" already exists.
> Zone "cli_browser" already exists.
> ```
The two zones (`cli_unlocker` and `cli_browser`) are the Web Unlocker API and Browser API endpoints the CLI uses when running scrapers. Bright Data creates them automatically on first login.
Pass a target URL and one sentence describing the data you want. Bright Data's AI Agent generates the output schema, writes the scraper code and returns a Collector ID.
```bash theme={null}
bdata scraper create https://news.ycombinator.com \
"Extract top stories: title, url, points, author, comment count"
```
The AI pipeline runs in seven stages, printed live: `user_intent_analyzer`, `planner`, `collector_maintainer`, `output_schema_generator`, `code_generator`, `input_schema_generator`, `preview_runner` and `preview_picker`. Typical wall-clock time is 5 to 15 minutes; complex targets can take up to 25 minutes.
> **Expected result:**
>
> ```text theme={null}
> Template created: c_mpohus372o5tmid1jk
> Triggering AI generation...
> Generating scraper...
> Step: user_intent_analyzer — polling (attempt 1/600)
> ...
> Done in 280 poll attempts.
> {"status":"done","completed_steps":[...],"step":"preview_picker"}
> ```
Save the Collector ID (the `c_*` string). It is the stable handle for every subsequent run, schedule or API call on this scraper.
Pass the Collector ID and a URL. Use `--pretty` to format the JSON output.
```bash theme={null}
bdata scraper run c_mpohus372o5tmid1jk https://news.ycombinator.com --pretty
```
The CLI tries realtime mode first. If the scraper triggers more pages than the realtime limit allows, the CLI silently falls back to batch mode (`POST /dca/trigger` then poll `GET /dca/dataset`) and continues. No flag needed.
> **Expected result:** a JSON array, one row per result.
>
> ```json theme={null}
> [
> {
> "title": "Last.fm is now independent",
> "url": "https://support.last.fm/t/last-fm-is-now-independent/118591",
> "points": 447,
> "author": "twistslider",
> "comment_count": 131
> },
> {
> "title": "DuckDuckGo search saw 28% more visits after Google said people love AI mode",
> "url": "https://www.pcgamer.com/hardware/duckduckgos-ai-free-search-saw-nearly-28-percent-more-visits-in-the-week-following-googles-insistence-that-people-love-ai-mode/",
> "points": 418,
> "author": "HelloUsername",
> "comment_count": 212
> }
> ]
> ```
## How do I use this from Claude Code, Cursor or Codex?
The Bright Data CLI runs inside any embedded terminal as-is. The coding agent is not building the scraper itself; the CLI calls Bright Data's AI Agent, and the coding agent calls the CLI on your behalf.
Two integrations make the CLI feel native inside a coding agent:
**Pin the Collector ID in the agent's rules file** so the agent re-uses your scraper across sessions instead of building a fresh one every time:
```text CLAUDE.md / .cursor/rules / CODEX.md theme={null}
SCRAPER_STUDIO_COLLECTOR_ID=c_mpohus372o5tmid1jk
HACKER_NEWS_SCRAPER_USAGE="bdata scraper run $SCRAPER_STUDIO_COLLECTOR_ID --pretty"
```
**Wire Bright Data's MCP server into your agent** with `brightdata add mcp`. The MCP server is separate from the Scraper Studio CLI but gives the agent additional scraping tools (`scrape_as_markdown`, `search_engine` and others) it can call directly:
```bash theme={null}
brightdata add mcp # interactive: pick Claude Code, Cursor or Codex
```
See the [Bright Data MCP server overview](/products/mcp-server/overview) for what the MCP exposes.
## What just happened?
Three CLI commands mapped to four Bright Data Scraper Studio API endpoints. Use this table to translate the CLI flow into raw HTTP calls when you are ready to integrate without the CLI:
| You ran | Bright Data API endpoint behind it |
| - | - |
| `bdata login` | Local credential store. Stores the API key from [Account Settings](https://brightdata.com/cp/setting). |
| `bdata scraper create` | `POST /dca/collector` then `POST /dca/collectors/{c_*}/automate_template` |
| `bdata scraper run` (small input) | `POST /dca/trigger_immediate` then `GET /dca/get_result` |
| `bdata scraper run` (large input) | `POST /dca/trigger` then poll `GET /dca/dataset?id=j_*` |
For a worked example of the underlying API in cURL, Python and Node.js, see the [Bright Data Scraper Studio API quickstart](/products/scraper-studio/quickstart). For every endpoint, see the [Scraper Studio API reference](/api-reference/scraper-studio-api/Choose_a_delivery_type_on_request_level).
## How do I fix a scraper when the site changes?
When a target site is redesigned and a scraper starts returning null or missing fields, fix it in place with `bdata scraper heal`. Self-healing keeps the same Collector ID, so every trigger, schedule and integration that references the scraper keeps working. The flow is run, inspect, heal, approve, re-run.
Pass the Collector ID and a plain-language description of what broke. Keep the prompt under 1,000 characters.
```bash theme={null}
bdata scraper heal c_mpohus372o5tmid1jk \
"The points and comment_count fields return null since the site redesign. Re-capture them from the new markup." \
--url https://news.ycombinator.com
```
By default, `heal` stops at an approval gate so you can review the proposed fix before it goes live.
> **Expected result:** the command returns an envelope with `status: "awaiting_approval"` and a `preview_result` showing sample output from the proposed fix, plus a `next_step` hint with the command to run next.
Review the preview. If it looks right, approve it. The fix commits to the existing scraper and the Collector ID does not change.
```bash theme={null}
bdata scraper approve c_mpohus372o5tmid1jk --url https://news.ycombinator.com
```
To discard the proposed fix and try a sharper prompt instead, reject it:
```bash theme={null}
bdata scraper approve c_mpohus372o5tmid1jk --reject
```
> **Expected result:** on approval, `status` advances to `done`. On reject, the scraper is left unchanged so you can run `heal` again with a clearer prompt.
Run the healed scraper on the same URL and confirm the previously broken fields are populated.
```bash theme={null}
bdata scraper run c_mpohus372o5tmid1jk https://news.ycombinator.com --pretty
```
For unattended workflows, add `--auto-approve` to the `heal` command. It approves the fix automatically and polls through to `done` in one step. Use it only when you trust the heal without a manual review.
## Frequently asked questions
AI generation timing depends on target complexity. Simple single-page scrapers finish in 5 to 10 minutes. Pages with lazy-load, pagination or anti-bot challenges can take 15 to 25 minutes. The CLI polls Bright Data's AI Flow API every five seconds and prints the current stage, so you can leave it running and check back. No action is needed while you wait.
Realtime mode caps the number of page loads per request. When a scraper triggers more pages than the realtime limit allows, the CLI prints `Realtime page limit exceeded, switching to batch mode...`, submits the same inputs to `POST /dca/trigger`, and polls `GET /dca/dataset?id=j_*` until the snapshot is ready. The switch is automatic and the final JSON shape is identical. See [Scraper Studio specifications](/products/scraper-studio/specifications) for the page-load limits.
The AI Agent's generated schema is per-row best-effort, not strict. Jobs posts, "Show HN" entries and very new submissions on Hacker News do not always have a points or comment count yet, so the scraper returns the row with those fields omitted rather than inventing a value. Treat missing fields as `null` in your own code. To enforce a stricter schema, open the scraper in [Scraper Studio](/products/scraper-studio/ai-agent) or rewrite the schema with the [Self-Healing tool](/products/scraper-studio/self-healing-tool).
Yes. The Collector ID returned by `bdata scraper create` (the `c_*` string) is the same handle the Bright Data Scraper Studio API uses. Pass it to `POST /dca/trigger` from any HTTP client. See the [Bright Data Scraper Studio API quickstart](/products/scraper-studio/quickstart) for cURL, Python and Node.js examples.
Fix it in place with `bdata scraper heal`, which keeps the same Collector ID. See [How do I fix a scraper when the site changes?](#how-do-i-fix-a-scraper-when-the-site-changes) above for the full run, heal, approve, re-run flow. Two alternatives:
* **Control panel:** use the [Self-Healing tool](/products/scraper-studio/self-healing-tool) to describe the fix in plain language.
* **Direct API:** the `heal` and `approve` commands wrap a three-call loop. `POST /dca/collectors/{c_*}/refactor_template` with the prompt, poll `GET .../refactor_template/progress` until `status` is `pending_answer`, then `POST .../resume_automation_job` to approve or reject. See [Trigger Self-Healing](/api-reference/scraper-studio-api/ai-flow/trigger-self-healing) and [Resume Self-Healing Job](/api-reference/scraper-studio-api/ai-flow/resume-self-healing-job). For a Node.js implementation, see the [Scraper Studio Self-Healing demo](https://github.com/anil-bd/scraper-studio-self-healing-demo).
The `bdata login` command requires a browser callback. For headless environments, export your API key as `BRIGHTDATA_API_KEY` and the CLI uses it directly without a login step:
```bash theme={null}
export BRIGHTDATA_API_KEY="your_api_key_here"
bdata scraper run c_mpohus372o5tmid1jk https://news.ycombinator.com
```
Copy the key from [Account Settings](https://brightdata.com/cp/setting).
## Related
Build the same scraper from the Bright Data control panel instead of the terminal
Trigger an existing scraper from cURL, Python or Node.js
Fix a scraper with a plain-language prompt when a target site changes
Every `bdata` command, with examples
# Scraper Studio coding agent prompts
Source: https://docs.brightdata.com/products/scraper-studio/coding-agent-prompts
Copy-pasteable Claude Code, Cursor and Codex prompts to build a Bright Data Scraper Studio scraper: a simple build-and-run prompt plus a full self-heal loop.
Use these copy-pasteable prompts to build a Bright Data Scraper Studio scraper through Claude Code, Cursor or Codex. Start with the one-prompt build-and-run if you just want a scraper fast: swap in your URL and the fields you want, paste, done. The longer flows below add the full build, run, heal, approve, re-run loop for when you need to extend a scraper's schema in place.
## Prerequisites
* A Bright Data account ([sign up free](https://brightdata.com/?hs_signup=1\&utm_source=docs), no card required).
* A coding agent with terminal access: Claude Code, Cursor or Codex.
You do not install the Bright Data CLI ahead of time. The prompts below run it through `npx`, which fetches the latest version on demand, so there is no global dependency to maintain.
## Build a scraper in one prompt
If you just want a scraper fast, paste this prompt and replace the two values in angle brackets: the target URL and the fields you want. The agent runs the Bright Data CLI through `npx`, builds the scraper and runs it once. No self-healing.
```text Prompt theme={null}
Build and run a Bright Data scraper. Run every Bright Data CLI command through `npx -p @brightdata/cli` so nothing is installed globally. Replace and , then do each step in order and stop if a step fails:
1. Authenticate by running `npx -p @brightdata/cli bdata login`. npx fetches the CLI on demand, so there is nothing to install.
2. Create a Bright Data scraper for that extracts: . Report the Collector ID.
3. Run that scraper on the same URL and pretty-print the result.
```
For example, to scrape a product page, the filled-in second step reads:
```text Prompt theme={null}
2. Create a Bright Data scraper for https://shopalto.xyz/product/aurora-wireless-headphones that extracts: product name, price, description and rating. Report the Collector ID.
```
> **Expected result:** the agent reports a Collector ID like `c_mpohus372o5tmid1jk`, then prints a JSON array with one row containing the fields you asked for.
Save the Collector ID. Reuse it to run the scraper on new URLs, or to extend its schema later with the self-heal flow below.
## Build, run and self-heal in one prompt
To run the full build, run, heal, approve, re-run loop in action, paste this single prompt and let the agent work through every step. The pattern is deliberate: build a minimal scraper first, then heal it to extend the schema, so the heal envelope's `preview_result` is easier to verify against a known-good baseline. The agent runs every Bright Data CLI command through `npx`, so nothing is installed globally.
```text Prompt theme={null}
Build, run, heal and verify a Bright Data scraper end to end. Run every Bright Data CLI command through `npx -p @brightdata/cli` so nothing is installed globally. Do every step in order and stop if a step fails:
1. Authenticate by running `npx -p @brightdata/cli bdata login`. npx fetches the CLI on demand, so there is nothing to install.
2. Create a Bright Data scraper for https://shopalto.xyz/product/aurora-wireless-headphones that extracts two fields: product name and price. Report the Collector ID.
3. Run that scraper on the same URL and pretty-print the result. Expect one row with name and price.
4. Heal the scraper in place to also capture description, image url and rating alongside the existing name and price. Keep the same Collector ID, anchor the heal on the same URL and show the approval envelope.
5. When the preview shows all five fields, approve the fix anchored on the same URL.
6. Run the scraper on the same URL again and confirm all five fields come back: name, price, description, image_url and rating.
```
> **Expected result:** the agent ends with a JSON row containing `name`, `price`, `description`, `image_url` and `rating`, and the Collector ID is unchanged from step 2.
## Run the flow step by step
Work through the prompts below one at a time when you want to inspect each Collector ID, run result and heal envelope before moving on.
```text Prompt theme={null}
Run every Bright Data CLI command through `npx -p @brightdata/cli` so nothing is installed globally. Authenticate by running `npx -p @brightdata/cli bdata login`, then confirm the version with `npx -p @brightdata/cli bdata --version` before continuing.
```
> **Expected result:** the agent prints a `bdata` version and confirms it is authenticated. npx fetches the CLI on demand, so nothing is installed globally.
```text Prompt theme={null}
Create a Bright Data scraper for https://shopalto.xyz/product/aurora-wireless-headphones that extracts just two fields: product name and price. Show me the Collector ID when it is done.
```
> **Expected result:** the agent reports a Collector ID like `c_mpohus372o5tmid1jk`. Hold onto it; the rest of the prompts reuse the same ID.
```text Prompt theme={null}
Run that scraper on https://shopalto.xyz/product/aurora-wireless-headphones and pretty-print the result.
```
> **Expected result:** a JSON array with one row, populated with `name` and `price` only.
```text Prompt theme={null}
Extend the scraper in place. Heal it to also capture description, image url and rating alongside the existing name and price. Keep the same Collector ID. Anchor the heal on https://shopalto.xyz/product/aurora-wireless-headphones and show me the approval envelope when it is ready.
```
> **Expected result:** the agent reports `status: "awaiting_approval"` with a `preview_result` row that now shows five fields.
```text Prompt theme={null}
The preview looks good. Approve the fix, anchored on https://shopalto.xyz/product/aurora-wireless-headphones.
```
> **Expected result:** `status` advances to `done`. The Collector ID is unchanged.
```text Prompt theme={null}
Run the scraper on https://shopalto.xyz/product/aurora-wireless-headphones again and confirm all five fields now come back: name, price, description, image_url and rating.
```
> **Expected result:** the same JSON shape as the earlier run, now with three additional fields per row.
For an unattended variant, ask the agent to add `--auto-approve` to the heal call. The agent skips the approval gate and polls through to `done` in one step. Use it only when you trust the heal without a manual review.
## Related
The canonical CLI tutorial: install, log in, create, run, heal
Fix a scraper from the Bright Data control panel
Trigger an existing scraper from cURL, Python or Node.js
Flag reference for scraper create, heal and approve
# Complete Scraper Studio IDE examples
Source: https://docs.brightdata.com/products/scraper-studio/complete-examples
Browse 6 end-to-end Bright Data Scraper Studio IDE examples covering interaction code, parser code, multi-result handling and advanced site navigation.
This article applies only to the IDE Normal V2 version, which is for internal use and not available to customers.
## Introduction
The collect and parse commands have been removed. The data will be returned from parser code as an object or array, and it will be automatically saved to the output:
```js Interaction code theme={null}
// Old code
navigate("https://example.com");
collect(parse());
// New code
navigate("https://example.com");
// New code alternative
navigate("https://example.com");
tag_html("html_key");
```
```js Parser code theme={null}
// Old code
return {
title: $('h1').text(),
};
// New code
return {
title: $('h1').text(),
};
// or
let $ = load_html(parser.html_key);
return {
title: $('h1').text(),
};
```
New commands have been added to provide access to the data from the interaction code: `tag_html`, `tag_request`, `tag_graphql`
In addition, some existing commands have been updated: `tag_response`, `tag_sitemap`, `tag_all_responses`. See [IDE documentation](/products/scraper-studio/develop-a-scraper) for more details.
When using any tag commands, you can provide a custom name. You can then access the data using this name in the parser code under parser.`YOUR_KEY`.
For `tag_html`, current browser location URL will be saved under parser.`YOUR_KEY_url`.
For simple cases when only a single `tag_html` is needed, it can be skipped, and it will be automatically saved under parser.page.
```js Interaction code theme={null}
navigate("https://example.com/1");
tag_html("page1");
navigate("https://example.com/2");
tag_html("page2");
navigate("https://example.com/3");
tag_html("page3");
```
```js Parser code theme={null}
{page1, page1_url, page2, page2_url, page3, page3_url} = parser;
let $ = load_html(page2);
return {
title2: $('h1').text(),
};
```
Sometimes, it's necessary to get parsed data within the interaction code and use it to make request. See examples how to do it:
```js Interaction code theme={null}
navigate("https://example.com/1");
tag_html("page1");
let page_html = html();
let page_html2 = wait_for_parser_value("page1"); // the same
let $ = load_html(page_html);
let req_id = $('.product-id').text();
tag_request("product_json", {url: "https://example.com/product/"+req_id});
```
```js Parser code theme={null}
{product_json} = parser;
return product_json;
```
## How to return multiple results
To collect multiple results, array can be returned from the parser code.
```js Interaction code theme={null}
navigate("https://example.com/products");
```
```js Parser code theme={null}
let items = $(".product").toArray().map(v=>$(v)).map(v=>({
title: v.find(".title").text(),
price: v.find(".price").text(),
url: new URL(v.find("a").attr("href")),
}));
return items;
```
## How to reparse collected data
Reparse is a new feature that allows to reparse the data that was already collected. It can be useful when you want to change the parser code without rerunning the entire interaction code:
## `next_stage` and `rerun_stage`
When scraper has more than one step, the parser code is only available in the last step. All other steps can only have `next_stage` calls. To parse something from the page, `load_html` should be used:
```js Interaction code step 1 theme={null}
navigate(input.domain_url);
tag_html('html');
const $ = load_html(wait_for_parser_value('html'));
$('a.layout-categories-category__link').toArray()
.map(v => new URL($(v).attr('href'), location.href))
.filter(x => !x?.href.includes('home')).forEach(i => {
next_stage({url: i})
});
```
```js Interaction code step 2 theme={null}
block(['*.ico', '*.png', '*.jpg', '/images/', '*gif']);
navigate(input.url);
wait('.view-option-selector-button, .product-groups_empty-list');
if(el_exists('.product-groups_empty-list'))
dead_page('There are no products in this category')
wait('[aria-describedby="onetrust-policy-text"]');
click('button#onetrust-reject-all-handler');
wait_hidden('[aria-describedby="onetrust-policy-text"]');
$('.view-option-selector-button').eq(2).click();
wait_page_idle(2000);
scroll_to('bottom');
wait_page_idle(2000);
```
```js Parser code theme={null}
return $('a[data-qa-action="product-click"]').toArray()
.map(v => ({url: new URL($(v).attr('href'))}));
```
## Basic PDP scraper
```js Interaction code theme={null}
let url = new URL(input.url.replace('https://www.slintel.com','https://6sense.com'));
url = new URL(url.pathname, 'https://6sense.com');
navigate(url);
if (location.href === 'https://6sense.com/company')
dead_page(`Page not found`);
tag_html('html');
```
```js Parser Code theme={null}
const $ = load_html(parser.html)
const nextData = JSON.parse($('#__NEXT_DATA__').html());
const pageProps = nextData.props.pageProps;
const companyInfo = pageProps.company_data.companyInfo;
const linkedin = companyInfo.linkedin ?
(companyInfo.linkedin.includes('http') ? companyInfo.linkedin :
'https://'+companyInfo.linkedin) : null;
const techCategories = pageProps.company_data?.technologies_mapper_view?.categories
? Object.values(pageProps.company_data.technologies_mapper_view.categories).map(v => Object.keys(v)).flat()
: null;
const pageData = {
"name": companyInfo.name,
"about": companyInfo.company_description,
"num_employees": companyInfo.employee_range,
"type": companyInfo.company_type,
"industries": companyInfo.industry_v2_ranked.filter(v=>v),
"techstack_arr": techCategories,
"country_code": companyInfo.country,
"website": companyInfo.display_domain
? new URL(companyInfo.display_domain.includes('http')
? companyInfo.display_domain
: 'https://'+companyInfo.display_domain)
: null,
"social_media_urls": linkedin,
"company_news": pageProps.company_data.company_news.map(v=>({
title: "company_news_data",
date: "",
link: "",
})),
"last_updated": new Date(companyInfo.last_updated_at*1e3),
"url": new URL(parser.html_url),
"logo": companyInfo.logo,
"location": companyInfo.location,
"region": [companyInfo.country, companyInfo.state].filter(v=>v).join(', '),
"id": nextData.query.companyid,
"slintel_resources": companyInfo.recommended_companies.map(v=>({
link: v.display_domain ? new URL(v.display_domain.includes('http') ? v.display_domain : 'https://'+v.display_domain) : null,
title: v.name,
type: v.company_type,
})),
"stock_symbol": companyInfo.stock_symbol,
};
return pageData;
```
## Multiple navigates example
```js Interaction code theme={null}
const tabs = [
'topactivity',
'answers',
'questions',
'tags',
'articles',
'badges',
'bookmarks',
'bounties'
];
function loadTab(url, name) {
return new Promise(async (resolve, reject)=>{
try {
const tabUrl = new URL(url);
tabUrl.searchParams.set('tab', name);
navigate(tabUrl, { allow_status: [404] });
const html_ = html();
tag_html(name);
resolve(html_);
} catch(e) { reject(e); }
});
}
const userIdFromUrl = (input.url && input.url.includes('https://stackoverflow.com/users/'))
? input.url.replace('https://stackoverflow.com/users/', '').split('/').shift()
: null;
const userId = input.user_id || userIdFromUrl || 1;
const userUrl = `https://stackoverflow.com/users/${userId}`;
navigate(userUrl, { allow_status: [404] });
tag_html('user');
Promise.all(tabs.map( async tabName => loadTab(userUrl, tabName)));
```
```js Parser code theme={null}
const steps = [
'user',
'topactivity',
'answers',
'questions',
'tags',
'articles',
'badges',
'bookmarks',
'bounties',
'finally'
];
const parse = Handlers();
let missed_parser_data = Object.keys(parse)
.filter(handler_key=>!steps.find(sk=> sk == handler_key));
if (missed_parser_data.length)
throw new Error('missed parser data')
return steps.reduce((acc, step) => {
console.log('handling: '+step)
if (step=='finally')
return parse.finally(acc);
let page_source = parser[step];
console.log(parser[step]);
console.log('parser[step]');
if (!page_source)
throw('unexpected empty data, data not saved')
const $ = load_html(page_source);
const data_chunk = parse[step]($);
acc = { ...acc, ...data_chunk };
return acc;
}, {});
function Handlers() {
return {
'finally': (res) =>{
res.url = new URL(res.url);
res.answers.map(v => v.tags = res.tags);
res.answers.map(v => v.url = new URL(v.url));
res.questions.map(v => v.url = new URL(v.url));
res.top_posts.map(v => v.date = new Date(v.date));
res.active_from = new Date(res.active_from)
return res;
},
'user': ($)=>{
const grid = $('#main-content .d-grid > .grid--item')
.toArray().map(v=>$(v));
let collectives = grid.find(v=>v.find('> div').text()
.includes('Collectives'));
if (collectives?.length) {
collectives = collectives.find('.fl1').toArray()
.map(v=>({
name: $(v).find('.fs-body2').text_sane() || null,
text: $(v).find('.fs-caption').text_sane() || null,
}));
}
else
collectives = [];
let communities = grid.find(v=>v.find('> div').text()
.includes('Communities'));
if (communities?.length) {
communities = communities.find('li.flex--item')
.toArray().map(v=>({
name: $(v).find('.fl-grow1').text_sane() || null,
score: $(v).find('.fl-shrink0').text_sane()
.replace(/,/gm, '') || null,
}));
}
else
communities = [];
let badges = $('.flex__fl-equal > .flex--item').toArray()
.map(v=>{
let $v = $(v);
return {
badge_type: $v.find('.mr12 .fc-gold').length
? 'gold'
: $v.find('.mr12 .fc-silver').length
? 'silver'
: $v.find('.mr12 .fc-bronze').length
? 'bronze'
: 'unknown',
badge_name: $v.find('.fs-caption').text_sane()
.replace(' badges', '') || null,
number_of_badges: $v.find('.fs-title').text()
.replace(/,/gm, '') || null,
badges: $v.find('a.badge').toArray().map(b=>({
name: $(b).text_sane() || null,
description: $(b).attr('title') || null
}))
};
});
return {
url: new URL(parser.user_url),
id: parser.user_url.split('/').pop(),
'id#': (input.url && input.url
.includes('https://stackoverflow.com/users/'))
? input.url?.match(/https:\/\/stackoverflow.com/users/(d{1,15}).*/)?.[1]/gm)
: input.user_id
|| 1,
user_id: +$('[property="og:url"]').attr('content')
.match(/https:\/\/stackoverflow.com/users/(d{1,15}).*/)?.[1]/)/gm
|| +input.url?.match(/https:\/\/stackoverflow.com/users/(d{1,15}).*/)?.[1]/gm)
|| +input.user_id
|| null,
name: $('.fs-headline2').text_sane() || null,
type: $('#mainbar-full > div:first-child .s-badge')
.text_sane() || null,
title: $('#mainbar-full > div:first-child .fs-title')
.text_sane() || null,
active_from: new Date(
$('#mainbar-full > div:first-child .list-reset span[title]').attr('title')),
last_seen: $('#mainbar-full > div:first-child .fs-title + ul li:last-child')
.text_sane() || null,
linkes: $('#mainbar-full > div:first-child ul.list-reset:last-child a')
.toArray().map(v => $(v).attr('href')),
location: $('#mainbar-full > div:first-child ul.list-reset:last-child li:last-child div[title]')
.attr('title') || null,
stats: $('#stats .flex--item').toArray()
.map(v => $(v).text_sane()).filter(v => v),
about: $('#main-content .s-prose p').toArray()
.map(v => $(v).text_sane()) || null, //.join('\r'),
collectives,
communities,
badges,
top_tags: $('#top-tags .p12').toArray().map(v=>({
name: $(v).find('.s-tag').text_sane() || null,
score: $(v).find('.d-flex > .d-flex:nth-child(1)')
.text_sane().replace(' Score', '')
.replace(/,/gm, '') || null,
posts: $(v).find('.d-flex > .d-flex:nth-child(2)')
.text_sane().replace(' Posts', '')
.replace(/,/gm, '') || null,
posts_percent: $(v).find('.d-flex > .d-flex:nth-child(3)')
.text_sane().replace(' Posts %', '').replace(/,/gm, ''),
})),
top_posts: $('#js-top-posts .p12 .d-flex').toArray().map(v=>({
type: $(v).find('.iconQuestion').length ? 'question'
: ($(v).find('.iconAnswer').length ? 'answer' : 'none'),
score: $(v).find('.s-badge').text_sane()
.replace(/,/gm, '') || null,
text: $(v).find('a.d-table')
.text_sane() || null,
date: new Date($(v).find('.relativetime')
.attr('title')),
})),
top_meta_posts: $('#js-top-posts + div .p12 .d-flex')
.toArray().map(v=>({
score: $(v).find('.s-badge').text_sane()
.replace(/,/gm, '') || null,
text: $(v).find('a.d-table').text_sane() || null,
})),
top_network_posts: $('#js-top-posts + div + div .p12 .d-flex')
.toArray().map(v=>({
score: $(v).find('.s-badge').text_sane()
.replace(/,/gm, '') || null,
text: $(v).find('a.d-table').text_sane() || null,
}))
};
},
'topactivity': ($) => {
let summary_graph_data;
try {
summary_graph_data = JSON.parse(
/graphDatas[^[]+([[^]]+])/gm.exec(
$('*').first().html())?.[1]);
} catch(e) {
console.log('graphData not found')
}
let [summary_people_reached, summary_posts_edited,
summary_helpful_flags, summary_votes_cast] =
$('div:not(.js-highlight-box-reputation) > h1.flex--item + div .flex--item .fs-body3')
.toArray().map(v=>$(v).text_sane().replace(/,/gm, '') || null);
return {
summary_reputation: +$('#top-cards h4.fs-headline1')
.text().replace(/D+/gm, ''),
summary_top_overall: $('a[href*="alltime"]')
.first().text_sane() || null,
summary_next_tag_badge: $('#rep-card-next-tag-badge a')
.text_sane() || null,
summary_graph_data,
summary_next_tag_score:
$('div.fl-shrink1 + div.fl-grow1 .fs-fine')
.first().text_sane().replace(/,/gm, '') || null,
summary_next_tag_answers:
$('div.fl-shrink1 + div.fl-grow1 .fs-fine')
.last().text_sane().replace(/,/gm, '') || null,
summary_badges: $('h3 +div .s-badge').toArray()
.map(v => $(v).attr('title')).join(', '),
summary_last_badge: $('#badge-card-last').text_sane() || null,
summary_next_badge: {
name: $('#js-badge-card-next').text_sane() || null,
progress: $('h4.flex--item.ws-nowrap + span')
.text_sane() || null
},
summary_people_reached,
summary_posts_edited,
summary_helpful_flags,
summary_votes_cast,
};
},
'answers': ($) => {
return {
answers: $('#js-post-summaries .s-post-summary')
.toArray().map(p=>{
let $p = $(p);
return {
id: $p.attr('data-post-id') || null,
votes: $p.find('.s-post-summary--stats-item-number')
.text().replace(/,/gm, '') || null,
accepted: !!$p.find('svg.iconCheckmarkSm').length,
url: new URL($p.find('.answer-hyperlink')
.attr('href'), location.href),
text: $p.find('.answer-hyperlink').text_sane() || null,
tags: $p.find('.post-tag').toArray()
.map(v => $(v).text_sane() || null ).map(name=>({name}))
};
}),
}
},
'questions': ($) => {
return {
questions: $('#js-post-summaries .s-post-summary')
.toArray().map(p=>{
let $p = $(p);
return {
id: $p.attr('data-post-id') || null,
votes: $p.find('.s-post-summary--stats-item-number')
.text().replace(/,/gm, '') || null,
accepted: !!$p.find('svg.iconCheckmarkSm').length,
answer_count:
$p.find('.s-post-summary--stats-item:nth-child(2) .s-post-summary--stats-item-number')
.text_sane().replace(/,/gm, '') || null,
url: new URL($p.find('.s-link').attr('href'),
location.href),
text: $p.find('.s-link').text_sane() || null,
tags: $p.find('.post-tag').toArray()
.map(v => $(v).text_sane() || null)
};
}),
};
},
'tags': ($) => {
return {
tags: $('#user-tab-tags .ba .p12').toArray()
.map(v=>$(v)).map(v=>({
name: v.find('.post-tag').text_sane() || null,
badge: v.find('.badge-tag').attr('title') || null,
description: v.attr('title') || null,
score: v.find('.flex--item:first-child > .fs-body3')
.text_sane().replace(/,/gm, '') || null,
posts: v.find('.flex--item:last-child > .fs-body3')
.text_sane().replace(/,/gm, '') || null,
})),
};
},
'articles': ($) => {
return {
articles_count: $('h2.fs-title').text()
.replace(/D+/gm, '') || null
};
},
'badges': ($) => {
return {
badges: $('#user-tab-badges .grid--item').toArray()
.map(v => ({
name: $(v).find('.mbn4').text_sane() || null,
count: $(v).find('.ml4').text_sane()
.replace(/D+/gm, '') || 0
})),
};
},
'bookmarks': ($) => {
return {
bookmarks: $('#js-post-summaries .s-post-summary')
.toArray().map(p=>{
let $p = $(p);
return {
id: $p.attr('data-post-id'),
votes:
$p.find('.s-post-summary--stats-item__emphasized .s-post-summary--stats-item-number')
.text_sane().replace(/,/gm, '') || null,
views: $p.find('.is-supernova .s-post-summary--stats-item-number')
.text_sane().replace(/,/gm, '') || null,
accepted: !!$p.find('svg.iconCheckmarkSm').length,
answer_count: $p.find('.s-post-summary--stats-item:nth-child(2) .s-post-summary--stats-item-number')
.text_sane().replace(/,/gm, '') || null,
url: new URL($p.find('.s-link').attr('href'),
location.href).href,
text: $p.find('.s-link').text_sane() || null,
tags: $p.find('.post-tag').toArray()
.map(v => $(v).text_sane() || null)
};
})
};
},
'bounties': ($) => {
return {
bounties_count: $('h2.fs-title').text()
.replace(/D+/gm, '') || null
};
},
}
}
```
## Multiple `tag_response`
```js Interaction code theme={null}
close_popup('._1piuevz', '._1piuevz');
tag_response('stay', /\/StaysPdpSections/);
tag_response('calendar', /\/PdpAvailabilityCalendar/);
tag_response('reviews', /api\/v3\/StaysPdpReviews/);
navigate(input.url+'?enable_auto_translate=false#availability-calendar');
if (el_exists('img[src*="error_pages/404"]'))
return dead_page('Page not found');
const room_id = /\/(\d+)(?:\?|$)/.exec(input.url)?.[1];
if (el_exists('._wgmchy ._1qx9l5ba'))
click('._wgmchy ._1qx9l5ba');
wait('#data-state, #data-deferred-state',{timeout: 6e4});
wait_for_parser_value('stay');
wait_for_parser_value('calendar');
wait_for_parser_value('reviews');
tag_html('html');
```
```js Parser code theme={null}
let room_id = /([0-9]+)(?:\?|$)/.exec(input.url)?.[1];
const $ = load_html(parser.html);
let price = +$('[data-section-id="BOOK_IT_SIDEBAR"] ._14y1gc ._tyxjp1')
.first().text().replace(/[^[0-9].,]+/gm,'')
let currency = $('._tyxjp1').last().text().replace(/[0-9,]/g, '').trim()
let data = JSON.parse($('#data-state, #data-deferred-state').html())
price = price ? new Money(price, currency) : null
let avatar = $('._9ofhsl').attr('src')
let name = $('.tehcqxo h2.hnwb2pb').text().split('by')?.[1]?.trim()
|| $('._cv5qq4 ._14i3z6h').text().split('by')?.[1]?.trim()
let { stay, calendar, reviews} = parser;
let cd = data.niobeMinimalClientData.find(f=>f.find(f2=>
f2?.data?.presentation?.stayProductDetailPage))?.find(f2=>
f2?.data?.presentation?.stayProductDetailPage)
.data?.presentation?.stayProductDetailPage?.sections;
let sections = stay.data.presentation.stayProductDetailPage.sections
.sections || [];
sections = sections.concat(cd?.sections);
function get_section(name) {
return sections.find(v=>v.sectionId == name).section;
}
let metadata = stay.data.presentation.stayProductDetailPage.sections
.metadata || cd?.metadata;
let category_rating = [];
sections.find(v=>v?.sectionId == "REVIEWS_DEFAULT")?.section
?.ratings?.forEach(r=>{
category_rating.push({
name: r.label,
value: r.localizedRating,
});
});
let sec_sheet = get_section("BOOK_IT_CALENDAR_SHEET");
let sec_book_it = get_section("BOOK_IT_SIDEBAR");
price = sec_book_it.structuredDisplayPrice.price
?.replace(/[^[0-9].]/,'') || price;
currency = data.userAttributes?.curr || currency;
let available_dates = [];
calendar.data.merlin.pdpAvailabilityCalendar.calendarMonths.forEach(m=>{
m.days.forEach(d=>{
if (d.available) available_dates.push(d.calendarDate);
});
});
let image = metadata.sharingConfig.imageUrl ||
sections.find(v=>v.sectionId == "PHOTO_TOUR_SCROLLABLE_MODAL")
.section.mediaItems[0].baseUrl;
if (image) image = new Image(image);
avatar = sections.find(v=>v?.sectionId == "HOST_PROFILE_DEFAULT")
?.section?.hostAvatar?.avatarImage?.baseUrl || avatar;
if (avatar) avatar = new Image(avatar);
return {
name: metadata.sharingConfig.title,
price,
image,
description: sections.find(v=>v?.sectionId == "DESCRIPTION_MODAL")
?.section?.items?.map(v=>((v?.title||'')+' '+v?.html?.htmlText).trim()
)?.join(', '),
category: metadata.seoFeatures.breadcrumbDetails[1].linkText,
availability: sec_sheet.available.toString(),
reviews: reviews.data.presentation.stayProductDetailPage.reviews
.reviews.map(r=>r.localizedReview?.comments || r.comments),
ratings: metadata.sharingConfig.starRating,
seller_info: {
name: sections.find(v=>v?.sectionId == "HOST_PROFILE_DEFAULT")
?.section?.title || name,
url: new URL(`https://www.airbnb.ru/contact_host/${room_id}/send_message`),
avatar,
features: sections.find(v=>v?.sectionId == "HOST_PROFILE_DEFAULT")
?.section?.hostFeatures?.map(f=>({
name: f.title,
value: f.subtitle
}))
},
breadcrumbs: sections.find(v=>v?.sectionId == "LOCATION_DEFAULT")
?.section?.previewLocationDetails?.[0]?.title,
location: sections.find(v=>v?.sectionId == "LOCATION_DEFAULT")
?.section?.previewLocationDetails?.[0]?.title,
lat: metadata.loggingContext.eventDataLogging.listingLat ||
sections.find(v=>v?.sectionId == "LOCATION_DEFAULT")?.section?.lat,
long: metadata.loggingContext.eventDataLogging.listingLng ||
sections.find(v=>v?.sectionId == "LOCATION_DEFAULT")?.section?.long,
guests: sec_book_it.maxGuestCapacity || sections
.find(v=>v?.sectionId == "AVAILABILITY_CALENDAR_DEFAULT")
?.section?.maxGuestCapacity,
pets_allowed: sec_book_it.petsAllowed,
description_items: sections.find(v=>
v?.sectionId == "AVAILABILITY_CALENDAR_DEFAULT")?.section
?.descriptionItems.map(v=>v.title),
category_rating,
house_rules: sections.find(v=>v?.sectionId == "POLICIES_DEFAULT")
?.section?.houseRules?.map(r=>r?.title),
details: sections.find(v=>v?.sectionId == "OVERVIEW_DEFAULT")
?.section?.detailItems?.map(f=>f?.title),
highlights: sections.find(v=>v?.sectionId == "HIGHLIGHTS_DEFAULT")
?.section?.highlights?.map(f=>({
name: f.title,
value: f.subtitle
})),
arrangement_details: sections.find(v=>
v?.sectionId == "SLEEPING_ARRANGEMENT_DEFAULT")?.section
?.arrangementDetails?.map(f=>({
name: f.title,
value: f.subtitle
})),
amenities: sections.find(v=>v?.sectionId == "AMENITIES_DEFAULT")
?.section?.seeAllAmenitiesGroups?.map(g=>({
group_name: g.title,
items: g.amenities.filter(f=>f.available).map(f=>({
name: f.title,
value: f.subtitle
}))
})),
images: sections.find(v=>v?.sectionId == "HERO_DEFAULT")
?.section?.previewImages?.map(v=>new Image(v?.baseUrl)),
available_dates,
url: new URL(input.url),
final_url: new URL(location.href),
};
```
# Develop a scraper with the IDE
Source: https://docs.brightdata.com/products/scraper-studio/develop-a-scraper
Build a custom web scraper in the Bright Data Scraper Studio IDE: write interaction and parser code, preview results, debug runs, and ship to production.
This guide walks through building a custom web scraper in the Bright Data Scraper Studio IDE from scratch. You will write interaction code that navigates the target site, parser code that extracts structured fields, and then save the scraper to production and configure delivery. By the end, you will have a runnable scraper you can trigger by API, manually, or on a schedule.
## Prerequisites
* An active [Bright Data account](https://brightdata.com/?hs_signup=1\&utm_source=docs) with access to Scraper Studio
* Basic JavaScript familiarity (variables, functions, control flow)
* A target URL you want to scrape
If you prefer describing the scraper in plain language instead of writing code, use the [Scraper Studio AI Agent](/products/scraper-studio/ai-agent). The agent generates the same kind of scraper the IDE would produce.
## How do I build a scraper in the IDE?
Go to [brightdata.com/cp/scrapers](https://brightdata.com/cp/scrapers), click **Scraper Studio**, then click **Open IDE** to open an empty scraper.
You can start from a blank scraper or use a template.
Templates are pre-built scraper starters for common patterns and sites. Use a template when your target site or scraping pattern is similar to one of the available options.
Use a blank scraper when you want full control over the scraping flow from the beginning.
Interaction code navigates the target site and captures the data you need into the parser. Select **Interaction code** from the left sidebar and write the code in the main editor.
A minimal interaction script:
```js theme={null}
navigate(input.url);
wait('.product-title');
let data = parse();
collect(data);
```
In this example:
* `navigate(input.url)` opens the page provided in the input.
* `wait('.product-title')` waits for the expected element.
* `parse()` runs the parser code.
* `collect(data)` adds the parsed record to the output dataset.
For a multi-page scrape, fan out with `next_stage()`:
```js theme={null}
navigate(input.url);
wait('.listing');
let listings = parse().listings;
for (let url of listings)
next_stage({url});
```
See [Scraper Studio functions](/products/scraper-studio/functions) for the full list of interaction commands.
Parser code reads the HTML or response loaded by interaction code and returns a structured JavaScript object.
Select **Parser code** from the left sidebar and define the fields you want to extract.
Parser code commonly uses Cheerio’s jQuery-like `$` selector:
```js theme={null}
return {
title: $('h1').text_sane(),
price: new Money(+$('.price').text().replace(/\D+/g, ''), 'USD'),
image: new Image($('img.product').attr('src')),
listings: $('.listing a').toArray().map(el => $(el).attr('href')),
};
```
The object returned by parser code is available wherever interaction code calls `parse()`. See [Scraper Studio functions](/products/scraper-studio/functions#parser-functions) for the parser helpers Bright Data Scraper Studio provides.
In the **Settings** panel, pick the worker type:
* **Code worker** (faster): for static HTML pages and public JSON endpoints
* **Browser worker**: for JavaScript-rendered pages, clicks, scrolling, popups, or captured background traffic
Start with **Code worker** when possible. Switch to **Browser worker** if the data you need is not available in the raw response or if you need [browser-only](/products/scraper-studio/functions#browser-only-functions) functions.
See [**Worker types**](/products/scraper-studio/worker-types) for the full comparison.
Use **Preview** to test the scraper before making it active for production use.
Preview runs the scraper code against the input selected in the **Input** tab at the bottom-left of the IDE. Use it to test interaction logic, parser logic, output structure, and errors before running the scraper on a larger input set.
The results appear in the **Output** tab. Use the **Run log** and **Browser network** tabs to debug failed runs.
> **Expected result:** the Output tab shows a structured record with the fields defined in your parser code.
When the preview returns the expected output, click **Finish editing** in the top-right corner. For an existing production scraper, click **Save to production** to apply your changes.
The scraper appears under **My Scrapers** in the control panel and can be triggered by API, manually, or on a schedule.
Open the scraper from **My Scrapers** and configure **Delivery preferences**. Choose a delivery destination, such as API download, webhook, Amazon S3, GCS, Azure, SFTP, Snowflake or email and a file format (JSON, NDJSON, CSV, XLSX). See [Initiate collection and delivery](/products/scraper-studio/initiate-collection-and-delivery-options) for all available options.
After the scraper is active and delivery is configured, start a production run. Choose the initiation method that matches your workflow:
* [Initiate by API](/products/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run) - start a run from your application or automation workflow.
* [Initiate manually](/products/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run) - start a run from the control panel by entering inputs or uploading a file.
* [Schedule a scraper](/products/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run) - run the scraper automatically on a recurring schedule.
## Frequently asked questions
Open the scraper in the Bright Data Scraper Studio IDE and check the **Last errors** tab. Every failed input is stored with its exact error message and error code (up to the most recent 1,000 failures). Re-run the failing input from the IDE to reproduce the problem locally, fix the interaction or parser code, and save a new production version.
Yes. Every scraper in Bright Data Scraper Studio, regardless of how it was created, can be opened and edited in the IDE. You can change extraction logic, tweak selectors, add or remove output fields, and change the worker type.
Click **Edit Schema** in the IDE's output schema panel and add the new fields, or return them from parser code and Bright Data Scraper Studio prompts you to update the schema when you save to production.
Use `collect()` to append one record at a time; it is the default way to emit data. Use `set_lines()` when you are collecting records progressively and want the most recent snapshot delivered even if a later step throws an error. Every call to `set_lines()` overrides the previous one. See [collect](/products/scraper-studio/functions#collect-append-a-record-to-the-dataset) and [set\_lines](/products/scraper-studio/functions#set-lines-set-output-lines-overriding-previous-calls).
## Related
Full reference for interaction and parser commands
Recommended patterns for fast, reliable scrapers
Reference for every panel and control in the IDE
Fix broken scrapers and add fields with plain-language prompts
# Scraper Studio error codes
Source: https://docs.brightdata.com/products/scraper-studio/error-codes
Look up Bright Data Scraper Studio error codes, status codes and warnings, with the meaning and suggested fix for each collection failure.
This reference lists the error codes Bright Data Scraper Studio returns when a crawl, parser, request, validation or delivery step fails during a collection run, and the suggested action for each one.
Read the `error_code`, `status_code` and raw `error` message together to understand what failed and what to do next.
These are Scraper Studio collection errors, not API authentication or API request errors. For proxy-layer HTTP errors returned outside Scraper Studio, see the [Error catalog](/proxy-networks/errorCatalog).
## How do I read a Scraper Studio error?
Scraper Studio uses five system fields to describe the status of each record.
| Field | Type | What it tells you |
| - | - | - |
| `error_code` | Machine-readable identifier | Why the crawl failed, for example `dead_page`, `bad_input`, `blocked`, `crawl_error`, `parse_error` or `load_sitemap`. Use this field for filtering, automation or retry logic |
| `error` | Human-readable message | The detailed failure reason, for example `Crawler error: Cannot find product title` |
| `status_code` | HTTP-like number | A summary of the crawl result. Optional, see the status code table below |
| `warning` | Human-readable message | A non-fatal issue on a record that was still delivered |
| `warning_code` | Machine-readable identifier | The machine-readable version of `warning`, for example `dead_page` or `validation` |
Branch on `error_code` rather than on the raw `error` string, because the message text changes as scrapers are updated:
```js theme={null}
if (line.error_code === 'dead_page') {
// Handle expired or removed URL
}
```
### What do the Scraper Studio status codes mean?
`status_code` summarizes the crawl result as an HTTP-like numeric value.
| Status code | Meaning |
| - | - |
| 400 | Bad request or invalid page response |
| 403 | Access blocked or denied by the target site |
| 404 | Page not found, removed or invalid URL |
| 407 | Account is suspended or access is restricted |
| 408 | Request timed out. The target site did not respond in time |
| 421 | Request was routed to the wrong server |
| 429 | Too many requests. Rate limit reached |
| 500 | Crawl, proxy or navigation failed before Scraper Studio received a valid page response |
| 503 | Target site or service is temporarily unavailable, overloaded or timed out |
`status_code` is sometimes missing or `undefined`, because some error paths do not assign a numeric status code. Treat `status_code` as optional and do not assume it is always present.
## How do errors and warnings differ?
An error means the record failed. A warning means the record was delivered but something should be reviewed.
* **Error:** the record failed and should be treated as unsuccessful. A failed record usually includes `error`, `error_code` and `status_code`
* **Warning:** the record was delivered, but something should be reviewed. A delivered record with a non-fatal issue usually includes `warning`, `warning_code` and `status_code`
* **Successful record:** none of `error`, `error_code`, `warning` or `warning_code` is populated
Warnings occur when a partial crawl returned data but also hit an issue, when a normally failing condition was configured as a non-error, or when output validation found an issue but the record was still kept.
When a `dead_page` error is downgraded to a warning, the output includes:
```text theme={null}
warning_code = dead_page
warning = Dead page detected
```
Schema validation warnings usually use:
```text theme={null}
warning_code = validation
```
The same underlying issue can appear either as an error or as a warning, depending on the scraper logic, validation rules and output schema settings.
## Where do Scraper Studio errors come from?
Scraper Studio errors come from one of three layers.
### What causes scraper errors?
Scraper errors come from the scraper's interaction code, parser code or input handling. Examples include invalid input, `wait_element_timeout`, `parse_error`, `click_timeout`, `dead_page`, `bad_input` and `blocked`.
Fix these by updating the scraper logic, parser selectors, validation rules or input data.
### What causes proxy and unblocker errors?
Proxy and unblocker errors come from Bright Data's proxy, routing or unblocking layer. Examples include proxy connection issues, target website blocking, geo or zone configuration issues, no available peers and rate limiting.
Fix these by retrying, reducing the request rate, changing country or geo settings, or contacting Bright Data support if the issue persists.
### What causes platform and infrastructure errors?
Platform and infrastructure errors come from the Scraper Studio platform, browser worker, parser sandbox, storage layer or internal infrastructure. Examples include worker timeout, browser disconnected, parser memory limit exceeded, upload failure and WebSocket connection issues.
These are often transient and are frequently resolved by retrying. If the issue persists, open a support ticket with the scraper ID, job ID, failed input and raw error message.
## Common scraper error codes
These error codes indicate scraper execution issues. The request may have failed because of invalid input, a missing or changed page element, target blocking, parsing problems, rate limits, CAPTCHA handling or scraper logic that needs to be updated.
| Error code | Meaning | Suggested action |
| - | - | - |
| `dead_page` | The scraper detected that the page does not exist or is no longer relevant. | Check whether the page still opens manually. Remove expired URLs or fix discovery logic that produces stale links. |
| `bad_input` | The scraper marked the input as invalid and skipped retries. This happens when required input fields are missing, the URL is invalid or the page is not the expected type. | Validate inputs before triggering the scraper. Remove invalid URLs or update discovery logic to avoid unsupported page types. |
| `blocked` | The `blocked()` function was used in the scraper code, or the target website refused access to the page. | Retry later, reduce the request rate, or review country, session, CAPTCHA and block-handling logic. |
| `crawl_error` | A general crawl execution error occurred. This happens when the page did not load correctly, required page data was missing, the browser session closed or scraper code tried to read missing data. | Open Debug crawl or Crawl inspector for the failed input. Check page loading, add null checks and rerun failed pages. |
| `wait_element_timeout` | The scraper waited for an element that did not appear in time. | Confirm the selector still exists and is stable. Increase the timeout or handle alternate page layouts if the page loads slowly. |
| `ajax_request_error` | A background request failed or timed out during page execution. | Review network requests in preview or debug mode. If the request is optional, relax validation or add fallback logic. |
| `collector_request_validation` | A custom request validation rule rejected an AJAX or network response, usually from `verify_requests()` logic. | Review the validation callback and confirm the rejected response is required. Relax the rule if the request is optional. |
| `captcha_timeout` | CAPTCHA solving did not complete within the expected time. | Retry the crawl and check whether the target repeatedly shows CAPTCHA. Review CAPTCHA handling and job deadline settings. |
| `close_popup_fail` | Scraper Studio failed to close a popup, such as a consent manager or modal. | Check whether the popup selector changed or appears conditionally. Update popup handling or make the scraper tolerate the popup if it does not block extraction. |
| `click_timeout` | A click action did not complete within the timeout. The element may be missing, hidden, disabled or not clickable. | Confirm the selector matches a visible, clickable element. Add `wait(selector)` before clicking or update the click logic. |
| `tag_response` | A tagged response callback failed, or the tagged response returned an unexpected status or format. | Review the tagged response callback and confirm the expected response exists. Check the browser network tab in the IDE to verify that the response loads correctly, and add checks before reading fields from it. |
| `load_sitemap` | Scraper Studio failed to load or parse a sitemap. Causes include fetch failure, request timeout or invalid sitemap format. | Open the sitemap URL manually and confirm it is reachable and valid. |
| `load_more_timeout` | The scraper waited for more items to load, but no new items appeared before the timeout. | Confirm the "load more" selector and list container still exist. Add a stop condition when no more results are available. |
| `child_input_size_validation` | A child input created by `next_stage()` exceeded the allowed input size. | Reduce the data passed to `next_stage()`. Pass only fields needed by the next stage, such as URL, ID or small metadata values. |
| `detect_block` | Scraper Studio detected block-page content, such as "Access Denied", sign-in walls or other known blocking patterns. | Open the failed page manually and confirm whether it shows a block page. If many inputs fail, reduce the request rate or review location and session behavior. |
| `ERR_INVALID_URL` | A URL value is invalid or not in a supported format. | Validate input URLs before triggering the scraper. Make sure URLs include `http://` or `https://` and do not contain malformed characters. |
| `bad_cmd_arg` | An argument passed to an IDE command is invalid or failed validation. This happens when a command receives a value in the wrong format, a value whose type does not match the expected schema, or is missing a required field. For example, a command that expects a number receives text. | Check the arguments passed to the failing command and correct the format, type or missing field. This code is also fixable by [Auto Self-Healing](/products/scraper-studio/auto-self-healing), which can adjust the generated code or command parameters so they pass validation. |
| `not_supported_cmd` | The scraper used a function that is not supported by the selected worker type. | Change the worker type or replace the function. Use a Browser worker for browser-only actions such as click, scroll, type and wait. See [Worker types](/products/scraper-studio/worker-types). |
| `detached_element` | The scraper referenced an element that was removed or re-rendered before the action completed. | Re-select the element immediately before interacting with it. Add waits after page updates and avoid storing element handles across DOM changes. |
| `timeout` | The scraper timed out while waiting for an element, request, response or page action to complete. | Confirm that the expected element or response exists. Increase the timeout only when needed and add fallback logic for optional or slow-loading content. |
### How do `block` and `blocked` differ?
`block` and `blocked` are different error codes. `blocked` is raised by scraper logic, while `block` is raised by the layer that fetched the page.
`blocked` is usually raised by scraper logic. The scraper calls `blocked()` when it detects a blocked page, login wall, CAPTCHA page or other condition that should mark the crawl as blocked:
```js theme={null}
blocked('Login page was shown');
```
`block` usually comes from navigation, request rejection, target-site blocking or proxy-level response handling. It may include the real HTTP status returned by the target site or upstream layer. Observed status codes for `block` are:
```text theme={null}
400, 401, 403, 404, 405, 409, 410, 418, 429, 500, 503
```
In short:
* `blocked` is collector-level block detection, usually raised by scraper logic
* `block` is target, proxy or navigation-level rejection, often with an HTTP status
## Navigation and browser error codes
These error codes indicate navigation or browser loading issues. The scraper may have failed because the page did not load in time, the browser could not reach the URL, the target kept network requests open, or the connection timed out or closed before completion.
| Error code | Meaning | Suggested action |
| - | - | - |
| `bad_navigate` | The browser failed to navigate to the requested URL. | Open the URL manually and check whether it redirects or blocks access. Retry the crawl if the failure is temporary. |
| `navigation_timeout` | The page did not finish navigation within the allowed time. | Increase the navigation timeout only if needed. Prefer `wait(selector)` or a different `wait_until` strategy instead of waiting for full page load. |
| `domcontentloaded_event_timeout` | The browser timed out while waiting for the `DOMContentLoaded` event. The page did not reach the expected load state in time. | If full page load is not required, use `navigate(input.url, { wait_until: 'navigate' })` and then `wait(selector)` for the element you need. |
| `networkidle_event_timeout` | The browser timed out while waiting for network activity to become idle. Some pages never reach network idle because they keep background requests open. | Avoid relying on full network idle for pages with continuous background traffic. Use `wait_until: 'navigate'` and wait for a specific selector instead. |
| `load_event_timeout` | The page did not fire the full `load` event in time. | Avoid waiting for full page load if it is not required. Use selector-based waits for the content needed by the parser. |
| `document_load_failed` | The browser started navigation, but the main document failed to load. Causes include an HTTP failure, SSL/TLS issue, empty response, missing credentials, redirect issue or proxy and peer problem. | Retry and verify the target URL opens in a browser. If the failure persists, try a different geo or open a support ticket with the job ID and full error. |
| `net_err_timed_out` | The network request timed out. | Retry and check target stability. Reduce concurrency if the timeout affects many inputs. |
| `net_err_closed` | The target site closed the connection unexpectedly. | Check whether the site is available and retry. If it happens frequently, review blocking or rate behavior. |
| `net_err_cert_date_invalid` | The SSL/TLS certificate presented by the server is not valid for the current date. | Verify that the target URL opens correctly in a browser and that the site certificate is valid. |
| `net_err_http2_protocol_error` | The browser encountered an HTTP/2 protocol error while loading the target page or resource. | Retry the request. If it persists, confirm whether the target site loads normally in a browser. |
| `net_err_cert_authority_invalid` | The target site's SSL/TLS certificate is signed by an untrusted or unknown certificate authority. | Verify that the target URL opens correctly in a browser and that the certificate is trusted. |
### Browser and infrastructure lifecycle errors
These error codes indicate that the browser session, runner or browser-control connection was interrupted during the crawl. They are usually transient platform or browser lifecycle errors.
Common lifecycle error codes are `runner_disconnected`, `network_error`, `cdp_conn_err`, `cdp_cmd_timeout`, `cdp_disconnect`, `bad_browser`, `browser_disconnected` and `ipc_timeout`.
**Suggested action:** retry the job. If the error persists, open a support ticket with the job ID, response IDs, failed input and raw error message.
### Rate limit variants
`global_rate_limit` and `bucket_rate_limit` mean that requests to the target domain are being rate limited. This usually happens when the target site is sensitive to high request volume or too many requests are sent in parallel.
**Suggested action:** retry after a short delay. If the issue repeats, reduce concurrency and queue jobs instead of running many jobs in parallel.
## Job lifecycle error codes
These error codes indicate that the collection did not complete because of runtime, deadline, cancellation or page-volume limits.
| Error code | Meaning | Suggested action |
| - | - | - |
| `job_run_timeout` | The collection or crawl exceeded the allowed runtime. This happens during long waits, slow page actions or CAPTCHA solving. | Optimize scraper logic and check for loops, long waits or repeated CAPTCHA solving. Reduce work per request or increase the deadline if available. |
| `crawl_timeout` | The crawler did not return a result within the internal timeout. | Retry the crawl. Simplify the crawl logic or reduce page complexity if it repeats. |
| `deadline_timeout` | The job reached its configured deadline. | Increase the deadline or reduce work per request. For large workloads, split the job into smaller runs. |
| `too_many_pages` | The job created too many child pages. | Reduce the number of child pages generated per input. Use batch collection for high fan-out jobs. |
| `uncrawled_page` | The page result was requested before the page was crawled, usually because the job deadline was reached. | Increase the deadline or reduce the number of pages created by the scraper. |
| `aborted_page` | The crawl was aborted because the job was canceled. | No action is required if the job was intentionally canceled. Otherwise, rerun the affected inputs. |
## Infrastructure and storage error codes
These error codes indicate platform, worker, storage or delivery issues. The collection may have failed because an internal service was unavailable, a worker was overloaded, the result was too large or delivery to an external destination failed.
| Error code | Meaning | Suggested action |
| - | - | - |
| `infra_error` | An internal infrastructure error occurred. | Retry the request. If the issue persists, contact support with the job or response ID. |
| `page_too_big` | The loaded page exceeded the Scraper Studio page size limit of 16 MB. | Collect less data per page. Avoid storing large HTML blobs or unnecessary fields. |
| `crawl_request_failed` | The crawl runner could not connect to a crawler worker after retries. | Retry the job. Escalate to support if the issue persists. |
| `worker_too_busy` | The crawler worker was overloaded. | Retry later and reduce concurrency. |
| `external_upload_fail` | Uploading a result or media file to an external destination failed. | Verify delivery destination configuration, credentials, permissions and path settings. |
| `failed_media_upload` | Scraper Studio failed to save a media file. | Retry the failed page and confirm the media URL is reachable. If the failure persists, check delivery and storage settings or contact support. |
## Proxy and unblocker error codes
These error codes indicate a routing or connection issue between Scraper Studio and the underlying proxy network. They usually happen when the proxy network cannot establish or maintain a stable connection to the target site.
| Error code | Meaning | Suggested action |
| - | - | - |
| `proxy` | A proxy-layer failure occurred during navigation or request. Possible causes include no peer availability, DNS failure, SSL issues, target rejection, timeout or protection pages. | Retry, switch country if needed and verify the target is accessible. If the failure persists, contact support with the job ID and raw error. |
| `proxy_error` | The proxy layer could not connect to the target website or received an unexpected upstream response. | Retry and verify the target URL. If the issue repeats, try another country or escalate with the job ID and error details. |
| `net_err_tunnel` | The browser could not establish a tunnel connection through the proxy. | Check whether the website is available and retry. Escalate if the issue persists across many inputs. |
| `no_peers` | Scraper Studio failed to establish a connection with a peer. | Retry later or remove restrictive peer settings such as country, location or session constraints. |
## Parser and payload error codes
These error codes indicate parser execution or payload-size issues. The scraper may have failed because parser code encountered invalid or missing data, the payload sent to the parser was too large, or parser execution exceeded memory or CPU limits.
| Error code | Meaning | Suggested action |
| - | - | - |
| `parse_error` | Parser code failed while extracting or processing data. Common causes include syntax errors, blocked or invalid content, missing fields, invalid URLs, or reading properties from `null` or `undefined`. | Run a preview with the failing input and inspect the parser code. Add defensive checks and validate selectors. |
| `parse_request_payload_large` | The payload sent to the parser exceeded the size limit. | Reduce the amount of data sent to the parser. Extract only the required HTML section or response fields. |
| `parse_mem_limit_exceeded` | Parser execution exceeded the memory limit, often because of very large HTML, JSON, arrays or embedded data. | Reduce the parser payload and avoid storing large objects in memory. Extract only the required fields. |
| `parse_cpu_limit_exceeded` | Parser execution exceeded the CPU limit, often because of heavy loops, regex or transformations. | Simplify parser logic and reduce loops over large datasets. Move unnecessary processing out of the parser. |
| `parse_req_error` | The parser request payload exceeded the allowed field size. | Reduce the size of the request or parsed payload. Avoid sending very large fields, files or HTML blocks into parser processing. |
## Access and permissions error codes
These error codes indicate an access or compliance issue. The request was stopped because the target is restricted by Bright Data's compliance safeguards or because account permissions need review.
| Error code | Meaning | Suggested action |
| - | - | - |
| `brul` | The target is restricted by Bright Data compliance. | If access is required, contact `compliance@brightdata.com` and request permission for the blocked resource. |
## How to troubleshoot a failed Scraper Studio collection
Work through these steps in order when a collection run returns errors.
1. Check the `error_code`.
2. Check the `status_code`, if present.
3. Read the raw `error` message.
4. Open the failed URL manually to confirm whether the page exists and is accessible.
5. Use Debug crawl or Crawl inspector to inspect the failed input, crawl stage, children, files, warnings and output records.
6. If the error is parser-related, run a preview and inspect HTML, Output and Last errors.
7. If many inputs fail with the same error, check the request rate, concurrency, blocking, worker type and recent target-site changes.
8. If the issue persists, open a support ticket with the scraper ID, job or response ID, failed input and raw error message.
## FAQs
### Why is `status_code` missing from my failed record?
Some Scraper Studio error paths do not assign a numeric status code, so `status_code` is sometimes missing or `undefined`. Treat the field as optional and branch on `error_code` instead.
### Does a warning mean the record was lost?
No. A warning means the record was delivered but something should be reviewed, such as a partial crawl, a condition configured as a non-error or an output validation issue. Only records with `error` and `error_code` are unsuccessful.
### Which errors should I retry automatically?
Retry transient platform, browser lifecycle, proxy and rate limit errors, such as `infra_error`, `runner_disconnected`, `browser_disconnected`, `worker_too_busy`, `proxy_error`, `global_rate_limit` and `bucket_rate_limit`. Do not retry `bad_input`, `ERR_INVALID_URL` or `dead_page`, because those need an input or discovery-logic fix.
# Scraper Studio FAQs
Source: https://docs.brightdata.com/products/scraper-studio/faqs
FAQs about Bright Data Scraper Studio: scraper types, inputs and outputs, billing, limits, snapshots and reporting issues for custom scraper builds.
This page answers the questions the Bright Data support team hears most often about Scraper Studio. If you need a walkthrough instead of a quick answer, start with [Understanding Scraper Studio](/products/scraper-studio/introduction).
## Frequently asked general questions
A Bright Data web scraper is an automated script that collects public web data at scale through Bright Data's proxy and unblocking infrastructure. It returns the collected data in a structured format, such as JSON, NDJSON, CSV or XLSX. (Parquet is also supported for selected delivery options.) Results can be delivered to an API endpoint, webhook, cloud storage, Snowflake or SFTP. Bright Data maintains hundreds of pre-built scrapers for popular sites in the [Scrapers Library](https://brightdata.com/products/web-scraper).
Bright Data Scraper Studio is a cloud-hosted environment for building custom scrapers. It offers two modes: an AI Agent that generates a scraper from a natural-language description, and an IDE where you write JavaScript directly. Both modes run on the same Bright Data proxy and unblocking infrastructure. See [Understanding Scraper Studio](/products/scraper-studio/introduction).
The [Scrapers Library](https://brightdata.com/products/web-scraper) contains pre-built scrapers Bright Data maintains for popular sites such as Amazon, LinkedIn, and Instagram. Bright Data Scraper Studio is the environment you use to build custom scrapers when the site you need is not in the library.
Yes. A single scraper can navigate to any URL you pass in as input. If you need different extraction logic per site, use multiple stages (`next_stage()`) or build one scraper per site.
You do not have to provide every URL. The Bright Data Scraper Studio AI Agent can build different scraper types depending on what input you already have:
* **PDP**: provide a list of product URLs you already have; get one row per URL.
* **Discovery**: provide a category or listing URL; get one row per item found on that listing.
* **Discovery + PDP**: provide a category or listing URL; get full per-product detail for every item.
* **Search**: provide a keyword and optional country when you do not have URLs.
* **Sitemap**: provide a domain or `sitemap.xml` URL; get either all URLs found in the sitemap or full per-page detail for every URL.
Each scraper is scoped to one data shape, so the AI Agent is not a general crawler; do not pass a homepage and ask for "everything." For deeper link discovery across a site, build a multi-stage scraper in the Bright Data Scraper Studio IDE with `next_stage()`, `rerun_stage()` and `load_sitemap()`. See [What can the AI Agent build?](/products/scraper-studio/ai-agent#what-can-the-ai-agent-build) and [Functions](/products/scraper-studio/functions).
Bright Data Scraper Studio scrapes the web, not native mobile apps. For mobile layouts, the Browser worker can emulate a mobile device with `emulate_device('iPhone X')` so the target site serves its mobile web pages. Native iOS or Android app data is out of scope. See [Functions](/products/scraper-studio/functions).
## Inputs, outputs, and schemas
An input is the parameter set Bright Data Scraper Studio passes into the scraper for a single run. Typical inputs are a URL, a search keyword, a product ID or ASIN, a profile handle, or a date range. Multiple inputs can be passed in one job via CSV upload or the API.
The output is the structured data the scraper returns for an input. Bright Data Scraper Studio can deliver output in JSON, NDJSON, CSV, XLSX or Parquet depending on the scraper's delivery preferences and selected delivery option.
```text theme={null}
Note: Parquet is not available for API download, Email, Webhook, or OCI PAR delivery.
```
One input can generate multiple records. For example, if you submit 5 product listing URLs and each listing page contains 20 products, the scraper returns 100 records from 5 inputs. The statistics page counts records, not inputs.
A search scraper takes a keyword as input instead of a URL. Bright Data Scraper Studio runs a search on the target site and extracts data from the result pages. Use a search scraper when you do not have specific URLs to scrape.
A discovery scraper collects data from listing pages such as search results, category pages, or directories. It extracts fields that appear directly on the listing (titles, prices, ratings) and can also collect product URLs or IDs for a follow-up product-page scrape.
When the input or output schema changes, the scraper must be updated to match. If you trigger the scraper before Bright Data has updated it, you will see an `input(output)_schema_incompatible` error.
To trigger the scraper anyway and ignore the schema mismatch, click **Trigger anyway** in the UI or add a parameter to your API request:
* Output schema incompatible: `override_incompatible_schema=1`
* Input schema incompatible: `override_incompatible_input_schema=1`
```bash theme={null}
curl "https://api.brightdata.com/dca/trigger?scraper=ID_COLLECTOR&queue_next=1&override_incompatible_schema=1" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer API_KEY" \
-d '[{"url":"https://targetwebsite.com/product_id/"}]'
```
## Building and editing scrapers
You have three options in Bright Data Scraper Studio:
* **Build with the AI Agent**: describe the data you want in plain language. See [Scraper Studio AI Agent](/products/scraper-studio/ai-agent).
* **Build in the IDE**: write JavaScript directly. See [Develop a scraper with the IDE](/products/scraper-studio/develop-a-scraper).
* **Request a managed scraper**: Bright Data's data team builds and maintains the scraper for you.
Yes. Every scraper, including ones the AI Agent generated, can be opened and edited in the Bright Data Scraper Studio IDE. You can change extraction logic, modify selectors, add or remove output fields, and tune performance. If you prefer not to write code, use the [Self-Healing tool](/products/scraper-studio/self-healing-tool) to request changes in plain language.
Pass the AI Agent a target URL and an optional description of the data you want. The agent asks clarifying questions, generates an output schema for your review, and writes the full scraper code once you approve the schema. You can then run the scraper immediately or schedule recurring runs. See [Scraper Studio AI Agent](/products/scraper-studio/ai-agent) for a full walkthrough.
Yes. You own every line of the scraper, including code the AI Agent generates, and you can read and edit it in the Bright Data Scraper Studio IDE. Scrapers are plain JavaScript, so you can copy the code out and keep it in your own repository. To run a scraper outside Bright Data you would need to supply your own proxy and unblocking infrastructure.
Yes. Choose the Browser worker for sites that render content with JavaScript; it runs a full headless browser so you can wait for elements, click, scroll and capture background network calls. Choose the Code worker for static HTML and HTTP responses, which is faster and cheaper. See [Worker types](/products/scraper-studio/worker-types).
## Self-Healing
You trigger Self-Healing; it does not fire automatically when a target site's DOM changes. You can trigger it three ways:
* **Control panel**: open the scraper in the Bright Data Scraper Studio IDE, describe the fix in plain language and accept the AI-generated diff. See [Self-Healing tool](/products/scraper-studio/self-healing-tool).
* **Bright Data CLI**: run `bdata scraper heal ` then `bdata scraper approve`. Add `--auto-approve` for unattended workflows. See [Build a scraper with the Bright Data CLI](/products/scraper-studio/build-with-the-cli#how-do-i-fix-a-scraper-when-the-site-changes).
* **Bright Data API**: call `POST /dca/collectors/{collector_id}/refactor_template`. See [Trigger Self-Healing](/api-reference/scraper-studio-api/ai-flow/trigger-self-healing).
Each path repairs the code you own and keeps the same Collector ID. The refactor runs on Bright Data infrastructure, not on your machines, and can take up to 15 minutes; Bright Data emails you when the diff is ready.
## Running and triggering scrapers
Bright Data Scraper Studio supports three trigger methods:
* **By API**: regular request, queue request, or replace request
* **Manually**: from the control panel
* **On a schedule**: hourly, daily, weekly, or custom
See [Initiate collection and delivery](/products/scraper-studio/initiate-collection-and-delivery-options).
A queued request tells Bright Data Scraper Studio to wait until the previous request for the same scraper finishes before starting the next one. Use it when you want serial execution instead of running multiple jobs in parallel.
Bright Data Scraper Studio runs up to **100 batch jobs in parallel** per scraper. Additional jobs queue automatically and start as capacity frees up. See [Scraper Studio specifications](/products/scraper-studio/specifications) for full limits.
In the Bright Data Scraper Studio dashboard, click the **Bug** icon under **Failed crawls** to open the scraper in the IDE. Failed inputs appear in the **Last errors** tab with the exact error message and error code. Bright Data stores the last 1,000 errors per virtual job so you can re-run failed inputs and diagnose the issue.
## Blocking and unblocking
Bright Data Scraper Studio runs every scraper on Bright Data's proxy and unblocking network, with automatic retries and real-user fingerprinting built in. The IDE provides `solve_captcha()` to solve CAPTCHAs, `detect_block()` and `blocked()` to detect and report blocks, and `country()` to route a request through a specific country when a site is geo-restricted. See [Functions](/products/scraper-studio/functions).
## Billing and limits
CPM stands for "cost per mille", meaning 1,000 page loads. Bright Data Scraper Studio bills page loads in CPM units. Current rates are on the [pricing page](https://brightdata.com/pricing/web-scraper/studio).
A billable event is any function that causes Bright Data Scraper Studio to load a page or perform a network request:
* `navigate()`
* `request()`
* `load_more()`
* Media file download (billed per GB, separate from CPM)
Scraper Studio does not have a separate page-load or record allotment, but it is covered by the account-level free tier. Every new account gets 5,000 free credits per month, a single shared pool usable across the Web Unlocker API, SERP API, Web Scraper API, Scraper Studio and Browser API. In the free tier, Scraper Studio consumes one credit per page load drawn from that pool; records are not billed by default. New accounts also get a one-time onboarding bonus credit usable on any Bright Data product. See the [free tier](/general/account/billing-and-pricing/free-tier) and [billing](/general/account/billing-and-pricing/billing) pages.
## Snapshots and data retention
Snapshot retention depends on the collection type:
* **Batch collections**: 16 days
* **Real-time collections**: 7 days
After that, snapshots are permanently deleted. Bright Data does not recover expired data. Download or export your data before the retention window closes, or configure the scraper to deliver results automatically via webhook, API download, or cloud storage.
## How to report issues
Open the scraper in the Bright Data Scraper Studio control panel and select **Report an issue** from the three-dots menu. Bright Data routes the ticket to a different team based on the issue type:
* **Data** (missing fields, missing records, parsing errors): routed to the scraper engineer. Available only for managed scrapers.
* **Collection and delivery** (incomplete delivery, slow scraper): routed to the support team.
* **Other** (UI issue, product question): routed to the account manager.
Include the affected job ID, a description of the problem, and a screenshot or file if it helps show the issue.
Include:
* The issue category (wrong data, missing records, delivery problem, IDE issue, other)
* A description of the exact problem
* The affected job ID
* A screenshot or file that shows the issue, if possible
Bright Data opens a ticket automatically and the R\&D team handles it.
You receive an email when a Bright Data engineer starts building the scraper and another email when it is ready. You can also track the status on the Scrapers dashboard.
## Related
How Bright Data Scraper Studio works and when to use it
Infrastructure limits, billing, and data retention
# Scraper Studio dashboard features
Source: https://docs.brightdata.com/products/scraper-studio/features
Reference for the Bright Data Scraper Studio dashboard: scraper types, statuses, run statistics columns and how to download collected data.
The Bright Data Scraper Studio dashboard is where every scraper lives. Use it to inspect scraper properties, trigger runs, configure delivery, manage versions and review collection results and statistics. This page is a reference for the controls on the **My Scrapers** page and the Runs tab.
## The Scrapers dashboard
Every scraper you use, whether generated by the AI Agent, built in the Bright Data Scraper Studio IDE, or built by Bright Data teams, appears under **My Scrapers** on the Scrapers dashboard.
The **My Scrapers** page lists every scraper type, not only Scraper Studio scrapers.
### Which scraper types appear on the dashboard?
The **My Scrapers** page lists three scraper types, distinguished by who builds and maintains each scraper.
| Type | Description |
| - | - |
| **Scraper API** | 700+ pre-built scrapers, built and maintained by Bright Data |
| **Scraper Studio** | Scrapers built and maintained by customers or AI |
| **Scraper Studio Managed** | Scrapers built and maintained by the Bright Data team |
### What does each scraper status mean?
Each scraper and run carries a status that shows where it is in the build and collection lifecycle.
| Status | Description |
| - | - |
| **Ready** | The run has been completed |
| **Canceled** | The run was canceled by the customer or system |
| **Active** | The scraper is saved to production and ready to initiate, but has not been triggered yet |
| **Draft** | The scraper is still being built and is not yet saved to production |
| **Running** | The collection is running |
To open a specific scraper's dashboard, click the scraper.
## What does the collection Runs page show?
The Runs page shows per-collection metrics for every run of a scraper.
| Column | Description |
| - | - |
| **ID** | Unique identifier for the collection job |
| **Trigger** | Who triggered the job and how (API, UI, or schedule) |
| **Template** | Template version number |
| **Inputs** | Number of inputs submitted to the collection |
| **Records** | Number of records the scraper returned |
| **Failed crawls** | Number of pages that failed to crawl |
| **Fulfillment** | Total number of unique pages discovered during scraping, including pages that were found but not necessarily scraped |
| **Success rate** | Percentage of inputs that produced successful records |
| **Queued at** | Timestamp when the job entered the queue |
| **Started at** | Timestamp when the scraper started collecting |
| **Finished at** | Timestamp when the scraper finished collecting |
| **Job time** | Total job duration |
| **Estimated time left** | Time remaining for an in-progress job |
| **Queue** | Name of the queue the job is running in (from the trigger's queue name) |
| **Page Loads** | Total page loads consumed by the job |
| **Spent** | Total amount spent on this collection run |
## Collection run actions
The three-dot **Actions** menu includes the following options. Available actions may vary depending on the run status and delivery configuration.
### **Redeliver results**
Use **Redeliver results** to send collected results again to the configured delivery destination.
You can choose:
* **Dataset only**, redelivers only the structured output data.
* **Dataset + media**, redelivers the structured output data together with collected media files, such as images, videos, PDFs, or documents.
Use this when the original delivery failed, the destination was unavailable, or you updated the destination and want to send the same results again.
### **Rerun or cancel a collection**
Use **Rerun job** to start a new run from an existing collection.
You can choose:
* **Rerun all from beginning**, starts a new run using all original input values.
* **Rerun failed pages**, starts a new run only for pages that failed in the selected collection.
Use **Cancel collection** to stop an active collection before it finishes. Data already collected may still be available, depending on the run status and delivery settings.
### **Inspect results and errors**
Use **Quick view** to preview up to 100 collection records in JSON format directly in the control panel.
Use **Errors** to review failed inputs and error details.
Use **Activity** to monitor a collection run over time. The page shows run status, start and finish times, success rate, crawl count, and performance charts. You can change the visible time range, such as **5 min**, **1 h**, **1 d**, or **all**.
The Activity page includes:
* **Run rate** - shows how many pages were crawled, failed, or retried at each point in time, along with the success rate.
* **Pages** - shows discovered pages, open pages, retriable pages, and pages currently being crawled.
* **Inputs** - shows how many input rows were submitted and prepared for crawling.
* **Crawl rate** - shows the system-calculated maximum number of allowed workers and the actual number of workers assigned to the run.
* **Crawl time per page** - shows the average time spent crawling each page.
Use **Activity** to investigate slow runs, error spikes, retries, or changes in crawl performance.
### **Debug crawl**
Use **Debug crawl** to investigate failed or unexpected scraper behavior.
## How do I download the data from the dashboard?
Click the **Download file options** button, select a file format, and the download starts immediately.
Real-time job outputs are stored on Bright Data's side but cannot be downloaded from the dashboard. To retrieve them, use the Scraper Studio API or send them to one of the delivery destinations configured in your delivery options.
## How do I trigger a scraper and get results?
To trigger a scraper, use the three initiation options documented on [Initiate collection and delivery](/products/scraper-studio/initiate-collection-and-delivery-options):
* [Initiate by API](/products/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run)
* [Initiate manually](/products/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run)
* [Schedule a scraper](/products/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run)
To retrieve results, open the scraper from My scrapers and select **Runs** tab. This opens the collection statistics page, where you can download the data.
## Related
Configure trigger methods, file formats and delivery destinations
Reference for every panel and control in the IDE
# Scraper Studio functions reference
Source: https://docs.brightdata.com/products/scraper-studio/functions
Reference for Bright Data Scraper Studio interaction and parser functions: navigation, waits, selectors and parser helpers with parameters and 30+ examples.
This reference documents every function available in Bright Data Scraper Studio's IDE: the interaction code that controls a browser session, and the parser code that turns HTML into structured records. Each function lists its parameters, return value, and a runnable example.
Functions marked with **⭐** work only with the Browser worker and throw an error when called from a Code worker. See [Browser-only functions](#browser-only-functions) for the full list.
## How is Scraper Studio code organized?
A Bright Data Scraper Studio scraper uses two code types:
| Code type | Role | Language and libraries |
| - | - | - |
| Interaction code | Navigates the target site: URL requests, clicks, scrolls, waits, and background traffic capture | JavaScript + Bright Data browser commands |
| Parser code | Extracts and structures data from the HTML returned by interaction code | JavaScript + Cheerio (`$`) |
You move data from one to the other with `parse()` (which runs the parser) and `collect()` (which appends a record to the final dataset).
## Interaction functions
Interaction functions run in the scraper's main JavaScript context and drive the browser or HTTP client. Use them to navigate, wait for elements, interact with the page, capture network traffic, and hand off data to the parser.
### Global objects
| Name | Type | Description |
| - | - | - |
| `input` | object | Input for the current stage, set by the trigger or by a previous `next_stage()`/`run_stage()`/`rerun_stage()` call. |
| `job` | object | Metadata about the current job (for example `job.created`, the job start timestamp). |
| `location` | object | Info about the current browser location. Field: `href`. |
| `parser` | object | Values captured by `tag_response`, `tag_script`, and related tagging functions, available after `wait_for_parser_value()`. |
```js theme={null}
navigate(input.url);
let {created} = job;
console.log('current url', location.href);
```
### Navigation
#### `navigate`, Load a URL in the browser
Navigates the browser to a URL. A 404 status throws a `dead_page` error by default; override with `allow_status`.
**Parameters**
| Parameter | Type | Required | Default | Description |
| - | - | - | - | - |
| `url` | string or URL | Yes | None | Target URL |
| `opt.wait_until` | string | No | `load` | `load`, `domcontentloaded`, `networkidle0`, or `networkidle2` |
| `opt.timeout` | number | No | `30000` | Navigation timeout in milliseconds |
| `opt.referer` | string | No | None | `Referer` header to send |
| `opt.allow_status` | number\[] | No | `[]` | HTTP status codes to accept without throwing |
| `opt.fingerprint` | object | No | None | Override browser fingerprint (`screen.width`, `screen.height`) |
| `opt.sniff_mime` | boolean | No | `false` | Detect the MIME type and store it as `sniffed_mime`. On code workers, only `text/html` and `application/pdf` are detected |
```js theme={null}
navigate(input.url);
navigate('https://example.com');
navigate('https://example.com', {wait_until: 'domcontentloaded'});
navigate('https://example.com', {referer: 'https://google.com'});
navigate('https://example.com', {timeout: 45000});
navigate('https://example.com', {allow_status: [404]});
navigate('https://example.com', {
fingerprint: {screen: {width: 400, height: 400}},
});
navigate('https://example.com', {sniff_mime: true});
```
#### `request`, Make a direct HTTP request
Sends an HTTP request without using a browser. Use on Code worker, or on Browser worker when you want to bypass the browser.
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `url` \| `options` | string or object | Yes | URL string, or an object with `url`, `method`, `headers`, `body` |
```js theme={null}
let res = request('https://www.example.com');
let res = request({
url: 'https://www.example.com',
method: 'POST',
headers: {'Content-type': 'application/json'},
body: {hello: 'world'},
});
```
#### `next_stage`, Queue input for the next stage
Runs the next stage of the scraper in a new browser session with the given input.
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `input` | object | Yes | Input object passed to the next stage |
```js theme={null}
next_stage({url: 'https://example.com', page: 1});
```
#### `run_stage`, Run a specific stage
Runs a named stage of the scraper in a new browser session.
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `stage` | number | Yes | Stage index (starts at 1) |
| `input` | object | Yes | Input object passed to that stage |
```js theme={null}
run_stage(2, {url: 'https://example.com', page: 1});
```
#### `rerun_stage`, Re-run the current stage with new input
Runs this stage again with a new input. Use it to fan out work (for example, one re-run per page in a pagination).
```js theme={null}
rerun_stage({url: 'https://example.com/other-page'});
```
#### `load_sitemap`, Read URLs from an XML sitemap
Loads a sitemap XML file and returns the URL list. Supports sitemap indexes and gzip-compressed sitemaps.
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `options.url` | string | Yes | Sitemap URL |
```js theme={null}
let {pages} = load_sitemap({url: 'https://example.com/sitemap.xml.gz'});
let {children} = load_sitemap({url: 'https://example.com/sitemap-index.xml'});
```
#### `resolve_url`, Follow a URL through redirects
Returns the final URL that the given URL argument leads to.
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `url` | string or URL | Yes | URL to resolve |
```js theme={null}
let {href} = parse().anchor_elem_data;
collect({final_url: resolve_url(href)});
```
#### `redirect_history`, Get the redirect chain
Returns the history of URL redirects since the last `navigate()` call.
```js theme={null}
navigate('http://google.com');
let redirects = redirect_history();
// ['http://google.com', 'http://www.google.com', 'https://www.google.com/']
```
#### `response_headers`, Read the last response headers
Returns the response headers from the last page load.
```js theme={null}
let headers = response_headers();
console.log('content-type', headers['content-type']);
```
#### `status_code`, Read the last response status
Returns the HTTP status code of the last page load.
```js theme={null}
collect({status_code: status_code()});
```
### Waiting on the page ⭐
All wait functions are Browser worker only.
#### ⭐ `wait`, Wait for an element to appear
**Parameters**
| Parameter | Type | Required | Default | Description |
| - | - | - | - | - |
| `selector` | string | Yes | None | CSS selector to wait for |
| `opt.timeout` | number | No | `30000` | Timeout in milliseconds |
| `opt.hidden` | boolean | No | `false` | Wait for the element to be hidden instead of visible |
| `opt.inside` | string | No | None | Selector of an iframe to look inside |
```js theme={null}
wait('#welcome-splash');
wait('.search-results .product');
wait('[href^="/product"]');
wait('#welcome-splash', {timeout: 5000});
wait('#welcome-splash', {hidden: true});
wait('#welcome-splash', {inside: '#iframe_id'});
```
#### ⭐ `wait_any`, Wait for any of several conditions
Waits for any matching condition to succeed. Returns when the first selector resolves.
```js theme={null}
wait_any(['#title', '#notfound']);
```
#### ⭐ `wait_visible`, Wait for an element to be visible
**Parameters**
| Parameter | Type | Required | Default | Description |
| - | - | - | - | - |
| `selector` | string | Yes | None | CSS selector |
| `opt.timeout` | number | No | `30000` | Timeout in milliseconds |
```js theme={null}
wait_visible('#welcome-splash');
wait_visible('#welcome-splash', {timeout: 5000});
```
#### ⭐ `wait_hidden`, Wait for an element to disappear
**Parameters**
| Parameter | Type | Required | Default | Description |
| - | - | - | - | - |
| `selector` | string | Yes | None | CSS selector |
| `opt.timeout` | number | No | `30000` | Timeout in milliseconds |
```js theme={null}
wait_hidden('#welcome-splash');
wait_hidden('#welcome-splash', {timeout: 5000});
```
#### ⭐ `wait_for_text`, Wait for text content
Waits for an element on the page to contain the given text.
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `selector` | string | Yes | CSS selector |
| `text` | string | Yes | Text to wait for |
```js theme={null}
wait_for_text('.location', 'New York');
```
#### `wait_for_parser_value`, Wait for a parser field to be populated
Use after `tag_response()` or `tag_script()` to wait until the captured data is available.
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `field` | string | Yes | Parser field path to wait on |
| `validate_fn` | function | No | Optional callback returning `true` when the value is valid |
| `opt.timeout` | number | No | Timeout in milliseconds |
```js theme={null}
wait_for_parser_value('profile');
wait_for_parser_value('listings.0.price', v => parseInt(v) > 0, {timeout: 5000});
```
#### ⭐ `wait_network_idle`, Wait until the browser network settles
Waits until the browser network has been idle for a given period.
**Parameters**
| Parameter | Type | Required | Default | Description |
| - | - | - | - | - |
| `opt.timeout` | number | No | `500` | Milliseconds of idleness required |
| `opt.ignore` | array | No | `[]` | Patterns (string or RegExp) for requests to exclude |
```js theme={null}
wait_network_idle();
wait_network_idle({
timeout: 1e3,
ignore: [/long_request/, 'https://example.com'],
});
```
#### ⭐ `wait_page_idle`, Wait until DOM mutations stop
Waits until no changes are made to the DOM tree for a given period.
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `opt.idle_timeout` | number | No | Milliseconds of stability required |
| `opt.ignore` | array | No | Selectors to exclude from mutation monitoring |
```js theme={null}
wait_page_idle();
wait_page_idle({
ignore: ['.live-clock', '.carousel'],
idle_timeout: 1000,
});
```
### Element interaction ⭐
All interaction functions require Browser worker.
#### ⭐ `click`, Click an element
Clicks an element, waiting for it to appear first.
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `selector` | string or array | Yes | CSS selector or Shadow DOM selector path |
| `opt.coordinates` | `{x, y}` | No | Click the closest match to given page coordinates |
```js theme={null}
click('#show-more');
$('#show-more').click();
// Click the map pin closest to the center of a map
let box = bounding_box('#map');
let center = {x: (box.left + box.right) / 2, y: (box.top + box.bottom) / 2};
click('.map-pin', {coordinates: center});
```
#### ⭐ `right_click`, Right-click an element
Same as `click` but uses the right mouse button.
```js theme={null}
right_click('#item');
```
#### ⭐ `hover`, Hover over an element
Moves the cursor over an element, waiting for it to appear first.
```js theme={null}
hover('#item');
```
#### ⭐ `mouse_to`, Move the cursor to a coordinate
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `x` | number | Yes | Target X position |
| `y` | number | Yes | Target Y position |
```js theme={null}
mouse_to(0, 0);
```
#### ⭐ `type`, Enter text into an input
Waits for the input to appear, then types the given text.
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `selector` | string | Yes | CSS selector |
| `text` | string or array | Yes | Text to type, or an array of strings and special keys |
| `opt.replace` | boolean | No | Clear existing text before typing |
```js theme={null}
type('#location', 'New York');
type('#location', 'New York', {replace: true});
type('[id$=input-box]', 'search term');
type('#search', ['Some text', 'Enter']);
type('#search', ['Backspace']);
```
#### ⭐ `press_key`, Press a special key
Types special keys like Enter or Backspace in the currently focused input.
```js theme={null}
press_key('Enter');
press_key('Backspace');
```
#### ⭐ `select`, Pick a value from a select element
**Parameters**
| Parameter | Type | Required | Description |
| - | - | - | - |
| `selector` | string | Yes | CSS selector of a `