Skip to main content
This page answers the questions the Bright Data support team hears most often about Scraper Studio. If you need a walkthrough instead of a quick answer, start with Understanding Scraper Studio.

​
Frequently asked general questions

A Bright Data web scraper is an automated script that collects public web data at scale through Bright Data’s proxy and unblocking infrastructure. It returns the collected data in a structured format, such as JSON, NDJSON, CSV or XLSX. (Parquet is also supported for selected delivery options.) Results can be delivered to an API endpoint, webhook, cloud storage, Snowflake or SFTP. Bright Data maintains thousands of pre-built scrapers for popular sites in the Scrapers Library.
Bright Data Scraper Studio is a cloud-hosted environment for building custom scrapers. It offers two modes: an AI Agent that generates a scraper from a natural-language description, and an IDE where you write JavaScript directly. Both modes run on the same Bright Data proxy and unblocking infrastructure. See Understanding Scraper Studio.
The Scrapers Library contains pre-built scrapers Bright Data maintains for popular sites such as Amazon, LinkedIn and Instagram. Bright Data Scraper Studio is the environment you use to build custom scrapers when the site you need is not in the library.
No. The best practice is to build a scraper per specific domain. This lets you analyze logs accurately, debug success rate issues more easily and keep the scraping process efficient.
You do not have to provide every URL. The Bright Data Scraper Studio AI Agent can build different scraper types depending on what input you already have:
  • PDP: provide a list of product URLs you already have; get one row per URL.
  • Discovery: provide a category or listing URL; get one row per item found on that listing.
  • Discovery + PDP: provide a category or listing URL; get full per-product detail for every item.
  • Search: provide a keyword and optional country when you do not have URLs.
  • Sitemap: provide a domain or sitemap.xml URL; get either all URLs found in the sitemap or full per-page detail for every URL.
Each scraper is scoped to one data shape, so the AI Agent is not a general crawler; do not pass a homepage and ask for “everything.” For deeper link discovery across a site, build a multi-stage scraper in the Bright Data Scraper Studio IDE with next_stage(), rerun_stage() and load_sitemap(). See What can the AI Agent build? and Functions.
Bright Data Scraper Studio scrapes the web, not native mobile apps. For mobile layouts, the Browser worker can emulate a mobile device with emulate_device('iPhone X') so the target site serves its mobile web pages. Native iOS or Android app data is out of scope. See Functions.
No. For sites the pre-built Scraper APIs already cover, such as the LinkedIn Scraper API, use the pre-built Scraper API. Build in Bright Data Scraper Studio only when the site you need is not in the Scrapers Library. Since November 13, 2025, LinkedIn restricts public access to the Position, Experience and Education fields, an industry-wide change that affects all data providers. See LinkedIn data policy change.

​
Inputs, outputs and schemas

An input is the parameter set Bright Data Scraper Studio passes into the scraper for a single run. Typical inputs are a URL, a search keyword, a product ID or ASIN, a profile handle or a date range. Multiple inputs can be passed in one job via CSV upload or the API.
The output is the structured data the scraper returns for an input. Bright Data Scraper Studio can deliver output in JSON, NDJSON, CSV, XLSX or Parquet depending on the scraper’s delivery preferences and selected delivery option.
One input can generate multiple records. For example, if you submit 5 product listing URLs and each listing page contains 20 products, the scraper returns 100 records from 5 inputs. The statistics page counts records, not inputs.
A search scraper takes a keyword as input instead of a URL. Bright Data Scraper Studio runs a search on the target site and extracts data from the result pages. Use a search scraper when you do not have specific URLs to scrape.
A discovery scraper collects data from listing pages such as search results, category pages or directories. It extracts fields that appear directly on the listing (titles, prices, ratings) and can also collect product URLs or IDs for a follow-up product-page scrape.
When the input or output schema changes, the scraper must be updated to match. If you trigger the scraper before Bright Data has updated it, you will see an input(output)_schema_incompatible error.To trigger the scraper anyway and ignore the schema mismatch, click Trigger anyway in the UI or add a parameter to your API request:
  • Output schema incompatible: override_incompatible_schema=1
  • Input schema incompatible: override_incompatible_input_schema=1
Yes. Turn on the html system field in the output schema to get a full HTML snapshot of the page as a file, downloaded to your configured storage destination. In interaction code, tag_html() saves the HTML of the current page to a named field. On Browser worker scrapers, you can also turn on WARC output to archive the exact HTTP responses the browser received for every page the scraper collects. See Input and output schema, Functions and WARC snapshots.
Yes. Add an image, pdf, doc or video field to the output schema and enable Download, and Bright Data Scraper Studio stores the file in your configured delivery destination. File downloads are billed separately from page loads. See Input and output schema.

​
Building and editing scrapers

You have three options in Bright Data Scraper Studio:
  • Build with the AI Agent: describe the data you want in plain language. See Scraper Studio AI Agent.
  • Build in the IDE: write JavaScript directly. See Develop a scraper with the IDE.
  • Request a managed scraper: Bright Data’s data team builds and maintains the scraper for you.
Yes. Every scraper, including ones the AI Agent generated, can be opened and edited in the Bright Data Scraper Studio IDE. You can change extraction logic, modify selectors, add or remove output fields and tune performance. If you prefer not to write code, use the Self-Healing tool to request changes in plain language.
Pass the AI Agent a target URL and an optional description of the data you want. The agent asks clarifying questions, generates an output schema for your review, and writes the full scraper code once you approve the schema. You can then run the scraper immediately or schedule recurring runs. See Scraper Studio AI Agent for a full walkthrough.
Yes. You own every line of the scraper, including code the AI Agent generates, and you can read and edit it in the Bright Data Scraper Studio IDE. Scrapers are plain JavaScript, so you can copy the code out and keep it in your own repository. To run a scraper outside Bright Data you would need to supply your own proxy and unblocking infrastructure.
Yes. Choose the Browser worker for sites that render content with JavaScript; it runs a full headless browser so you can wait for elements, click, scroll and capture background network calls. Choose the Code worker for static HTML and HTTP responses, which is faster and cheaper. See Worker types.
It depends on the worker type. The Browser worker runs a real headless browser that executes JavaScript and can click, scroll and fill in forms. The Code worker sends raw HTTP requests, like curl, without running JavaScript, which makes it faster and cheaper. See Worker types.
Yes. Build the scraper in the Bright Data Scraper Studio IDE rather than with the AI Agent, and run it on a Browser worker, which supports login flows, sessions and form input. In interaction code, use set_session_cookie() to set a session cookie or set_session_headers() to send extra HTTP headers. See Functions and Worker types.
Yes. Call country() with a two-character ISO country code to route the scraper through that country, or proxy_location() for finer control by country, latitude, longitude and radius. proxy_location() requires a Browser worker. Call preserve_proxy_session() to reuse the same proxy session across child stages. See Functions.
Split the scraper into stages. Each stage is a separate crawl step, and next_stage() sends new input values, such as a result URL, to the next stage. The page that calls next_stage() is the parent crawl and each page it creates is a child crawl, which you can trace in the crawl inspector. Parser code is available only in the last stage, so earlier stages use load_html to read the page. See Web scraping basics in Scraper Studio and Complete Scraper Studio IDE examples.

​
Self-Healing

Yes, if you turn on Auto Self-Healing. Bright Data Scraper Studio repairs scrapers in two ways. Auto Self-Healing starts a healing job when a scraper’s success rate falls below a threshold you set (default 40 percent) or one of six supported scraper errors occurs. You can also trigger Self-Healing on demand from the IDE with the Self-Healing tool, with bdata scraper heal in the Bright Data CLI, or with POST /dca/collectors/{collector_id}/refactor_template (Trigger Self-Healing). With Auto save code Off, Scraper Studio emails the suggested fix for review before it reaches production. Every path keeps the same Collector ID.
Yes. With Auto save code Off, Auto Self-Healing runs the healing job without saving the generated code and emails the users selected under Notifications to review the suggested fix first. In the Self-Healing tool, accepted changes go to a draft and reach production only after you click Save to Production. If you already saved a fix to production, open the Versions menu on the scraper dashboard to roll back to an earlier version. See Auto Self-Healing and Self-Healing tool.

​
Running and triggering scrapers

Bright Data Scraper Studio supports three trigger methods:
  • By API: regular request, queue request, or replace request
  • Manually: from the control panel
  • On a schedule: hourly, daily, weekly or custom
See Initiate collection and delivery.
A queued request tells Bright Data Scraper Studio to wait until the previous request for the same scraper finishes before starting the next one. Use it when you want serial execution instead of running multiple jobs in parallel.
Bright Data Scraper Studio runs up to 100 batch jobs in parallel per scraper. Additional jobs queue automatically and start as capacity frees up. See Scraper Studio specifications for full limits.
In the Bright Data Scraper Studio dashboard, click the Bug icon under Failed crawls to open the scraper in the IDE. Failed inputs appear in the Last errors tab with the exact error message and error code. Bright Data stores the last 1,000 errors per virtual job so you can re-run failed inputs and diagnose the issue.

​
Blocking and unblocking

Bright Data Scraper Studio unblocks automatically on every run through Bright Data’s proxy and unblocking infrastructure, and the IDE adds solve_captcha(), detect_block(), country() and proxy_location() for edge cases. Self-Healing repairs scraper code, not blocks, so for persistent block or blocked errors see How does Scraper Studio handle blocking and CAPTCHAs?.

​
Billing and limits

CPM stands for “cost per mille”, meaning 1,000 page loads. Bright Data Scraper Studio bills page loads in CPM units. Current rates are on the pricing page.
A billable event is any function that causes Bright Data Scraper Studio to load a page or perform a network request:
  • navigate()
  • request()
  • load_more()
  • Media file download (billed per GB, separate from CPM)
Scraper Studio does not have a separate page-load or record allotment, but it is covered by the account-level free tier. Every new account gets 5,000 free credits per month, a single shared pool usable across the Web Unlocker API, SERP API, Web Scraper API, Scraper Studio and Browser API. In the free tier, Scraper Studio consumes one credit per page load drawn from that pool; records are not billed by default. New accounts also get a one-time onboarding bonus credit usable on any Bright Data product. See the free tier and billing pages.

​
Snapshots and data retention

Snapshot retention depends on the collection type:
  • Batch collections: 16 days
  • Real-time collections: 7 days
After that, snapshots are permanently deleted. Bright Data does not recover expired data. Download or export your data before the retention window closes, or configure the scraper to deliver results automatically via webhook, API download, or cloud storage.

​
How to report issues

Open the scraper in the Bright Data Scraper Studio control panel and select Report an issue from the three-dots menu. Bright Data routes the ticket to a different team based on the issue type:
  • Data (missing fields, missing records, parsing errors): routed to the scraper engineer. Available only for managed scrapers.
  • Collection and delivery (incomplete delivery, slow scraper): routed to the support team.
  • Other (UI issue, product question): routed to the account manager.
Include the affected job ID, a description of the problem, and a screenshot or file if it helps show the issue.
Include:
  • The issue category (wrong data, missing records, delivery problem, IDE issue, other)
  • A description of the exact problem
  • The affected job ID
  • A screenshot or file that shows the issue, if possible
Bright Data opens a ticket automatically and the R&D team handles it.
You receive an email when a Bright Data engineer starts building the scraper and another email when it is ready. You can also track the status on the Scrapers dashboard.

Understanding Scraper Studio

How Bright Data Scraper Studio works and when to use it

Specifications

Infrastructure limits, billing and data retention