> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brightdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Develop a scraper with the IDE

> Build a custom web scraper in the Bright Data Scraper Studio IDE in 5 steps: write interaction and parser code in JavaScript, preview output and save.

This guide walks through building a custom web scraper in the Bright Data Scraper Studio IDE from scratch. You will write interaction code that navigates the target site, parser code that extracts structured fields, and then save the scraper to production and configure delivery. By the end, you will have a runnable scraper you can trigger by API, manually, or on a schedule.

**Time to complete:** about 15 to 30 minutes per scraper, depending on site complexity.

## Prerequisites

* An active [Bright Data account](https://brightdata.com/?hs_signup=1\&utm_source=docs) with access to Scraper Studio
* Basic JavaScript familiarity (variables, functions, control flow)
* A target URL you want to scrape

<Tip>
  If you prefer describing the scraper in plain language instead of writing code, use the [Scraper Studio AI Agent](/datasets/scraper-studio/ai-agent). The agent generates the same kind of scraper the IDE would produce.
</Tip>

## How do I build a scraper in the IDE?

<Steps>
  <Step title="Open the Scraper Studio IDE">
    Go to [brightdata.com/cp/scrapers](https://brightdata.com/cp/scrapers), click **Scraper Studio**, then click **Develop a web scraper (IDE)** to open an empty scraper.
  </Step>

  <Step title="Start from scratch or pick a template">
    Choose a template from the **Templates** panel if your target site has a matching starter, or start from a blank scraper. Templates are pre-built scrapers for common patterns and sites; they are a fast way to learn the idioms Bright Data Scraper Studio expects.
  </Step>

  <Step title="Write the interaction code">
    Interaction code navigates the target site and captures the data you need into the parser. Use the **Interaction code** editor on the left.

    A minimal interaction script:

    ```js theme={null}
    navigate(input.url);
    wait('.product-title');

    let data = parse();
    collect(data);
    ```

    For a multi-page scrape, fan out with `next_stage()`:

    ```js theme={null}
    navigate(input.url);
    wait('.listing');
    let listings = parse().listings;
    for (let url of listings)
      next_stage({url});
    ```

    See [Scraper Studio functions](/datasets/scraper-studio/functions) for every interaction command.
  </Step>

  <Step title="Write the parser code">
    Parser code reads the HTML of the loaded page and returns a structured record. Use Cheerio's jQuery-like `$` selector.

    ```js theme={null}
    return {
      title: $('h1').text_sane(),
      price: new Money(+$('.price').text().replace(/\D+/g, ''), 'USD'),
      image: new Image($('img.product').attr('src')),
      listings: $('.listing a').toArray().map(el => $(el).attr('href')),
    };
    ```

    Parser code returns data to whichever interaction function called `parse()`. See [Scraper Studio functions](/datasets/scraper-studio/functions#parser-functions) for the parser helpers Bright Data Scraper Studio provides.
  </Step>

  <Step title="Choose a worker type">
    In the **Settings** panel, pick the worker type:

    * **Code worker** (faster, cheaper): for static HTML pages and public JSON endpoints
    * **Browser worker**: for JavaScript-rendered pages, clicks, scrolling, popups, or captured background traffic

    Start with Code worker. Switch to Browser worker if you need any function from the [browser-only list](/datasets/scraper-studio/worker-types#which-functions-are-browser-worker-only).
  </Step>

  <Step title="Run a preview">
    Click the **Preview** button to run the scraper against a single test input. The results appear in the **Output** tab. Use the **Run log** and **Browser network** tabs to debug failed runs.

    > **Expected result:** the Output tab shows a structured record with the fields defined in your parser code.
  </Step>

  <Step title="Save to production">
    Click **Save to Production** in the top-right corner. The scraper appears under **My Scrapers** in the control panel and can be triggered by API, manually, or on a schedule.
  </Step>

  <Step title="Configure delivery">
    Open the scraper in **My Scrapers**, click **Delivery preferences**, and choose a destination (API download, webhook, S3, GCS, Azure, SFTP, or email) and a format (JSON, NDJSON, CSV, XLSX). See [Initiate collection and delivery](/datasets/scraper-studio/initiate-collection-and-delivery-options) for every option.
  </Step>

  <Step title="Initiate the scraper">
    Trigger the first production run. Pick the initiation method that matches your workflow:

    * [Initiate by API](/datasets/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run)
    * [Initiate manually](/datasets/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run)
    * [Schedule a scraper](/datasets/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run)
  </Step>
</Steps>

## Frequently asked questions

<AccordionGroup>
  <Accordion title="How do I debug a scraper that fails on a specific input?">
    Open the scraper in the Bright Data Scraper Studio IDE and check the **Last errors** tab. Every failed input is stored with its exact error message and error code (up to the most recent 1,000 failures). Re-run the failing input from the IDE to reproduce the problem locally, fix the interaction or parser code, and save a new production version.
  </Accordion>

  <Accordion title="Can I edit a scraper that was generated by the AI Agent?">
    Yes. Every scraper in Bright Data Scraper Studio, regardless of how it was created, can be opened and edited in the IDE. You can change extraction logic, tweak selectors, add or remove output fields, and change the worker type.
  </Accordion>

  <Accordion title="How do I add fields to the output schema?">
    Click **Edit Schema** in the IDE's output schema panel and add the new fields, or return them from parser code and Bright Data Scraper Studio prompts you to update the schema when you save to production.
  </Accordion>

  <Accordion title="What's the difference between collect() and set_lines()?">
    Use `collect()` to append one record at a time; it is the default way to emit data. Use `set_lines()` when you are collecting records progressively and want the most recent snapshot delivered even if a later step throws an error. Every call to `set_lines()` overrides the previous one. See [collect](/datasets/scraper-studio/functions#collect-append-a-record-to-the-dataset) and [set\_lines](/datasets/scraper-studio/functions#set-lines-set-output-lines-overriding-previous-calls).
  </Accordion>
</AccordionGroup>

## Related

<CardGroup cols={2}>
  <Card title="Scraper Studio functions" icon="code" href="/datasets/scraper-studio/functions">
    Full reference for interaction and parser commands
  </Card>

  <Card title="Best practices" icon="list-check" href="/datasets/scraper-studio/best-practices">
    Recommended patterns for fast, reliable scrapers
  </Card>

  <Card title="Scraper Studio IDE interface" icon="display-code" href="/datasets/scraper-studio/scraper-studio-ide-interface">
    Reference for every panel and control in the IDE
  </Card>

  <Card title="Self-Healing tool" icon="wand-magic-sparkles" href="/datasets/scraper-studio/self-healing-tool">
    Fix broken scrapers and add fields with plain-language prompts
  </Card>
</CardGroup>