Skip to main content

​
The IDE

This is where you write and test your JavaScript code for web scraping. It provides a complete coding environment with built-in tools for efficient data extraction. Learn more about The basics of Web Scraping.
A: See more examples
  • Pre-built template code created by our scraper engineers to help you get started quickly with common websites and scraping patterns.
B: Add another step (stage)
  • Use stages when you need to collect data across multiple pages. For example, if you want to collect all products from an Amazon search result page and gather details for each product, you can: Stage 1 (Discovery): Collect product URLs from search results and pass them to Stage 2 Stage 2 (Product Page): Visit each URL to extract product details
  • next_stage, run_stage commands are available to interact between stages.
C: Help D: Debugging Tabs
  • Input : Define your input parameters and run a test (preview) with an input set
  • Output : List of collected data
  • Children : List of children which will be input sets of the next stage
  • Run log : Code execution log
  • Browser console : scraper browser console logs [browser > developer tool > ‘console tab’]
  • Browser network : scraper browser network logs [browser > developer tool > ‘network tab’]
  • Last errors : List of latest error information
E: Input
  • Add input parameter : Define an input parameter including its name and type
  • Add another input : Add the value of an input set to test
  • Preview : Run a test with a selected input set
F: Code settings
  • Error mode : Set a code behavior of the scraper error case
  • Take screenshot : Take screenshots during preview test. You will be able to check loaded pages during the test.

​
Dashboard - scraper action menu

The collescraperctor action menu allows performing different actions with the scraper.
  • Initiate by API - start a data collection without having to enter the control panel
  • Initiate manually - Bright Data’s control panel makes it easy to get started collecting data
  • Run on schedule - select precisely when to collect the data you need
  • Versions - review the modified versions of the scraper
  • Report an issue - You can use this form to communicate any problems you have with the platform, the scraper, or the dataset results
  • Copy link - copy the link of the scraper to share it with your colleagues
  • Tickets - view the status of your tickets
  • Advanced options:
    • Edit the code - edit the scraper’s code within the IDE.
    • Disable scraper - temporarily disable the scraper, but you can reactivate it if needed.
    • Delete scraper - permanently delete the scraper.