Skip to main content
This guide shows the coding patterns the Bright Data Scraper Studio team recommends for writing fast, reliable scrapers in the IDE. Each section contrasts a common mistake with the preferred pattern and explains the reason.

​
How do I detect dead pages reliably?

When using navigate(), add a dead_page() condition so the scraper does not retry pages that do not exist. Bright Data Scraper Studio automatically marks HTTP 404 responses as dead, but many sites return 200 with a “not found” template, so you must check for that yourself. Do not wrap wait() in a try/catch and call dead_page() from the catch block. A thrown wait() only tells you the selector did not appear within the timeout, not that the page is actually dead. 
try {
  // Waits 30 seconds for 'ok-selector' even if the page is already dead
  wait('ok-selector');
} catch (e) {
  // You cannot prove the page is dead from a wait timeout alone
  dead_page("Page doesn't exist");
}

​
How do I minimize requests to the browser?

Interaction commands like click, type, el_exists, el_is_visible, wait, and wait_visible each send a request to the browser. Combine selectors into a single call instead of chaining several calls. 
if (!(el_exists('#price1')) || el_exists('#price2')
  || el_exists('#price3') || el_exists('#discount'))
{
    dead_page('No price found');
}

​
How do I paginate without blocking parallelization?

When a site has paginated results and you want data from every page, call rerun_stage() once from the root page for every page you need. Do not call rerun_stage() from inside each page as you walk the pagination: that serializes the work and Bright Data Scraper Studio cannot parallelize the requests. 
navigate(input.url);
let $ = html_load(html());
let next_page_url = $('.next_page').attr('href');
rerun_stage({url: next_page_url});

​
How do I close popups without waiting for them?

Use close_popup('popup_selector', 'close_button_selector') to register a background watcher that closes popups whenever they appear. Do not poll for a popup with wait_visible() before each interaction: popups can appear at any time, and explicit waits add latency on every step. 
navigate('https://example.com');
try {
  wait_visible('.cky-btn-accept', {timeout: 5000});
  click('.cky-btn-accept');
} catch (e) {
    console.log('Accept cookies button does not exist, continue');
}

​
How do I wait for a tagged response before parsing?

When you use tag_response() to capture a background API call, follow it with wait_for_parser_value() to make sure the request has finished before you read parser. Without the wait, the parser may run before the response has arrived and parser.<field> will be undefined. 
tag_response('product', /api\/product/);
navigate('https://example.com');

// Parser code:
// The request may not have finished yet; product could be undefined
let {product} = parser;
return product.data;

​
Should I throw custom error messages?

No. Let built-in errors from Bright Data Scraper Studio bubble up. They include the selector, the timeout, and the stage, which is more useful than a hand-written “Page not loaded properly”. Only throw a custom error when you are checking a domain-specific condition that the platform cannot detect on its own, such as a missing product title. 
try {
  wait('selector1');
  // some code
  wait('selector2');
  // some code
} catch (e) {
  throw "Page not loaded properly"
}

​
How do I handle slow websites without over-extending timeouts?

Keep the default 30-second timeout for most waits. If a specific page is consistently slow, raise it to 45 or 60 seconds. Do not push beyond 60 seconds: a slower peer is usually the cause, and Bright Data Scraper Studio automatically retries with a fresh peer session when a page reports a timeout error. 
// 120 seconds is too long; the platform cannot recycle a stuck peer
wait('selector', {timeout: 120000});

​
Should I build my own retry loop?

No. Bright Data Scraper Studio handles retries at the job level with a new peer session. A custom retry loop inside your scraper reuses the same session, which is the reason the first attempt failed. Report the error and let the platform retry. 
let counter = input.counter || 5;
while (counter > 1) {
  try {
    wait('selector', {timeout: 500});
    click('selector');
    type('selector');
    // some code
    break;
  } catch (e) {
    // rerun_stage creates a new session but this pattern spends extra CPM
    return rerun_stage({...input, counter: --counter});
  }
}

​
Should I wrap parser expressions in try/catch?

No. Use optional chaining (?.) and nullish coalescing (??) instead. A silent try/catch around a property access hides real bugs, and a try/catch around a wait() wastes browser time. 
try {
  const example = obj.prop;
} catch (e) {}

​
How do I extract values from a set of elements in parser code?

Use toArray().map() instead of each(). It is shorter, returns a real array, and reads as a single expression. 
const links = [];
$('.card.product-wrapper').each(function (i, el) {
  links.push({url: $(this).find('h4 a').attr('href')});
})
return links;

​
How do I normalize text in parser code?

Call $(selector).text_sane(). Bright Data Scraper Studio adds this custom method to the Cheerio prototype: it collapses every run of whitespace to a single space and trims the result. For numeric extraction, strip non-digits with a regex. 
$.prototype.clearText = function () {
  return this.text().replace(/\s+/g, ' ').trim();
}

Scraper Studio functions

Full reference for Bright Data Scraper Studio interaction and parser commands

Worker types

Choose between Browser worker and Code worker for your scraper