Bright Data CDP Events

When navigating a page with Scraping Browser, our integrated CAPTCHA solver automatically solves all CAPTCHAs by default. You can monitor this auto-solving process in your code with the following custom CDP events.

Once a CAPTCHA is solved, if there is a form to submit, it will be submitted by default.

CAPTCHA Solver - Automatic Solve

CAPTCHA Solver - Manual Control

If you would like to either manually configure or fully disable our default CAPTCHA solver and instead call the solver manually or solve on your own, see the following CDP commands and functionality.

Integration with .NET Puppeteer Sharp

Integration with the Scraping browser product with C# requires patching the PuppeteerSharp library to add support for websocket authentication. This can be done like the following:

C# PuppeteerSharp
using PuppeteerSharp;  
using System.Net.WebSockets;  
using System.Text;  
  
// Set the authentication credentials  
var auth = "USER:PASS";  
// Construct the WebSocket URL with authentication  
var ws = $"wss://{auth}@zproxy.lum-superproxy.io:9222";  
// Custom WebSocket factory function  
async Task<WebSocket> ws_factory(Uri url, IConnectionOptions options, CancellationToken cancellationToken)  
  
{  
    // Create a new ClientWebSocket instance
    var socket = new ClientWebSocket();  
    // Extract the user information (username and password) from the URL  
    var user_info = url.UserInfo;  
    if (user_info != "")  
    {  
        // Encode the user information in Base64 format  
        var auth = Convert.ToBase64String(Encoding.Default.GetBytes(user_info));  
        // Set the "Authorization" header of the WebSocket options with the encoded credentials  
        socket.Options.SetRequestHeader("Authorization", $"Basic {auth}");  
    }  
  
    // Disable the WebSocket keep-alive interval  
    socket.Options.KeepAliveInterval = TimeSpan.Zero;  
    // Connect to the WebSocket endpoint  
    await socket.ConnectAsync(url, cancellationToken);  
    return socket;  
}  
  
// Create ConnectOptions and configure the options  
var options = new ConnectOptions()  
  
{  
    // Set the BrowserWSEndpoint to the WebSocket URL  
    BrowserWSEndpoint = ws,  
    // Set the WebSocketFactory to the custom factory function  
    WebSocketFactory = ws_factory,  
};  
  
// Connect to the browser using PuppeteerSharp  
Console.WriteLine("Connecting to browser...");  
  
using (var browser = await Puppeteer.ConnectAsync(options))  
{  
    Console.WriteLine("Connected! Navigating...");  
    // Create a new page instance  
    var page = await browser.NewPageAsync();  
    // Navigate to the specified URL  
    await page.GoToAsync("https://example.com");  
    Console.WriteLine("Navigated! Scraping data...");  
    // Get the content of the page  
    var content = await page.GetContentAsync();  
    Console.WriteLine("Done!");  
    Console.WriteLine(content);  
}

Set Proxy Location

The Proxy.setLocation function enables users to dynamically change the location of their proxy to a desired destination within a specified radius. This feature is particularly useful for operations that require geographically specific data or testing of geo-targeted content.

Parameters

lat
float

Specifies the latitude of the desired proxy location.

lon
float

Specifies the longitude of the desired proxy location.

distance
float

Defines the maximum distance from the provided coordinates within which the proxy can be located.

Unit: Kilometers.

Usage:

The Proxy.setLocation command should be invoked before navigating to the site for which the proxy is intended. This ensures that the proxy location is set accurately according to the specified parameters before any data requests are made.

How to run examples

You need to get Scraping Browsers credentials in the control panel. Pass it in format USER:PASS as environment variable AUTH

export AUTH=brd-customer-hl_01234567-zone-scraping_browser:abcdefghijkl
You can also pass TARGET_URL environment variable to change default targeted website.

Code Examples Examples of BrightData’s Scraping Browsers usage using common browser-control libraries.

Please make sure to install required libraries before continuing

Simple scraping of targeted page

Select your pefered tech-stack

#!/usr/bin/env node
const playwright = require('playwright');
const {
    AUTH = 'USER:PASS',
    TARGET_URL = 'https://example.com',
} = process.env;

async function scrape(url = TARGET_URL) {
    if (AUTH == 'USER:PASS') {
        throw new Error(`Provide Scraping Browsers credentials in AUTH`
            + ` environment variable or update the script.`);
    }
    console.log(`Connecting to Browser...`);
    const endpointURL = `wss://${AUTH}@brd.superproxy.io:9222`;
    const browser = await playwright.chromium.connectOverCDP(endpointURL);
    try {
        console.log(`Connected! Navigating to ${url}...`);
        const page = await browser.newPage();
        await page.goto(url, { timeout: 2 * 60 * 1000 });
        console.log(`Navigated! Scraping page content...`);
        const data = await page.content();
        console.log(`Scraped! Data: ${data}`);
    } finally {
        await browser.close();
    }
}

if (require.main == module) {
    scrape().catch(error => {
        console.error(error.stack || error.message || error);
        process.exit(1);
    });
}