Disable CAPTCHA Solving

By default, as part of our full proxy unblocking solution, Web Unlocker API also solves CAPTCHAs that are encountered while returning your proxy request.

When disabling CAPTCHA solver, our intelligent algorithm still takes care of the entire ever-changing flow of finding the best proxy network, customizing headers, fingerprinting, and more, but intentionally does not solve CAPTCHAs automatically, giving your team a lightweight, streamlined solution, that broadens the scope of your potential scraping opportunities.

Best for:

  • Scraping data from websites without getting blocked

  • Emulating real-user web behavior

  • Teams that don’t have an unblocking infrastructure in-house and don’t want their scraper to solve CAPTCHAs automatically

Web Unlocker API Premium Domains

Premium domains are a part of Bright Data’s tiered website classification system. These are websites that are more challenging to unblock than others and require additional Web Unlocker API resources.

In this article, we will see the current list of Premium domains, understand how to target them, and go over the special pricing.

Current List of Premium Domains

The premium domains list is updated quarterly using our website classification logic and we’ll notify you via email 30 days in advance of any changes to your domains. You can always access the most up-to-date list in your Web Unlocker API zone.

Loading…

Enable Premium Domains

When creating your Web Unlocker API zone, check the ‘Premium Domains’ box under Special features

Enable Premium Domains

Pricing

Once enabled, the premium price will be reflected in the “Estimated cost” section. Check out the pricing page to see exact numbers, but keep in mind these prices are usually for “Pay as you go” plans and you can enjoy significant discounts if you sign up to a package or talk to our sales people!

Even after enabled, only specific requests to these domains will be priced at the higher rate. Requests to other domains will be kept at the default lower tier.

Geolocation Targeting -country-country_code

Web Unlocker API automatically selects the optimal IP location for targeting your domain, eliminating the need for manual selection in most cases.

If you want to target from a specific country in Web Unlocker API refer to geolocation targeting.

Mobile User-agent Targeting -ua-mobile

By default, Web Unlocker API uses desktop-specific user agents for your requests. To use a mobile user agent instead, simply append -ua-mobile to your request. 

Custom Web Unlocker API

Gain enhanced control over your requests with flexible options to fine-tune website behavior and optimize request handling.

By default, Web Unlocker API automatically manages all request headers, cookies, expect elements etc., to get the best known results and any extra elements that are sent along with the request are disregarded.

Custom Web Unlocker API allows you to override the automated parameters and send your own custom values specific to your needs.

Custom features:

How to enable

In the Control Panel, go to your specific Web Unlocker API zone -> Configuration -> Advanced Settings, and enable the Custom Web Unlocker API feature you want to use.

Enable Premium Domains

Once enabled, you can now send Custom Web Unlocker API requests in accordance with each custom feature’s procedure.

Billing

Unlike the regular Web Unlocker API billing logic, which only charges for successful requests, when any of the above Custom Web Unlocker API features are enabled, you’ll be billed for 100% of the requests (both successful and failed).

Since you are now in control of certain request paramaters, Bright Data can no longer take full responsibility for the unlocking process and its performance.

Be advised:

  • We do not allow cookies for login/authentication purposes

  • Adding custom paramaters to your request may result in blocking and a drop in the success rate.

Manual headers & cookies

Override automated headers/cookies and send your own custom values in order to target specific versions of a website.

Enabling Custom Headers & Cookies results in the following

Manual ‘expect’ elements

Gives you the ability to pass an ‘expect’ parameter along with your request to wait for specific elements or text on the page (i.e. expect.element, text, body).

You can configure this per request with the x-unblock-expect header seen below.

Add x-unblock-expect header

curl -vk \
    -x brd-customer-$CUSTOMER_ID-zone-$ZONE:$PASSWORD@brd.superproxy.io:33335 \
    -H 'x-unblock-expect: {"element": ".some-css-selector"}' \
    https://example.com

Monitor Web Unlocker API Usage

To review your current Web Unlocker API CPM, navigate to My Proxies page, and review the Traffic column.

Understanding usage

The number displayed in the traffic column above is the number of successful requests. In the example above, 115k requests are equal to 115 CPM, so you’ll be billed according to the rate of 10 CPMs for that billing cycle.

How is usage calculated?

Your Web Unlocker API usage is measured by CPM. CPM is the cost of 1000 successful requests, meaning only successful Web Unlocker API requests will count toward your billing.

See our Billing & Pricing page to learn more.

Debugging Web Unlocker API

Sometimes it’s useful to extract some debug info about your requests to understand what happened inside them in more detail.

We provide the x-brd-debug header for this purpose.

To activate the headers, add -debug-full to your proxy username.

This feature is only available for the Web Unlocker and is NOT available for our proxy products.

Enabling debug headers in your request
curl -vk \
    -x brd-customer-$CUSTOMER_ID-zone-$ZONE-debug-full:$PASSWORD@brd.superproxy.io:33335 \
    https://example.com

The format of the x-brd-debug header looks like this:

req_id=hl_d09913c7_a1lw123bkcg; bytes_up=2842; bytes_down=562418; billed=false; destination_ip=162.219.225.118; used_req_headers=accept-language,accept; peer_ip=r868133f79d0c3fa9d7c7ccca0151af2e; peer_country=us; render=false
FieldDescription
req_idThe internal ID of your request into our system. Providing this can be helpful in bug reports as it will help us see details on what happened in your specific request.
bytes_upThe amount of outgoing traffic our system recorded while processing this request
bytes_downThe amount of incoming traffic our system recorded while processing this request
billedWhether our system considers this a billable request or not
destination_ipThe IP address of the remote server used to fetch this data
used_req_headersWhich request headers (if any) were used in the final request. This mainly applies when using Custom Web Unlocker API
peer_ipA unique identifier for the IP address used to make the request. This is mainly useful to validate IP rotation is working as you expect
peer_countryThe country of the peer that was used for the request
renderShows whether the returned page is the result after a browser rendered the HTML or the response body of a single HTTP request

Common Error Codes

Occasionally, for a number of reasons, you might receive an unexpected error code in response to your Web Unlocker API request.

The following list will provide you with a deeper understanding of what the source of the issue may be.

ErrorDescription
404Page not found. Invalid URL, which suggests the URL might be broken or dead.
403The URL you’re trying to access is valid, but you are forbidden from accessing that URL.
502Error code 502 is the most common error for Web Unlocker API users, the descriptive part is under the x-luminati-error-code.
407This error code suggests one of your account credentials is incorrect (password or zone’s name).
429This error code implies a rate limit (rare). In such cases, if the response appears as below, Bright Data is applying auto-throttling to the request, and you should open a ticket or email support@brightdata.com for assistance.
401
411
444
Bad request, usually happens in API requests when headers or cookies missing.
503HTTP error code 503 means “Service Unavailable”. Browser check failed or browser check wasn’t completed
< HTTP/1.1 429 The request was auto-throttled due to low success rate  
< x-luminati-error-code: sr_rate_limit
< x-luminati-error: The request was auto-throttled due to low success rate
< x-brd-error-code: sr_rate_limit
< x-brd-error: The request was auto-throttled due to low success rate
< date: Tue, 23 Jan 2024 17:07:19 GMT
< connection: keep-alive
< keep-alive: timeout=5
< transfer-encoding: chunked
< 
* Connection #0 to host brd.superproxy.io left intact

Get Success Rate Statistics Per Domain

The following API endpoint will provide Web Unlocker API success rate statistics from the past 7 days.

The statistics can be obtained per single domain like example.com or for a wildcard domain like example.* in order to get statistics for all top-level domains.

Note: calling this API endpoint requires using your API token

How to get statistics for a single domain?

    curl "https://api.brightdata.com/unblocker/success_rate/example.com" -H "Content-Type: application/json" -H "Authorization: Bearer API_TOKEN"

How to get statistics for all monitored top level domains?

    curl "https://api.brightdata.com/unblocker/success_rate/example.*" -H "Content-Type: application/json" -H "Authorization: Bearer API_TOKEN"