Web Unlocker Features
Explore Web Unlocker’s advanced features like CAPTCHA solving, Premium Domains, Headers & Cookies management, and Browser Rendering for efficient web scraping.
Disable CAPTCHA Solving
By default, as part of our full proxy unblocking solution, Web Unlocker also solves CAPTCHAs that are encountered while returning your proxy request.
When disabling CAPTCHA solver, our intelligent algorithm still takes care of the entire ever-changing flow of finding the best proxy network, customizing headers, fingerprinting, and more, but intentionally does not solve CAPTCHAs automatically, giving your team a lightweight, streamlined solution, that broadens the scope of your potential scraping opportunities.
Best for:
- Scraping data from websites without getting blocked
- Emulating real-user web behavior
- Teams that don’t have an unblocking infrastructure in-house and don’t want their scraper to solve CAPTCHAs automatically
Web Unlocker Premium Domains
Premium domains are a part of Bright Data’s tiered website classification system. These are websites that are more challenging to unblock than others and require additional Web Unlocker resources.
In this article, we will see the current list of Premium domains, understand how to target them, and go over the special pricing.
Current List of Premium Domains
This list is frequently changing based on our website classification logic. You can see the most up-to-date list within your proxy zone.
Enable Premium Domains
When creating your Web Unlocker zone, check the ‘Premium Domains’ box under Special features
Pricing
Once enabled, the premium price will be reflected in the “Estimated cost” section. Check out the pricing page to see exact numbers, but keep in mind these prices are usually for “Pay as you go” plans and you can enjoy significant discounts if you sign up to a package or talk to our sales people!
Even after enabled, only specific requests to these domains will be priced at the higher rate. Requests to other domains will be kept at the default lower tier.
Geolocation Targeting -country-country_code
Web Unlocker automatically selects the optimal IP location for targeting your domain, eliminating the need for manual selection in most cases.
If you want to target from a specific country in Web Unlocker refer to geolocation targeting.
Mobile User-agent Targeting -ua-mobile
By default, Web Unlocker uses desktop-specific user agents for your requests. To use a mobile user agent instead, simply append -ua-mobile to your request.
Custom Web Unlocker
Gain enhanced control over your requests with flexible options to fine-tune website behavior and optimize request handling.
By default, Web Unlocker automatically manages all request headers, cookies, expect elements etc., to get the best known results and any extra elements that are sent along with the request are disregarded.
Custom Web Unlocker allows you to override the automated parameters and send your own custom values specific to your needs.
Custom features:
How to enable
In the Control Panel, go to your specific Web Unlocker zone -> Configuration -> Advanced Settings, and enable the Custom Web Unlocker feature you want to use.
Once enabled, you can now send Custom Web Unlocker requests in accordance with each custom feature’s procedure.
Billing
Unlike the regular Web Unlocker billing logic, which only charges for successful requests, when any of the above Custom Web Unlocker features are enabled, you’ll be billed for 100% of the requests (both successful and failed).
Since you are now in control of certain request paramaters, Bright Data can no longer take full responsibility for the unlocking process and its performance.
Be advised:
- We do not allow cookies for login/authentication purposes
- Adding custom paramaters to your request may result in blocking and a drop in the success rate.
Manual headers & cookies
Override automated headers/cookies and send your own custom values in order to target specific versions of a website.
Enabling Custom Headers & Cookies results in the following
Manual ‘expect’ elements
Gives you the ability to pass an ‘expect’ parameter along with your request to wait for specific elements or text on the page (i.e. expect.element, text, body).
You can configure this per request with the x-unblock-expect header seen below.
Add x-unblock-expect header
Monitor Web Unlocker Usage
To review your current Web Unlocker CPM, navigate to My Proxies page, and review the Traffic column.
Understanding usage
The number displayed in the traffic column above is the number of successful requests. In the example above, 115k requests are equal to 115 CPM, so you’ll be billed according to the rate of 10 CPMs for that billing cycle.
How is usage calculated?
Your Web Unlocker usage is measured by CPM. CPM is the cost of 1000 successful requests, meaning only successful Web Unlocker requests will count toward your billing.
See our Billing & Pricing page to learn more.
Web Unlocker Target Domain Tiers
Web Unlocker allows you to choose between two modes:
- All domains, the default mode where any website can be targeted
- Ecommerce mode where you can target a closed list of popular Ecommerce websites at a discounted rate of 30% below your standard Web Unlocker pricing.
Targeting domains which are not included in a tier
When selecting the E-commerce tier, you may target any of the included domains, while attempting to target other domains will result in an error:
[domain_name is not a part of ecomm tier]
How does it work
When opening a Web Unlocker zone, you can either leave the default All domains mode or switch to the Ecommerce mode, in which case your pricing will be 30% lower and you will be limited to the websites that are included in this mode. Click the ‘check available sites’ link to browse through all included websites.
Debugging Web Unlocker
Sometimes it’s useful to extract some debug info about your requests to understand what happened inside them in more detail.
We provide the x-brd-debug header for this purpose.
To activate the headers, add -debug-full to your proxy username.
The format of the x-brd-debug header looks like this:
| Field | Description |
|---|---|
| req_id | The internal ID of your request into our system. Providing this can be helpful in bug reports as it will help us see details on what happened in your specific request. |
| bytes_up | The amount of outgoing traffic our system recorded while processing this request |
| bytes_down | The amount of incoming traffic our system recorded while processing this request |
| billed | Whether our system considers this a billable request or not |
| destination_ip | The IP address of the remote server used to fetch this data |
| used_req_headers | Which request headers (if any) were used in the final request. This mainly applies when using Custom Web Unlocker |
| peer_ip | A unique identifier for the IP address used to make the request. This is mainly useful to validate IP rotation is working as you expect |
| peer_country | The country of the peer that was used for the request |
| render | Shows whether the returned page is the result after a browser rendered the HTML or the response body of a single HTTP request |
Common Error Codes
Occasionally, for a number of reasons, you might receive an unexpected error code in response to your Web Unlocker request.
The following list will provide you with a deeper understanding of what the source of the issue may be.
| Error | Description |
|---|---|
404 | Page not found. Invalid URL, which suggests the URL might be broken or dead. |
| 403 | The URL you’re trying to access is valid, but you are forbidden from accessing that URL. |
| 502 | Error code 502 is the most common error for Web Unlocker users, the descriptive part is under the x-luminati-error-code. |
407 | This error code suggests one of your account credentials is incorrect (password or zone’s name). |
429 | This error code implies a rate limit (rare). In such cases, if the response appears as below, Bright Data is applying auto-throttling to the request, and you should open a ticket or email support@brightdata.com for assistance. |
401411444 | Bad request, usually happens in API requests when headers or cookies missing. |
503 | HTTP error code 503 means “Service Unavailable”. Browser check failed or browser check wasn’t completed |
Get Success Rate Statistics Per Domain
The following API endpoint will provide Web Unlocker success rate statistics from the past 7 days.
The statistics can be obtained per single domain like example.com or for a wildcard domain like example.* in order to get statistics for all top-level domains.
Note: calling this API endpoint requires using your API token
How to get statistics for a single domain?
How to get statistics for all monitored top level domains?
Was this page helpful?