Web Unlocker Features
Explore Web Unlocker’s advanced features like CAPTCHA solving, Premium Domains, Headers & Cookies management, and Browser Rendering for efficient web scraping.
Disable CAPTCHA Solving
By default, as part of our full proxy unblocking solution, Web Unlocker also solves CAPTCHAs that are encountered while returning your proxy request.
When disabling CAPTCHA solver, our intelligent algorithm still takes care of the entire ever-changing flow of finding the best proxy network, customizing headers, fingerprinting, and more, but intentionally does not solve CAPTCHAs automatically, giving your team a lightweight, streamlined solution, that broadens the scope of your potential scraping opportunities.
Best for:
- Scraping data from websites without getting blocked
- Emulating real-user web behavior
- Teams that don’t have an unblocking infrastructure in-house and don’t want their scraper to solve CAPTCHAs automatically
Web Unlocker Premium Domains
Premium domains are a part of Bright Data’s tiered website classification system. These are websites that are more challenging to unblock than others and require additional Web Unlocker resources.
In this article, we will see the current list of Premium domains, understand how to target them, and go over the special pricing.
Current List of Premium Domains
This list is frequently changing based on our website classification logic. You can see the most up-to-date list within your proxy zone.
Enable Premium Domains
When creating your Web Unlocker zone, check the ‘Premium Domains’ box under Special features
Pricing
Once enabled, the premium price of $5 more per CPM ($8 total), will be reflected in the “Estimated cost” section.
Even after enabled, only specific requests to these domains will be priced at the higher rate. Requests to other domains will be kept at the default lower tier.
Headers & Cookies Management
Web Unlocker automatically manages all headers and cookies in your request to get the best results. Therefore, any header/cookies sent along with the request are disregarded.
If your use case requires additional headers or cookies, please create a ticket and send us the complete details about your request and what you expect in the response, including which headers/cookies you want to add.
Be advised:
- We do not allow cookies for login/authentication purposes
- Adding special headers/cookies to the request may result in blocking and a drop in the success rate.
- Performance drops due to headers added to the request are not been monitored by bright data
systems and handling blocking events might take longer to resolve
Browser Rendering with -render
Enhance your Web Unlocker requests by utilizing the -render flag to render a browser alongside your request. This feature, available at no additional cost for our Web Unlocker zones, enables more precise data extraction from dynamic web content.
You can enable this feature by adding the -render flag to your proxy username:
brd-customer-<customer_id>-zone-<zone_name>-render
While enabling browser rendering enhances your request, it is also likely to increase the response time due to the additional overhead of initializing the browser. It can be activated per request when needed.
Common Error Codes
Occaisianly, for a number of reasons, you might receive an unexpected error code in response to your Web Unlocker request.
The following list will provide you with a deeper understanding of what the source of the issue may be.
| Error | Description |
|---|---|
404 | Page not found. Invalid URL, which suggests the URL might be broken or dead. |
| 403 | The URL you’re trying to access is valid, but you are forbidden from accessing that URL. |
| 502 | Error code 502 is the most common error for Web Unlocker users, the descriptive part is under the x-luminati-error-code. |
407 | This error code suggests one of your account credentials is incorrect (password or zone’s name). |
429 | This error code implies a rate limit (rare). In such cases, if the response appears as below, Bright Data is applying auto-throttling to the request, and you should open a ticket or email support@brightdata.com for assistance. |
401411444 | Bad request, usually happens in API requests when headers or cookies missing. |
503 | HTTP error code 503 means “Service Unavailable”. Browser check failed or browser check wasn’t completed |
< HTTP/1.1 429 The request was auto-throttled due to low success rate
< x-luminati-error-code: sr_rate_limit
< x-luminati-error: The request was auto-throttled due to low success rate
< x-brd-error-code: sr_rate_limit
< x-brd-error: The request was auto-throttled due to low success rate
< date: Tue, 23 Jan 2024 17:07:19 GMT
< connection: keep-alive
< keep-alive: timeout=5
< transfer-encoding: chunked
<
* Connection #0 to host brd.superproxy.io left intact
Monitor Web Unlocker Usage
To review your current Web Unlocker CPM, navigate to My Proxies page, and review the Traffic column.
Understanding usage
The number displayed in the traffic column above is the number of successful requests. In the example above, 115k requests are equal to 115 CPM, so you’ll be billed according to the rate of 10 CPMs for that billing cycle.
How is usage calculated?
Your Web Unlocker usage is measured by CPM. CPM is the cost of 1000 successful requests, meaning only successful Web Unlocker requests will count toward your billing.
See our Billing & Pricing page to learn more.
Web Unlocker Target Domain Tiers
Web Unlocker allows you to choose between two modes:
- All domains, the default mode where any website can be targeted
- Ecommerce mode where you can target a closed list of popular Ecommerce websites at a discounted rate of 30% below your standard Web Unlocker pricing.
Targeting domains which are not included in a tier
When selecting the E-commerce tier, you may target any of the included domains, while attempting to target other domains will result in an error:
[domain_name is not a part of ecomm tier]
How does it work
When opening a Web Unlocker zone, you can either leave the default All domains mode or switch to the Ecommerce mode, in which case your pricing will be 30% lower and you will be limited to the websites that are included in this mode. Click the ‘check available sites’ link to browse through all included websites.