Web Unlocker API Features
Explore Web Unlocker API’s advanced features like CAPTCHA solving, Premium Domains, Headers & Cookies management, and Browser Rendering for efficient web scraping.
Disable CAPTCHA Solving
By default, as part of our full proxy unblocking solution, Web Unlocker API also solves CAPTCHAs that are encountered while returning your proxy request.
When disabling CAPTCHA solver, our intelligent algorithm still takes care of the entire ever-changing flow of finding the best proxy network, customizing headers, fingerprinting, and more, but intentionally does not solve CAPTCHAs automatically, giving your team a lightweight, streamlined solution, that broadens the scope of your potential scraping opportunities.
Best for:
-
Scraping data from websites without getting blocked
-
Emulating real-user web behavior
-
Teams that don’t have an unblocking infrastructure in-house and don’t want their scraper to solve CAPTCHAs automatically
Web Unlocker API Premium Domains
Premium domains are a part of Bright Data’s tiered website classification system. These are websites that are more challenging to unblock than others and require additional Web Unlocker API resources.
In this article, we will see the current list of Premium domains, understand how to target them, and go over the special pricing.
Current List of Premium Domains
The premium domains list is updated quarterly using our website classification logic and we’ll notify you via email 30 days in advance of any changes to your domains. You can always access the most up-to-date list in your Web Unlocker API zone.
Enable Premium Domains
When creating your Web Unlocker API zone, check the ‘Premium Domains’ box under Special features
Pricing
Once enabled, the premium price will be reflected in the “Estimated cost” section. Check out the pricing page to see exact numbers, but keep in mind these prices are usually for “Pay as you go” plans and you can enjoy significant discounts if you sign up to a package or talk to our sales people!
Even after enabled, only specific requests to these domains will be priced at the higher rate. Requests to other domains will be kept at the default lower tier.
Geolocation Targeting -country-country_code
Web Unlocker API automatically selects the optimal IP location for targeting your domain, eliminating the need for manual selection in most cases.
If you want to target from a specific country in Web Unlocker API refer to geolocation targeting.
Mobile User-agent Targeting -ua-mobile
By default, Web Unlocker API uses desktop-specific user agents for your requests. To use a mobile user agent instead, simply append -ua-mobile
to your request.
Custom Web Unlocker API
Gain enhanced control over your requests with flexible options to fine-tune website behavior and optimize request handling.
By default, Web Unlocker API automatically manages all request headers, cookies, expect elements etc., to get the best known results and any extra elements that are sent along with the request are disregarded.
Custom Web Unlocker API allows you to override the automated parameters and send your own custom values specific to your needs.
Custom features:
How to enable
In the Control Panel, go to your specific Web Unlocker API zone -> Configuration -> Advanced Settings, and enable the Custom Web Unlocker API feature you want to use.
Once enabled, you can now send Custom Web Unlocker API requests in accordance with each custom feature’s procedure.
Billing
Unlike the regular Web Unlocker API billing logic, which only charges for successful requests, when any of the above Custom Web Unlocker API features are enabled, you’ll be billed for 100% of the requests (both successful and failed).
Since you are now in control of certain request paramaters, Bright Data can no longer take full responsibility for the unlocking process and its performance.
Be advised:
-
We do not allow cookies for login/authentication purposes
-
Adding custom paramaters to your request may result in blocking and a drop in the success rate.
Manual headers & cookies
Override automated headers/cookies and send your own custom values in order to target specific versions of a website.
Enabling Custom Headers & Cookies results in the following
Manual ‘expect’ elements
Gives you the ability to pass an ‘expect’ parameter along with your request to wait for specific elements or text on the page (i.e. expect.element, text, body).
You can configure this per request with the x-unblock-expect
header seen below.
Add x-unblock-expect
header
Monitor Web Unlocker API Usage
To review your current Web Unlocker API CPM, navigate to My Proxies page, and review the Traffic column.
Understanding usage
The number displayed in the traffic column above is the number of successful requests. In the example above, 115k requests are equal to 115 CPM, so you’ll be billed according to the rate of 10 CPMs for that billing cycle.
How is usage calculated?
Your Web Unlocker API usage is measured by CPM. CPM is the cost of 1000 successful requests, meaning only successful Web Unlocker API requests will count toward your billing.
See our Billing & Pricing page to learn more.
Debugging Web Unlocker API
Sometimes it’s useful to extract some debug info about your requests to understand what happened inside them in more detail.
We provide the x-brd-debug
header for this purpose.
To activate the headers, add -debug-full
to your proxy username.
This feature is only available for the Web Unlocker and is NOT available for our proxy products.
The format of the x-brd-debug
header looks like this:
Field | Description |
---|---|
req_id | The internal ID of your request into our system. Providing this can be helpful in bug reports as it will help us see details on what happened in your specific request. |
bytes_up | The amount of outgoing traffic our system recorded while processing this request |
bytes_down | The amount of incoming traffic our system recorded while processing this request |
billed | Whether our system considers this a billable request or not |
destination_ip | The IP address of the remote server used to fetch this data |
used_req_headers | Which request headers (if any) were used in the final request. This mainly applies when using Custom Web Unlocker API |
peer_ip | A unique identifier for the IP address used to make the request. This is mainly useful to validate IP rotation is working as you expect |
peer_country | The country of the peer that was used for the request |
render | Shows whether the returned page is the result after a browser rendered the HTML or the response body of a single HTTP request |
Common Error Codes
Occasionally, for a number of reasons, you might receive an unexpected error code in response to your Web Unlocker API request.
The following list will provide you with a deeper understanding of what the source of the issue may be.
Error | Description |
---|---|
404 | Page not found. Invalid URL, which suggests the URL might be broken or dead. |
403 | The URL you’re trying to access is valid, but you are forbidden from accessing that URL. |
502 | Error code 502 is the most common error for Web Unlocker API users, the descriptive part is under the x-luminati-error-code . |
407 | This error code suggests one of your account credentials is incorrect (password or zone’s name). |
429 | This error code implies a rate limit (rare). In such cases, if the response appears as below, Bright Data is applying auto-throttling to the request, and you should open a ticket or email support@brightdata.com for assistance. |
401 411 444 | Bad request, usually happens in API requests when headers or cookies missing. |
503 | HTTP error code 503 means “Service Unavailable”. Browser check failed or browser check wasn’t completed |
Get Success Rate Statistics Per Domain
The following API endpoint will provide Web Unlocker API success rate statistics from the past 7 days.
The statistics can be obtained per single domain like example.com
or for a wildcard domain like example.*
in order to get statistics for all top-level domains.
Note: calling this API endpoint requires using your API token
How to get statistics for a single domain?
How to get statistics for all monitored top level domains?
Was this page helpful?