​
Disable CAPTCHA Solving

By default, as part of our full proxy unblocking solution, Web Unlocker API also solves CAPTCHAs that are encountered while returning your proxy request. When disabling CAPTCHA solver, our intelligent algorithm still takes care of the entire ever-changing flow of finding the best proxy network, customizing headers, fingerprinting, and more, but intentionally does not solve CAPTCHAs automatically, giving your team a lightweight, streamlined solution, that broadens the scope of your potential scraping opportunities. Best for:
  • Scraping data from websites without getting blocked
  • Emulating real-user web behavior
  • Teams that don’t have an unblocking infrastructure in-house and don’t want their scraper to solve CAPTCHAs automatically

​
Web Unlocker API Premium Domains

Premium domains are a part of Bright Data’s tiered website classification system. These are websites that are more challenging to unblock than others and require additional Web Unlocker API resources. In this article, we will see the current list of Premium domains, understand how to target them, and go over the special pricing.
The premium domains list is updated quarterly using our website classification logic and we’ll notify you via email 30 days in advance of any changes to your domains. You can always access the most up-to-date list in your Web Unlocker API zone.

​
Current List of Premium Domains

​
Enable Premium Domains

When creating your Web Unlocker API zone, check the ‘Premium Domains’ box under Special features
Enable Premium Domains

​
Pricing

Once enabled, the premium price will be reflected in the “Estimated cost” section. Check out the pricing page to see exact numbers, but keep in mind these prices are usually for “Pay as you go” plans and you can enjoy significant discounts if you sign up to a package or talk to our sales people!
Even after enabled, only specific requests to these domains will be priced at the higher rate. Requests to other domains will be kept at the default lower tier.

​
Geolocation Targeting -country-country_code

Web Unlocker automatically selects the optimal IP location for targeting your domain, reducing the need for manual configuration in most cases. Manual geo-targeting is useful when accessing region-restricted or location-specific data.
If you want to target from a specific country in Web Unlocker API refer to geolocation targeting.

​
Mobile User-agent Targeting -ua-mobile

By default, Web Unlocker API uses desktop-specific user agents for your requests. To use a mobile user agent instead, simply append -ua-mobile to your request. 

​
Scrape as markdown

Web Unlocker is able to live convert your pages from HTML to markdown. This makes it easier to feed your LLM training system, for example. To activate the feature, add the x-unblock-data-format: markdown header when using the native proxy interface, or set data_format: 'markdown' when using the API. 
API_KEY=your_api_key_here
ZONE=your_zone_here
curl -v \
   -H "Authorization: Bearer $API_KEY" \
   -H 'content-type: application/json' \
   --data '{"url": "https://example.com", "zone": "'$ZONE'", "format": "raw", "data_format": "markdown"}' \
   https://api.brightdata.com/request

​
Get a screenshot

Web Unlocker is able to take a screenshot of the page you are trying to scrape. This can be useful for debugging or for monitoring the page’s appearance. To activate the feature, add the x-unblock-data-format: screenshot header when using the native proxy interface or add data_format: screenshot to request body when using API interface. Output format is .png 
curl -k https://api.brightdata.com/request \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR API KEY>" \
-d '{"zone":"unblocker","url":"https://example.com","format":"raw","data_format":"screenshot"}' \
--silent --output example_com.png

​
Custom Web Unlocker API

Gain enhanced control over your requests with flexible options to fine-tune website behavior and optimize request handling. By default, Web Unlocker API automatically manages all request headers, cookies, expect elements etc., to get the best known results and any extra elements that are sent along with the request are disregarded. Custom Web Unlocker API allows you to override the automated parameters and send your own custom values specific to your needs.

​
Custom features:

​
How to enable

In the Control Panel, go to your specific Web Unlocker API zone -> Configuration -> Advanced Settings, and enable the Custom Web Unlocker API feature you want to use. Enable Premium Domains Once enabled, you can now send Custom Web Unlocker API requests in accordance with each custom feature’s procedure.

​
Billing

Unlike the regular Web Unlocker API billing logic, which only charges for successful requests, when any of the above Custom Web Unlocker API features are enabled, you’ll be billed for 100% of the requests (both successful and failed). Since you are now in control of certain request paramaters, Bright Data can no longer take full responsibility for the unlocking process and its performance.
Be advised:
  • We do not allow cookies for login/authentication purposes
  • Adding custom paramaters to your request may result in blocking and a drop in the success rate.

​
Manual headers & cookies

Override automated headers/cookies and send your own custom values in order to target specific versions of a website.
Enabling Custom Headers & Cookies results in the following

​
Manual ‘expect’ elements

In case of receiving partially rendered/loaded pages, you can use the x-unblock-expect header to instruct Web Unlocker to wait for specific elements or text before returning the response.
This feature allows you to define expectations—such as a CSS selector, specific text, or the page body—that must be present before the page is considered fully loaded. You can configure this per request with the x-unblock-expect header seen below. Add header 
curl -vk \
    -x brd-customer-$CUSTOMER_ID-zone-$ZONE:$PASSWORD@brd.superproxy.io:33335 \
    -H 'x-unblock-expect: {"element": ".some-css-selector"}' \
    https://example.com

​
Amazon-specific geolocation headers

Web Unlocker allows you to pass custom headers to simulate a user-selected city and ZIP code on Amazon, enabling access to region-specific content, pricing, and delivery options.
  • x-unblock-city - Simulates selecting a city.
  • x-unblock-zipcode - Simulates selecting a ZIP code on Amazon.
curl -vk \
-x brd-customer-$CUSTOMER_ID-zone-$ZONE:$PASSWORD@brd.superproxy.io:33335 \
-H 'x-unblock-zipcode: 10001' \
-H 'x-unblock-city: New York' \
https://www.amazon.com/

​
Monitor Web Unlocker API Usage

To review your current Web Unlocker API CPM, navigate to My Proxies page, and review the Traffic column.

​
Understanding usage

The number displayed in the traffic column above is the number of successful requests. In the example above, 115k requests are equal to 115 CPM, so you’ll be billed according to the rate of 10 CPMs for that billing cycle.

​
How is usage calculated?

Your Web Unlocker API usage is measured by CPM. CPM is the cost of 1000 successful requests, meaning only successful Web Unlocker API requests will count toward your billing. See our Billing & Pricing page to learn more.

​
Debugging Web Unlocker API

Sometimes it’s useful to extract some debug info about your requests to understand what happened inside them in more detail. We provide the x-brd-debug header for this purpose. To activate the headers, add -debug-full to your proxy username.
This feature is only available for the Web Unlocker and is NOT available for our proxy products.
Enabling debug headers in your request
curl -vk \
    -x brd-customer-$CUSTOMER_ID-zone-$ZONE-debug-full:$PASSWORD@brd.superproxy.io:33335 \
    https://example.com
The format of the x-brd-debug header looks like this: 
req_id=hl_d09913c7_a1lw123bkcg; bytes_up=2842; bytes_down=562418; billed=false; destination_ip=162.219.225.118; used_req_headers=accept-language,accept; peer_ip=r868133f79d0c3fa9d7c7ccca0151af2e; peer_country=us; render=false
FieldDescription
req_idThe internal ID of your request into our system. Providing this can be helpful in bug reports as it will help us see details on what happened in your specific request.
bytes_upThe amount of outgoing traffic our system recorded while processing this request
bytes_downThe amount of incoming traffic our system recorded while processing this request
billedWhether our system considers this a billable request or not
destination_ipThe IP address of the remote server used to fetch this data
used_req_headersWhich request headers (if any) were used in the final request. This mainly applies when using Custom Web Unlocker API
peer_ipA unique identifier for the IP address used to make the request. This is mainly useful to validate IP rotation is working as you expect
peer_countryThe country of the peer that was used for the request
renderShows whether the returned page is the result after a browser rendered the HTML or the response body of a single HTTP request

​
Common Error Codes

Occasionally, for a number of reasons, you might receive an unexpected error code in response to your Web Unlocker API request. The following list will provide you with a deeper understanding of what the source of the issue may be.
ErrorDescription
404Page not found. Invalid URL, which suggests the URL might be broken or dead.
403The URL you’re trying to access is valid, but you are forbidden from accessing that URL.
502Error code 502 is the most common error for Web Unlocker API users, the descriptive part is under the x-luminati-error-code.
407This error code suggests one of your account credentials is incorrect (password or zone’s name).
429This error code implies a rate limit (rare). In such cases, if the response appears as below, Bright Data is applying auto-throttling to the request, and you should open a ticket or email support@brightdata.com for assistance.
401 411 444Bad request, usually happens in API requests when headers or cookies missing.
503HTTP error code 503 means “Service Unavailable”. Browser check failed or browser check wasn’t completed
< HTTP/1.1 429 The request was auto-throttled due to low success rate  
< x-luminati-error-code: sr_rate_limit
< x-luminati-error: The request was auto-throttled due to low success rate
< x-brd-error-code: sr_rate_limit
< x-brd-error: The request was auto-throttled due to low success rate
< date: Tue, 23 Jan 2024 17:07:19 GMT
< connection: keep-alive
< keep-alive: timeout=5
< transfer-encoding: chunked
< 
* Connection #0 to host brd.superproxy.io left intact

​
Get Success Rate Statistics Per Domain

The following API endpoint will provide Web Unlocker API success rate statistics from the past 7 days. The statistics can be obtained per single domain like example.com or for a wildcard domain like example.* in order to get statistics for all top-level domains. Note: calling this API endpoint requires using your API key How to get statistics for a single domain? 
    curl "https://api.brightdata.com/unblocker/success_rate/example.com" -H "Content-Type: application/json" -H "Authorization: Bearer API_KEY"
How to get statistics for all monitored top level domains? 
    curl "https://api.brightdata.com/unblocker/success_rate/example.*" -H "Content-Type: application/json" -H "Authorization: Bearer API_KEY"