Presets configuration

Long single session (IP)

Use this preset if you need full page loads. Connect from the browser manually (for example Chrome/Firefox) or programatically (for example Puppeteer/Selenium). All requests share the same IP. You can control when you refresh the IP from the UI or API.

Rotating

Use this preset if you want to get a fresh new IP on each single request. This preset also rotates the User-Agent header automatically. It’s the best for scraping API when you don’t load the full pages.

Custom

Build your own preset fit your needs
Note that using a custom preset requires testing in the internal environment before going to production to verify the process is working as expected

Rules and Headers configuration

Navigate to Rules and start building your own rules based on triggers such as:

  • URL suffixes
  • Status codes
  • String values found in the Response body
  • Requests latency

You can choose an action to be taken to each trigger respectively. 

For example, an implementation of a rule to save bandwidth by nullifying media URLs’ outputs.

In addition, you can implement the desired headers in advance under the Rules section. 

For example, adding the user-agent header of a Linux desktop on i686 CPU (More about user-agents can be found here)

Port targeting configuration

Bright Data proxy manager enables targeting the Proxy using IP:PORT format. For example, targeting a port indexed as 24000 when the proxy manager is installed locally:

curl --proxy 127.0.0.1:24000 "<TARGET SITE>"

(If installed remotely simply switch 127.0.0.1 with the remote server IP address)

Instead of the IP:PORT USERNAME:PASSWORD 

curl --proxy brd.superproxy.io:22225 --proxy-user brd-customer-<USERNAME>-zone-<ZONE>:<PASSWORD> "<TARGET SITE>"

This allows:

  • Removing the username:password section from the request.
  • Integration with 3rd party softwares that accept only IP:Port format
  • Clean code. Ports can be configured with all necessary headers and rules in advance no need to adjust the command/request itself. More information is available here

Add other proxy providers to Proxy Manager

Bright Data Proxy Manager supports external proxies from other vendors. Connecting external proxies will allow you to optimize and manage all of your proxies in one place.

  • Login to your Proxy Manager
  • Set a new Port.
  • Select ‘External’
  • Add your proxies in the next format
['<proxy\_peer\_IP>', '<username>:<password>@<proxyprovider\_server>:<port>']
  • Click ‘Save’ and you will have the external proxies available as one of the Proxy Manager’s port

Difference in the calculation of traffic in Proxy Manager panel and Control Panel

Proxy Manager is a middleman between the request initiator and our Super Proxy servers. Proxy Manager statistics can be seen here, and Super Proxy statistics can be seen here. So each request that you send to Proxy Manager eventually reaches Super Proxy. But there could be a difference in traffic calculations, here’s why:

  • Proxy Manager calculates incoming requests as they are being sent, but after they reach the Proxy Manager, it attaches additional headers, so that request’s response will have more information about the request’s flow (timeline, proxy IP, etc)
  • Requests that reach the Super Proxy server from Proxy Manager are incoming with those additional parameters; that’s why Super Proxy calculates slightly bigger requests than it initially was when reached Proxy Manager
  • Proxy Manager sometimes adds headers to present a better view of logs, but this data is not being calculated in the billing

In conclusion, the main source of traffic statistics should be Zones’ page since it represents statistics formulated by Super Proxy.

 

As a rule - Bright Data uses the Zones’ page for calculation and one source of truth 

Invoices and billing events will be triggered based on the zones calculations