robots.txt are blocked until KYC is approved, and some targets additionally require special permission under Bright Data’s compliance policy.
What can I use without KYC?
Every Web Unlocker API capability below is available in immediate access mode, which is the mode every account starts in before KYC approval.What requires KYC?
Why does a request fail with ub_bad_endpoint_robots?
Without KYC your account is in immediate access mode, which honors the target’srobots.txt. Requesting a disallowed endpoint returns ub_bad_endpoint_robots with HTTP status 400:
ub_bad_endpoint_robots response
Read the target robots.txt
reddit.com/robots.txt is User-agent: * followed by Disallow: /, so no Reddit endpoint is available in immediate access mode.
Which targets require special permission?
Bright Data’s compliance policy restricts three groups of targets no matter what the target’srobots.txt allows:
- Government websites. Blocked by default and returns
policy_20051. The match is not limited to.govor.gov.[country_code]suffixes, so a government site on any domain still qualifies. - Domains in a blocked classification. Bright Data classifies domains and blocks some classifications, and returns
policy_20000. The response names the classification that matched. - Domains outside your approved use case. After KYC, access is scoped to the use case the Bright Data compliance team approved. Requests outside that use case return an
Access deniederror naming the classified category.
x-brd-err-code, and KYC is the route to request access. For more on government targets, see How do I target government websites? in the proxy FAQs.
Which error codes relate to access?
The Web Unlocker API error codes reference and the proxy errors catalog are the source of truth for every code below.
The status appears in a different place in each access mode. On the Direct API the outer response is
200 and the result status is in x-brd-status-code. On the native proxy the status is the response’s own HTTP status, with the message in the status reason phrase. The x-brd-error-code value is the same in both modes. See Web Unlocker API error codes for the full comparison.
How do I complete KYC?
- Add funds to your account balance. KYC cannot be started in Playground or Limited Trial mode.
- Add a user with a company email domain. KYC applications are accepted only from registered businesses, so an account on a personal email domain cannot be approved.
- Submit the KYC form. Describe your business and your use case, and attach a company registration document. Access is granted against the use case you describe, so describe it fully. Start KYC verification
- Wait for review. A person reviews every submission, so approval is never instant. You are updated within 48 hours. Check the status in your control panel under Settings > Profile.
FAQ
Is KYC the same as account verification?
No. Account verification lifts Limited Trial rate and balance limits. KYC is a separate, human-reviewed compliance check. An HTTP 429 rate limit error points to account verification, whileub_bad_endpoint_robots points to KYC.
Does KYC give unlimited access?
No. KYC is reviewed against the use case you submit, and access is granted for that use case. Targets that require special permission are assessed by the Bright Data compliance team case by case, and the Acceptable Use Policy continues to apply.Can I use the Web Unlocker API while my KYC is under review?
Yes. The Web Unlocker API keeps working in immediate access mode while the Bright Data compliance team reviews your KYC submission. Only the requests listed in this page’s capability table stay blocked until approval.Related
- Web Unlocker API error codes: Every Web Unlocker API error code, grouped by HTTP status.
- Browser API and KYC: The same access rules for the Browser API.
- Residential network access policy: Why KYC exists and what the Bright Data compliance team reviews.
- Proxy errors catalog: Full text for every
policy_code.