To set your delivery preferences for the dataset, simply click on the ‘Delivery settings’ tab:

You can also set delivery preferences per snapshot, to do that simply click on the snapshot and then click ‘Export’, set required storage parameters:

There is an option to download the dataset directly from the CP by clicking ‘download + format’ button:

​
AWS S3 User Role Permissions

To control access to S3 resources, you can use IAM (Identity and Access Management) to create and manage AWS users and their permissions. One way to do this is by creating IAM roles and attaching them to S3 resources.

1

Create a Policy

Go to the “Policies” section in the IAM console

Create a new policy that defines the permissions for the S3 resources you want to grant access to.

Example of AWS policy:

{
"Version": "2012-10-17",
"Statement": [
    {
    "Effect": "Allow",
    "Action": [
        "s3:PutObject",
        "s3:GetObject"
    ],
    "Resource": "*"
    }
]
}
2

Create a Role

Go to the “Roles” section in the IAM console

Create a new role and specify the policy created in step 1 in the “Permission policies” section.

Make a note of the ARN of the role, which will be used for delivery credentials. (The ARN will look like arn:aws:iam::<ROLE_ID>:role/<ROLE_NAME>)

Example of User Role:

{
"Version": "2012-10-17",
"Statement": [
    {
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::422310177405:role/brd.ec2.zs-dca-delivery"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
        "StringEquals": {
        "sts:ExternalId": "YOUR EXTERNAL ID"
        }
    }
    }
]
}
3

Use the ARN of the role

In the S3 resources that you want to grant access to, attach the role created in step 2 by using the ARN.

​
How to find your Google Cloud Private Key

  1. Go to the Google Cloud Platform Console home page - https://console.cloud.google.com/

  2. Expand the menu by Google Cloud Platform, and click IAM & Admin.

  3. Click Service accounts.

  1. Choose an existing service account from the list or create one.

If the button is not visible, create a project first in order to Create Service Account.

  1. Create the service account by entering the name, ID, and description at the Create Service Account process. Then grant the access and create the account.

  1. Click on Email of the service account.

  1. To access the keys, click on the ‘KEYS’ tab. Click the “Add Key” dropdown and then select ‘Create New Key’.

  1. Choose JSON as the key type.

  1. The service account key JSON file is automatically downloaded to your local machine.

  2. Copy private_key from the downloaded JSON file & Paste it to Data Collector delivery settings.

​
Snowflake Delivery Configuration Guide only visible to agents and admins

Snowflake configuration is available for ‘Datasets’ delivery and not available for data collectors (Web Scraper)

Getting Started

In order to allow efficient delivery of Datasets to your Snowflake environment, we provide a step-by-step guide to set it up. Just follow these steps:

1

Select or Create a Database

Firstly, decide if you will use an existing database or create a new one. If you opt for a new database, here’s the command you need:

CREATE DATABASE <database>;

Remember to replace <database> with the name you want for your database.

2

Select or Create a Schema

Decide if you will use an existing schema or create a new one. By default every database has a PUBLIC schema. If you wish to use a different schema, here’s the command you need:

CREATE SCHEMA <schema>;

Replace <schema> with your own schema name.

3

Select or Create a Warehouse

Choose an existing warehouse or create a new one. When creating a new warehouse, consider Snowflake’s recommendations for configuring a warehouse specifically for data loading. Use the following command to create a warehouse:

CREATE WAREHOUSE <warehouse>;

Replace <warehouse> with your desired warehouse name.

4

Select or Create an Internal Named Stage

Next, choose an existing internal named stage or create a new one. To create a new stage, use this command:

CREATE STAGE <stage>;

Don’t forget to replace <stage> with your preferred stage name.

5

Create a Role

You’ll need a role that can write to your chosen stage. To create one, use:

CREATE ROLE <role_name>;

Change <role_name> to your chosen role name.

6

Grant Warehouse Rights to the Role

Now, grant your new role the necessary rights to operate on your chosen warehouse using:

GRANT OPERATE ON WAREHOUSE <warehouse> TO ROLE <role_name>;

Remember to replace <warehouse> and <role_name> with your specific warehouse and role name respectively.

7

Enable Write Operations on the Stage for the Role

To enable your role to write on the stage, use the command:

GRANT WRITE ON STAGE <stage> TO ROLE <role_name>;

Again, replace <stage> and <role_name> with your chosen stage and role names.

8

Create a BrightData User

Next, create a new user for BrightData that will be used to upload data directly into Snowflake. The command is as follows:

create user <user_name>
PASSWORD = '<password>'
LOGIN_NAME = <login>
MUST_CHANGE_PASSWORD = FALSE
DISABLED = FALSE  
COMMENT = 'user for BrightData to upload data directly into Snowflake'

Replace <user_name>, <password>, and <login> with your chosen username, password, and login name.

9

Grant Role Privileges to the New User

Finally, grant your new user the privileges of the role you created:

GRANT ROLE <role_name> TO USER <user_name>;

Replace <role_name> and <user_name> with your role and user names.

10

Whitelist IPs

If you have an active Network Policy applied in your Snowflake account you need to whiltelist following IPs:

ALTER NETWORK POLICY <policy_name>  
SET ALLOWED_IP_LIST=(  
<existing_whitelisted_ips>,  
'35.169.71.210',  
'34.233.211.38',  
'44.194.183.74',  
'54.243.177.151');

Replace <policy_name> with your network policy name. Replace <existing_whiltelisted_ips> with the list of existing whitelisted IPs.

And that’s it! You have now configured your Snowflake environment to receive data from our platform.

If you have any issues or need further assistance, please contact our support team.

If you want to learn more about Data Loading Performance and Warehouse Size Considerations click here.

Was this page helpful?

RequestCustom Dataset API