Overview

We are thrilled to introduce the Web scraper API, designed to simplify and enrich your data acquisition process. This new service allows for a more robust and streamlined way to collect data according to your specific needs. We offer real-time support for up to 20 URL inputs, and batch support for more than 20 inputs, regardless of the scraper type.

​

How To Collect? (High level)

​

Trigger a Collection (Demo)

1. Choose the target website from our variety of API offerings

2. Pick the specific scraper you need

3. Update the desired list of Inputs via JSON or CSV

4. Enable the “Include errors report with the results” toggle button

5. Enable the “Deliver results to external storage” toggle OR the “Send to webhook” toggle button according to your preference

6. Select whether to deliver the data to an external storage or to deliver it to a webhook using the desired option toggle button

​

Via Webhook:

1. Set your webhook URL and Authorization header if needed

2. Select your preferred file format (JSON, NDJSON, JSON lines, CSV)

3. Choose whether to send it compressed or not

4. Test webhook to validate that the operation runs successfully (using sample data)

5. Copy the code and run it.

​

Via Deliver to external storage:

1. Select your preferred delivery location (S3, Google Cloud, Snowflake, or any other available option)

2. Fill out the needed credentials according to your pick

3. Select your preferred file format (JSON, NDJSON, JSON lines, CSV)

4. Copy the code and run it.

​

Limit records

While running a discovery API, you can set a limit of the number of results per input

In the example below, we’ve set a limitation of 10 results per input

​

Management APIs

​

Get snapshot list

Check your snapshot history with this API. It returns a list of all available snapshots, including the snapshot ID, creation date, and status.

​

Monitor Progress

Check your data collection status with this API. It should return “collecting” while gathering data, “digesting” when processing, and “ready” when available.

​

System limitations

​

File size

Input	up to 1GB
Webhook delivery	up to 1GB
API Download	up to 5GB (for bigger files use API delivery)
Delivery API	unlimited