Skip to main content
Instead of creating duplicate scrapers for each delivery type, you can choose a delivery type per job using API.
hero-image.png
Scraper Studio supports three API collection modes:
  • Batch collection: runs one or more inputs in the background and returns a collection_id.
  • Asynchronous real-time collection: runs one input in the background and returns a response_id.
  • Synchronous real-time collection: runs one input and returns the collected data directly in the same API response, if the request completes within the timeout.
The scraper’s Delivery preferences in the control panel affect which API modes can be used. A scraper configured for Batch delivery can still be triggered with the real-time API endpoints. However, a scraper configured for Real-time delivery cannot be triggered with the batch endpoint.
Delivery preferences tab in the Scraper Studio control panel, showing the "Choose when to get the data" options: "On a job completion (batch)" selected, "Realtime (single request)" and "Split delivery to small batches"

​
Delivery preference compatibility

Before triggering a scraper by API, check its Delivery preferences in the control panel. If you call the batch endpoint for a scraper configured as real-time, the API returns an error:

​
Choose a collection mode

​
Batch collection

Use batch collection when you want to run one or more inputs in the background. Batch collection is best for:
  • Multiple URLs, keywords or input rows
  • Large jobs
  • Scheduled or recurring data collection
  • Jobs where results can be retrieved later
Batch collection is available only when the scraper is not configured as Real-time in Delivery preferences. See Trigger batch collection for request parameters and examples.

​
Asynchronous real-time collection

Use asynchronous real-time collection when you want to run a single input in the background and retrieve the result later. Async real-time is best for:
  • One input per request
  • Low-latency workflows where the client should not keep the HTTP request open
  • Applications that prefer to trigger now and retrieve results later
A scraper configured for Batch delivery in the control panel can still be triggered with the async real-time endpoint. See Trigger async real-time collection for request parameters and examples.

​
Synchronous real-time collection

Use synchronous real-time collection when you want the API request to wait for the scraper result and return the collected data directly. Sync real-time is best for:
  • One input per request
  • Applications that need an immediate response
  • Scrapers that usually complete within the configured timeout
A scraper configured for Batch delivery in the control panel can still be triggered with the sync real-time endpoint. Synchronous real-time requests use the timeout query parameter. The timeout must be between 25s and 50s. If the scraper finishes within the timeout, the API returns 200 OK with the collected data. If the scraper is still running when the timeout is reached, the API may return 202 Accepted with a response_id. Use this response_id with the real-time data endpoint to retrieve the result asynchronously. See Trigger sync real-time collection for request parameters and examples.

​
Delivery behavior

Collection mode controls how the scraper runs. Delivery preferences control default delivery behavior configured in the control panel. Depending on your scraper configuration, results can be retrieved by API or delivered to a configured destination, such as:
  • API download
  • Webhook
  • Amazon S3
  • Google Cloud Storage
  • Azure Blob Storage
  • SFTP / FTP
  • Snowflake
  • Email
To override the configured destination for a single batch request, pass the deliver query parameter on POST /dca/trigger, and use the notify parameter to send a notification when the job or delivery completes. See request-level delivery and notifications. Important behavior:
  • A scraper configured for Batch delivery can be triggered by either batch or real-time API endpoints.
  • A scraper configured for Real-time delivery can be triggered only by real-time API endpoints.
  • Batch endpoint calls return a collection_id.
  • Async real-time endpoint calls return a response_id.
  • Sync real-time endpoint calls return collected data directly, unless the timeout is reached.
Batch responses begin with j_**** and real-time responses begin with d****