Web Scraper IDE API
Trigger a scraper for batch collection method
API Endpoint: POST /dca/trigger?collector={ID_COLLECTOR}
POST Body
raw JSON -d '[{"url":"https://targetwebsite.com/product_id/"}]'
Query Parameters
A unique identification of scraper
Set to dev to trigger the development version of the scraper
human_name - A human readable name for the batch
Choose trigger_behavior parameters:
If there’s already something in the crawl queue, push this job into the queue
Send another batch of requests that will start after the last one is finished
Cancel running job and run instead, submit the request and cancel running one
Disable media file download
Set job time deadline, job will be terminated after specified time
- h : hours
- m : minutes
- s : seconds
Headers
Authorization Bearer Key -H "Authorization: Bearer API_KEY"
JSON/CSV/Multipart -H "Content-Type: application/json"