此更新提供了一种更细粒度且更简化的方式来请求和管理数据采集，从而根据您的特定需求更有效地生成数据集。
何时使用各 API:
无客户定义视图的初始采集：
三个主要 API 端点在数据采集工作流中具有不同用途，帮助您以结构化和高效的方式获取定制数据集。
请求采集：
端点：POST https://api.brightdata.com/datasets/request_collection
参数：
discover_new 或 url_collection
curl "https://api.brightdata.com/datasets/request_collection?dataset_id=gd_l1viktl72bvl7bjuj0&type=discover_new" -H "Authorization: Bearer API_KEY" -H "Content-Type: application/json" -k -d '[{"id":"user-id"}]'
discover_new 时，查找所有链接 (PDPs) 可能需要一些时间。
检查采集状态：
端点：GET https://api.brightdata.com/datasets/request_collection
参数：
设置数据的新鲜度。如果数据在此时间范围内（例如请求 1 周前，采集于 5 天前），则不会进行新的抓取。如果数据不新鲜，则立即抓取。
- 1 周：604,800,000 毫秒
- 1 月：2,592,000,000 毫秒
curl -k "https://api.brightdata.com/datasets/request_collection?request_id=REQUEST_ID&freshness_ms=2592000000" -H "Authorization: Bearer API_KEY"
{
"dataset_id": request_job.dataset_id,
"total_lines": 100,
"fresh_count": 30,
"name": "linkedin_companies custom input",
"status": "done",
"request_id": "XXXX"
}
{
"total_lines": 100,
"status": "running"
}
{
"request_id": "xxxx",
"error": "Validation failed",
"error_code": "validation",
"validation_errors": [
{
"line": "{\"URL\":\"https://www.tiktok.com/search?q=tjd\"}",
"index": 1,
"errors": [
["url", "Required field"]
]
}
]
}
启动采集：
端点：POST https://api.brightdata.com/datasets/initiate_collection
参数：
curl -X POST -k "https://api.brightdata.com/datasets/initiate_collection" -d '{"request_id":"j_ln2x567b2961de0d1x","freshness_ms":2592000000}' -H "Authorization: Bearer API_KEY" -H "content-type: application/json"
定义视图后的采集：
启动采集：
端点：POST https://api.brightdata.com/datasets/initiate
参数：
discover_new 或 url_collection
curl "https://api.brightdata.com/datasets/initiate?dataset_id=XXX_DATASET_ID&type=url_collection&view=XXX_VIEW_ID" -H "Authorization: Bearer API_KEY" -H "Content-Type: application/json" -k -d '[{"id":"user-id"}]'
获取已采集快照结果
curl "https://api.brightdata.com/datasets/snapshots/snapshot_id/download" -H "Authorization: Bearer API_KEY"
