Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.brightdata.com/llms.txt

Use this file to discover all available pages before exploring further.

This tutorial walks you through sending a synchronous request to each Bright Data X Scraper API endpoint. By the end, you’ll have working examples for profiles and posts.

​
Prerequisites

​
Request structure

Every synchronous request follows the same pattern: 
POST https://api.brightdata.com/datasets/v3/scrape?dataset_id={DATASET_ID}&format=json
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json

[{"url": "https://x.com/..."}]
The only thing that changes between endpoints is the dataset_id and the input URL format.
Synchronous requests support up to 20 URLs and have a 1-minute timeout. If the request takes longer, the API automatically returns a snapshot_id instead. See async requests.

​
How to scrape X (Twitter) profiles

Dataset ID: gd_lwxmeb2u1cniijd7t4 
curl -X POST \
  "https://api.brightdata.com/datasets/v3/scrape?dataset_id=gd_lwxmeb2u1cniijd7t4&format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://x.com/elonmusk"}]'
You should see a 200 response. This takes 10-30 seconds. 
[
  {
    "user_name": "elonmusk",
    "name": "Elon Musk",
    "description": "Read @WallStreetSilv",
    "followers": 214000000,
    "following": 870,
    "number_of_tweets": 52000,
    "is_verified": true,
    "profile_image_link": "https://..."
  }
]
Full Profiles response schema

​
How to scrape X (Twitter) posts

Dataset ID: gd_lwxkxvnf1cynvib9co 
curl -X POST \
  "https://api.brightdata.com/datasets/v3/scrape?dataset_id=gd_lwxkxvnf1cynvib9co&format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://x.com/elonmusk/status/1234567890123456789"}]'

[
  {
    "url": "https://x.com/elonmusk/status/1234567890123456789",
    "user_posted": "elonmusk",
    "description": "Exciting times ahead...",
    "date_posted": "2024-04-03T14:30:00.000Z",
    "likes": 125000,
    "retweets": 18000,
    "replies": 5200,
    "hashtags": ["technology", "innovation"]
  }
]
Full Posts response schema

​
Quick reference: dataset IDs

EndpointDataset IDURL pattern
Profilesgd_lwxmeb2u1cniijd7t4x.com/{username}
Postsgd_lwxkxvnf1cynvib9cox.com/{username}/status/{id}

​
Output formats

Control the response format with the format query parameter:
ValueDescription
jsonJSON array (default)
ndjsonNewline-delimited JSON, one record per line
csvComma-separated values

​
Next steps

Async batch requests

Scrape hundreds of URLs in a single batch job.

API reference

Full parameter and response field reference.