> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brightdata.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Build a daily Amazon price monitor
> Schedule a daily Amazon SKU scrape with the Bright Data Amazon Scraper API, delivered to S3 and orchestrated by a GitHub Actions cron workflow.
You track prices and stock on a list of Amazon SKUs and you need fresh numbers every morning before your team starts work. You don't want to run a server just for this. You don't want to babysit a cron job on a laptop.
In this tutorial we'll build exactly that pipeline. You'll commit a SKU list to a GitHub repo, write a small Python script that triggers the Bright Data Amazon Scraper API against it, wrap the script in a GitHub Actions workflow that runs on a daily cron, and configure Bright Data to deliver the results directly to an S3 bucket. Each morning a fresh JSON file lands in S3, keyed by snapshot ID, ready for your BI pipeline to pick up.
No servers, no webhook handlers, no glue code. Just a workflow file, a script and a delivery config.
## What you'll build
A GitHub repository containing:
1. A `skus.json` file listing the Amazon product URLs to monitor
2. A Python script that POSTs the SKU list to the Bright Data Amazon Scraper API
3. A GitHub Actions workflow that runs the script on a daily schedule
4. Bright Data configured to deliver each snapshot to your S3 bucket
By the end, you'll see a new JSON file in S3 every day, each containing fresh price, rating and availability data for every SKU on the list.
**Estimated time:** 30 minutes.
## Prerequisites
* A [Bright Data account](https://brightdata.com/?hs_signup=1\&utm_source=docs) with an API key ([get your key](https://brightdata.com/cp/setting/users))
* An S3 bucket with Bright Data delivery already configured. Follow [Amazon to S3 delivery](/products/scrapers/scrapers-library/data-delivery) once, then come back. This tutorial assumes the delivery destination is already saved in your Amazon scraper's settings.
* A [GitHub account](https://github.com/signup) and a new (empty) repository
* Python 3.9+ installed locally
* Git installed locally
## Part 1: Create the SKU list
Clone your empty GitHub repo locally and create a `skus.json` file at the repo root:
```json skus.json theme={null}
[
"https://www.amazon.com/dp/B0D1XD1ZV3",
"https://www.amazon.com/dp/B0863TXGM3",
"https://www.amazon.com/dp/B09V3KXJPB"
]
```
These are three real product URLs (AirPods Pro 2, Sony WH-1000XM4 headphones and iPad Air M1). Swap them for your own SKUs later.
Keeping the SKU list in the repo means every edit is version-controlled and every change ships through normal pull-request review. Large SKU lists can live in a CSV loaded by the script instead, we'll mention that in "Next steps."
## Part 2: Write the trigger script
Create `trigger_scrape.py` at the repo root:
```python trigger_scrape.py theme={null}
import json
import os
import sys
import requests
DATASET_ID = "gd_l7q7dkf244hwjntr0" # Amazon products by URL
TRIGGER_URL = "https://api.brightdata.com/datasets/v3/trigger"
def main() -> int:
api_key = os.environ.get("BRIGHTDATA_API_KEY")
if not api_key:
print("BRIGHTDATA_API_KEY environment variable is not set.")
return 1
with open("skus.json") as f:
urls = json.load(f)
payload = [{"url": url} for url in urls]
response = requests.post(
TRIGGER_URL,
params={"dataset_id": DATASET_ID, "format": "json"},
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
json=payload,
timeout=30,
)
response.raise_for_status()
snapshot_id = response.json().get("snapshot_id")
print(f"Triggered scrape for {len(urls)} SKUs. Snapshot: {snapshot_id}")
return 0
if __name__ == "__main__":
sys.exit(main())
```
Then create `requirements.txt`:
```text theme={null}
requests==2.32.3
```
Two things worth noticing:
* **The script does not wait for results.** It fires the trigger and exits. Bright Data runs the scrape asynchronously and delivers the results directly to S3 via the delivery config you saved in your scraper settings. That's the whole point: the script is a cheap, stateless trigger.
* **The API key comes from an environment variable.** Never commit keys to a repo. We'll wire this to GitHub Actions Secrets in Part 4.
## Part 3: Run it locally
Install the dependency and run the script with your key:
```bash theme={null}
pip install -r requirements.txt
export BRIGHTDATA_API_KEY=your_actual_key_here
python trigger_scrape.py
```
You should see output like:
Example output. Your values will differ.
```text theme={null}
Triggered scrape for 3 SKUs. Snapshot: sd_mntfmunq1yy7gi201q
```
Wait 60 to 90 seconds, then check your S3 bucket:
```bash theme={null}
aws s3 ls s3://your-bucket-name/amazon/products/
```
You should see a new file named with the snapshot ID:
Example output. Your values will differ.
```text theme={null}
2026-04-10 09:14:22 14382 sd_mntfmunq1yy7gi201q.json
```
Download it and inspect one record:
```bash theme={null}
aws s3 cp s3://your-bucket-name/amazon/products/sd_mntfmunq1yy7gi201q.json ./latest.json
python -m json.tool latest.json | head -30
```
You should see structured product data for each SKU:
Example output. Your values will differ.
```json theme={null}
[
{
"title": "Sony WH-1000XM4 Wireless Premium Noise Canceling Overhead Headphones",
"asin": "B0863TXGM3",
"brand": "Sony",
"final_price": 209.99,
"currency": "USD",
"rating": 4.6,
"reviews_count": 62492,
"availability": "Only 1 left in stock - order soon.",
"url": "https://www.amazon.com/dp/B0863TXGM3"
}
]
```
The price field is `final_price`, and it can be `null` for products that are out of stock or currency-ambiguous listings. Your BI pipeline should handle that case explicitly rather than crashing on a missing key.
Notice that the file is keyed by `snapshot_id`, not by date. That's deliberate: each snapshot is immutable, and you can walk the bucket chronologically by listing creation timestamps or by enabling versioning. We'll discuss naming conventions in "Next steps."
## Part 4: Schedule the workflow on GitHub Actions
Now let's move the trigger off your laptop and onto a daily schedule.
Create `.github/workflows/daily-scrape.yml`:
```yaml .github/workflows/daily-scrape.yml theme={null}
name: Daily Amazon price scrape
on:
schedule:
- cron: "0 6 * * *" # 06:00 UTC every day
workflow_dispatch: # Allows manual runs from the Actions tab
jobs:
trigger:
runs-on: ubuntu-latest
steps:
- name: Check out the repo
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install dependencies
run: pip install -r requirements.txt
- name: Trigger Bright Data scrape
env:
BRIGHTDATA_API_KEY: ${{ secrets.BRIGHTDATA_API_KEY }}
run: python trigger_scrape.py
```
Two key elements:
* **`schedule: cron: "0 6 * * *"`** runs the job at 06:00 UTC every day. Adjust the cron expression for your timezone. GitHub does not guarantee the start time of a scheduled workflow. GitHub's [events that trigger workflows](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#schedule) reference says a scheduled run "can be delayed during periods of high loads," and high load includes the start of every hour.
* **`workflow_dispatch`** adds a **Run workflow** button in the Actions tab so you can kick off the job on demand without waiting for the schedule.
Now add your Bright Data key as a repo secret:
1. In your GitHub repo, go to **Settings** > **Secrets and variables** > **Actions**
2. Click **New repository secret**
3. Name it `BRIGHTDATA_API_KEY` and paste your key
4. Click **Add secret**
Treat your Bright Data API key like a password. Never commit it to the repo, never paste it into workflow logs, and rotate it if you suspect it's been exposed.
## Part 5: Push and verify
Commit everything and push:
```bash theme={null}
git add skus.json trigger_scrape.py requirements.txt .github/workflows/daily-scrape.yml
git commit -m "Add daily Amazon price monitor"
git push
```
Open your repo on GitHub and go to the **Actions** tab. You should see the **Daily Amazon price scrape** workflow listed.
Click **Run workflow** > **Run workflow** to kick it off manually. Within a few seconds a new run appears. Click into it and watch the steps execute. The final step should log:
Example output. Your values will differ.
```text theme={null}
Triggered scrape for 3 SKUs. Snapshot: sd_mntfn4abcdefghij
```
Wait a minute, then check S3 again:
```bash theme={null}
aws s3 ls s3://your-bucket-name/amazon/products/
```
You should now see two snapshot files, the one from Part 3 and the one from the workflow run.
From here, GitHub Actions will run the workflow every day at the scheduled time, and a fresh file will appear in S3 each morning. No server, no cron job, no babysitting.
## Troubleshooting
### Why is there no file in S3 after 2 minutes?
Check the job status before you check the bucket. Send the `snapshot_id` that `trigger_scrape.py` printed in Part 3 to [Monitor progress](/api-reference/scrapers/management-apis/monitor-progress) (`GET /datasets/v3/progress/{snapshot_id}`) and read the `status` field:
* `starting` or `running`: the Amazon Scraper API is still collecting. Wait and check again.
* `ready`: the scrape finished, so the problem is in delivery. Check the S3 destination saved in the Amazon scraper's delivery settings and the bucket permissions. The IAM role must allow `s3:PutObject` on the bucket. See [Files not appearing in S3?](/products/scrapers/scrapers-library/data-delivery#files-not-appearing-in-s3) in the delivery guide.
### Why does the trigger script fail with a 401?
The Bright Data API rejected the API key. A 401 with `Invalid credentials` means the key value is wrong. Locally, check the value you exported as `BRIGHTDATA_API_KEY`. In GitHub Actions, check the repository secret under **Settings** > **Secrets and variables** > **Actions**. The secret name must be exactly `BRIGHTDATA_API_KEY`. When the secret is missing, the script prints `BRIGHTDATA_API_KEY environment variable is not set.` instead of a 401.
### Why is `final_price` null?
The Amazon Scraper API returns `final_price` as `null` for products that are out of stock and for currency-ambiguous listings. Check the `availability` field in the same record. Handle `null` in your BI pipeline instead of treating it as an error.
### Why didn't the scheduled workflow run?
GitHub runs scheduled workflows in UTC by default, so `0 6 * * *` fires at 06:00 UTC, not 06:00 in your timezone. The [GitHub Actions `schedule` event reference](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#schedule) also lists these limits:
* A scheduled run can be delayed during high load, which includes the start of every hour. If the load is high enough, GitHub may drop queued jobs. GitHub suggests scheduling at a different minute of the hour, for example `17 6 * * *`
* Scheduled workflows run only from the workflow file on the default branch
* In a public repository, GitHub disables scheduled workflows after 60 days with no repository activity
To run the scrape without waiting for the schedule, open the **Actions** tab, select **Daily Amazon price scrape** and click **Run workflow**.
For every Scraper API error message, see [Scraper API error codes](/products/scrapers/error-codes). For job, payload and download limits, see [Scraper API limits](/products/scrapers/limits).
## Congratulations
You've built a fully automated daily price monitor:
* A **SKU list** version-controlled in GitHub
* A **Python trigger script** that fires the Bright Data scrape and exits
* A **GitHub Actions workflow** that runs on a daily cron and authenticates via a repo secret
* **Bright Data S3 delivery** that drops each snapshot into your bucket asynchronously
Every moving part is declarative and in the repo. Editing the SKU list, the cron schedule or the destination is a one-line pull request.
## Next steps
Use `stream_max_lines` to start receiving batches as soon as the first records are ready.
Full parameter list for the async trigger endpoint, including `include_errors` and `limit_per_input`.
Programmatically check snapshot status and delivery result from inside the workflow.
Swap S3 for GCS, Azure, Snowflake or SFTP with the same trigger call.