Build a daily Amazon price monitor

You track prices and stock on a list of Amazon SKUs and you need fresh numbers every morning before your team starts work. You don’t want to run a server just for this. You don’t want to babysit a cron job on a laptop. In this tutorial we’ll build exactly that pipeline. You’ll commit a SKU list to a GitHub repo, write a small Python script that triggers the Bright Data Amazon Scraper API against it, wrap the script in a GitHub Actions workflow that runs on a daily cron, and configure Bright Data to deliver the results directly to an S3 bucket. Each morning a fresh JSON file lands in S3, keyed by snapshot ID, ready for your BI pipeline to pick up. No servers, no webhook handlers, no glue code. Just a workflow file, a script and a delivery config.

What you’ll build

A GitHub repository containing:

A skus.json file listing the Amazon product URLs to monitor
A Python script that POSTs the SKU list to the Bright Data Amazon Scraper API
A GitHub Actions workflow that runs the script on a daily schedule
Bright Data configured to deliver each snapshot to your S3 bucket

By the end, you’ll see a new JSON file in S3 every day, each containing fresh price, rating and availability data for every SKU on the list. Estimated time: 30 minutes.

Prerequisites

A Bright Data account with an API key (get your key)
An S3 bucket with Bright Data delivery already configured. Follow Amazon to S3 delivery once, then come back. This tutorial assumes the delivery destination is already saved in your Amazon scraper’s settings.
A GitHub account and a new (empty) repository
Python 3.9+ installed locally
Git installed locally

Part 1: Create the SKU list

Clone your empty GitHub repo locally and create a skus.json file at the repo root:

skus.json

[
  "https://www.amazon.com/dp/B0D1XD1ZV3",
  "https://www.amazon.com/dp/B0863TXGM3",
  "https://www.amazon.com/dp/B09V3KXJPB"
]

These are three real product URLs (AirPods Pro 2, Sony WH-1000XM4 headphones and iPad Air M1). Swap them for your own SKUs later.

Keeping the SKU list in the repo means every edit is version-controlled and every change ships through normal pull-request review. Large SKU lists can live in a CSV loaded by the script instead, we’ll mention that in “Next steps.”

Part 2: Write the trigger script

Create trigger_scrape.py at the repo root:

trigger_scrape.py

import json
import os
import sys

import requests

DATASET_ID = "gd_l7q7dkf244hwjntr0"  # Amazon products by URL
TRIGGER_URL = "https://api.brightdata.com/datasets/v3/trigger"


def main() -> int:
    api_key = os.environ.get("BRIGHTDATA_API_KEY")
    if not api_key:
        print("BRIGHTDATA_API_KEY environment variable is not set.")
        return 1

    with open("skus.json") as f:
        urls = json.load(f)

    payload = [{"url": url} for url in urls]

    response = requests.post(
        TRIGGER_URL,
        params={"dataset_id": DATASET_ID, "format": "json"},
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        },
        json=payload,
        timeout=30,
    )
    response.raise_for_status()

    snapshot_id = response.json().get("snapshot_id")
    print(f"Triggered scrape for {len(urls)} SKUs. Snapshot: {snapshot_id}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Then create requirements.txt:

requests==2.32.3

Two things worth noticing:

The script does not wait for results. It fires the trigger and exits. Bright Data runs the scrape asynchronously and delivers the results directly to S3 via the delivery config you saved in your scraper settings. That’s the whole point: the script is a cheap, stateless trigger.
The API key comes from an environment variable. Never commit keys to a repo. We’ll wire this to GitHub Actions Secrets in Part 4.

Part 3: Run it locally

Install the dependency and run the script with your key:

pip install -r requirements.txt
export BRIGHTDATA_API_KEY=your_actual_key_here
python trigger_scrape.py

You should see output like: Example output. Your values will differ.

Triggered scrape for 3 SKUs. Snapshot: sd_mntfmunq1yy7gi201q

Wait 60 to 90 seconds, then check your S3 bucket:

aws s3 ls s3://your-bucket-name/amazon/products/

You should see a new file named with the snapshot ID: Example output. Your values will differ.

2026-04-10 09:14:22   14382 sd_mntfmunq1yy7gi201q.json

Download it and inspect one record:

aws s3 cp s3://your-bucket-name/amazon/products/sd_mntfmunq1yy7gi201q.json ./latest.json
python -m json.tool latest.json | head -30

You should see structured product data for each SKU: Example output. Your values will differ.

[
  {
    "title": "Sony WH-1000XM4 Wireless Premium Noise Canceling Overhead Headphones",
    "asin": "B0863TXGM3",
    "brand": "Sony",
    "final_price": 209.99,
    "currency": "USD",
    "rating": 4.6,
    "reviews_count": 62492,
    "availability": "Only 1 left in stock - order soon.",
    "url": "https://www.amazon.com/dp/B0863TXGM3"
  }
]

The price field is final_price, and it can be null for products that are out of stock or currency-ambiguous listings. Your BI pipeline should handle that case explicitly rather than crashing on a missing key.

Notice that the file is keyed by snapshot_id, not by date. That’s deliberate: each snapshot is immutable, and you can walk the bucket chronologically by listing creation timestamps or by enabling versioning. We’ll discuss naming conventions in “Next steps.”

Part 4: Schedule the workflow on GitHub Actions

Now let’s move the trigger off your laptop and onto a daily schedule. Create .github/workflows/daily-scrape.yml:

.github/workflows/daily-scrape.yml

name: Daily Amazon price scrape

on:
  schedule:
    - cron: "0 6 * * *"   # 06:00 UTC every day
  workflow_dispatch:        # Allows manual runs from the Actions tab

jobs:
  trigger:
    runs-on: ubuntu-latest
    steps:
      - name: Check out the repo
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Trigger Bright Data scrape
        env:
          BRIGHTDATA_API_KEY: ${{ secrets.BRIGHTDATA_API_KEY }}
        run: python trigger_scrape.py

Two key elements:

schedule: cron: "0 6 * * *" runs the job at 06:00 UTC every day. Adjust the cron expression for your timezone. GitHub does not guarantee the start time of a scheduled workflow. GitHub’s events that trigger workflows reference says a scheduled run “can be delayed during periods of high loads,” and high load includes the start of every hour.
workflow_dispatch adds a Run workflow button in the Actions tab so you can kick off the job on demand without waiting for the schedule.

Now add your Bright Data key as a repo secret:

In your GitHub repo, go to Settings > Secrets and variables > Actions
Click New repository secret
Name it BRIGHTDATA_API_KEY and paste your key
Click Add secret

Treat your Bright Data API key like a password. Never commit it to the repo, never paste it into workflow logs, and rotate it if you suspect it’s been exposed.

Part 5: Push and verify

Commit everything and push:

git add skus.json trigger_scrape.py requirements.txt .github/workflows/daily-scrape.yml
git commit -m "Add daily Amazon price monitor"
git push

Open your repo on GitHub and go to the Actions tab. You should see the Daily Amazon price scrape workflow listed. Click Run workflow > Run workflow to kick it off manually. Within a few seconds a new run appears. Click into it and watch the steps execute. The final step should log: Example output. Your values will differ.

Triggered scrape for 3 SKUs. Snapshot: sd_mntfn4abcdefghij

Wait a minute, then check S3 again:

aws s3 ls s3://your-bucket-name/amazon/products/

You should now see two snapshot files, the one from Part 3 and the one from the workflow run. From here, GitHub Actions will run the workflow every day at the scheduled time, and a fresh file will appear in S3 each morning. No server, no cron job, no babysitting.

Troubleshooting

Why is there no file in S3 after 2 minutes?

Check the job status before you check the bucket. Send the snapshot_id that trigger_scrape.py printed in Part 3 to Monitor progress (GET /datasets/v3/progress/{snapshot_id}) and read the status field:

starting or running: the Amazon Scraper API is still collecting. Wait and check again.
ready: the scrape finished, so the problem is in delivery. Check the S3 destination saved in the Amazon scraper’s delivery settings and the bucket permissions. The IAM role must allow s3:PutObject on the bucket. See Files not appearing in S3? in the delivery guide.

Why does the trigger script fail with a 401?

The Bright Data API rejected the API key. A 401 with Invalid credentials means the key value is wrong. Locally, check the value you exported as BRIGHTDATA_API_KEY. In GitHub Actions, check the repository secret under Settings > Secrets and variables > Actions. The secret name must be exactly BRIGHTDATA_API_KEY. When the secret is missing, the script prints BRIGHTDATA_API_KEY environment variable is not set. instead of a 401.

Why is `final_price` null?

The Amazon Scraper API returns final_price as null for products that are out of stock and for currency-ambiguous listings. Check the availability field in the same record. Handle null in your BI pipeline instead of treating it as an error.

Why didn’t the scheduled workflow run?

GitHub runs scheduled workflows in UTC by default, so 0 6 * * * fires at 06:00 UTC, not 06:00 in your timezone. The GitHub Actions schedule event reference also lists these limits:

A scheduled run can be delayed during high load, which includes the start of every hour. If the load is high enough, GitHub may drop queued jobs. GitHub suggests scheduling at a different minute of the hour, for example 17 6 * * *
Scheduled workflows run only from the workflow file on the default branch
In a public repository, GitHub disables scheduled workflows after 60 days with no repository activity

To run the scrape without waiting for the schedule, open the Actions tab, select Daily Amazon price scrape and click Run workflow. For every Scraper API error message, see Scraper API error codes. For job, payload and download limits, see Scraper API limits.

Congratulations

You’ve built a fully automated daily price monitor:

A SKU list version-controlled in GitHub
A Python trigger script that fires the Bright Data scrape and exits
A GitHub Actions workflow that runs on a daily cron and authenticates via a repo secret
Bright Data S3 delivery that drops each snapshot into your bucket asynchronously

Every moving part is declarative and in the repo. Editing the SKU list, the cron schedule or the destination is a one-line pull request.

Next steps

Stream large snapshots

Use stream_max_lines to start receiving batches as soon as the first records are ready.

Amazon async reference

Full parameter list for the async trigger endpoint, including include_errors and limit_per_input.

Monitor delivery status

Programmatically check snapshot status and delivery result from inside the workflow.

Deliver results

Swap S3 for GCS, Azure, Snowflake or SFTP with the same trigger call.

Get started

Data extraction

Web search

AI training data

Proxy networks

Build a daily Amazon price monitor

What you’ll build

Prerequisites

Part 1: Create the SKU list

Part 2: Write the trigger script

Part 3: Run it locally

Part 4: Schedule the workflow on GitHub Actions

Part 5: Push and verify

Troubleshooting

Why is there no file in S3 after 2 minutes?

Why does the trigger script fail with a 401?

Why is `final_price` null?

Why didn’t the scheduled workflow run?

Congratulations

Next steps

Stream large snapshots

Amazon async reference

Monitor delivery status

Deliver results

​What you’ll build

​Prerequisites

​Part 1: Create the SKU list

​Part 2: Write the trigger script

​Part 3: Run it locally

​Part 4: Schedule the workflow on GitHub Actions

​Part 5: Push and verify

​Troubleshooting

​Why is there no file in S3 after 2 minutes?

​Why does the trigger script fail with a 401?

​Why is final_price null?

​Why didn’t the scheduled workflow run?

​Congratulations

​Next steps

Stream large snapshots

Amazon async reference

Monitor delivery status

Deliver results

What you’ll build

Prerequisites

Part 1: Create the SKU list

Part 2: Write the trigger script

Part 3: Run it locally

Part 4: Schedule the workflow on GitHub Actions

Part 5: Push and verify

Troubleshooting

Why is there no file in S3 after 2 minutes?

Why does the trigger script fail with a 401?

Why is `final_price` null?

Why didn’t the scheduled workflow run?

Congratulations

Next steps