Release Notes

​

Release notes for week ending Mar 20, 2025

​

Datasets

​

🆕 New features

Large Dataset Splitting and Download in UI: You can now download large datasets directly from the UI. Large files are automatically split into smaller parts with easy download links through the email, making big data access smoother and more convenient.

​

🚀 Improvements

• Improved Delivery System: We made backend improvements to ensure faster and more reliable dataset deliveries.
• Filter API Tweaks: We fine-tuned the Filter API for better performance and more accurate filtering.
• Support for Free Datasets: We added billing-side support for free datasets, making it easier to access datasets that are available at no cost.

​

Error Status Documentation Added

We updated our documentation to include error status codes for the Dataset API, helping developers troubleshoot issues faster.

​

👉 View Documentation

​

Download Large Snapshots via Email

You can now receive an email with download links for snapshots larger than 5GB, offering another easy way to access big datasets.

​

Snapshot Metadata API Update

The Snapshot Metadata API now returns error details, error codes, and initiation type, giving you better visibility into your dataset operations.

​

🐞 Bug Fixes

• Fixed “Custom Dataset” Button: The “Custom Dataset” button now works properly.
• Fixed Parquet Delivery “Out of Memory” Error: Large Parquet file deliveries no longer cause memory errors.
• Fixed Filter Issues
• Missing “list exact match” and “include” options in filters were fixed.
• Filtering issues for the Enriched Employee Business dataset were resolved.
• Unclear filter error messages in the UI are now clearer and more helpful.
• Fixed Snapshots Table UI: The snapshots table now displays correctly without empty columns.

​

Release notes for week ending Mar 20, 2025

​

General:

​

🆕 New features

• AI-Powered Support
  • BrightAI now serves as the first responder to support tickets for small and new customers, providing immediate assistance while maintaining our commitment to quality service. You’ll receive faster initial responses, but rest assured that you can request a human support specialist at any point during your conversation if needed.

​

Datasets:

​

🆕 New features

• Easier Filtering with Auto-Complete
  • Filtering datasets is now faster and more intuitive with auto-complete suggestions. For example, when selecting an Amazon category, you can now choose from a predefined list instead of searching manually on Amazon.
• New API for Dataset Metadata
  • Developers can now access dataset metadata through our API, making it easier to integrate with automation tools like Zapier, Make.com, and n8n.io. This allows you to retrieve dataset details and schema to set up automated workflows and get the exact data you need. 👉 View API Documentation (link)

​

🚀 Improvements

• Better Search in the Marketplace
  • Searching for datasets is now more flexible, allowing for slight variations and even typos. This makes it easier to find what you need, even if your search isn’t perfectly accurate.
• Clearer Messages for Filter Limits
  • Reaching the daily filtering limit now triggers a clear message explaining the limit and next steps. You can also submit a request to our data experts for help in refining your dataset filters.
• Helpful Tooltips for Date Columns
  • Date columns now include tooltips that explain the date format, ensuring you understand that records for the selected dates are inclusive.

​

🐞 Bug Fixes

• Improved Usability and UI
  • We’ve made several design and usability improvements, including fixing typos, to make your experience smoother.

​

Scrapers:

​

🆕 New features

• New endpoint allows sync calls to retrieve the scraped data to the requested origin
  • We’ve introduced a new endpoint that enables synchronous calls to retrieve scraped data directly from the requested origin, improving efficiency and streamlining the process. This enhancement alongside the upcoming design changes that will be implemented in the next weeks will allow users to access and test their scraped data more quickly and reflect the potential value (link)
• New “Wikipedia articles” scrapers
  • Users can now discover (by keyword) and collect article information from Wikipedia.

​

🚀 Improvements

• Search and navigation
  • We’ve made several improvements to enhance search results and navigation, making it easier for users to find relevant data and explore related products. Search results will now prioritize domain-only results, ensuring more accurate and streamlined discovery. Additionally, Dataset (DS) results will be displayed below Scraper results and vice versa, providing a more structured and intuitive search experience.
• A new banner in SERP to redirect users interested in Chat GPT and perplexity search results
  • This banner redirects users to additional search engine results data, making it easier to access the AI-powered result searches

​

🐞 Bug Fixes

• Refresh Token Fix
  • We’ve resolved an issue affecting the token refresh option to ensure seamless authentication and session continuity.
• Scraper Request Call Scheduled Notification Fix
  • Fixed a bug where scheduled notifications for new custom scraper requests were not triggering correctly, ensuring timely and accurate alerts to the relevant personnel

​

Release notes for week ending Mar 13, 2025

​

Web Archive API:

​

🚀 Improvements

• Hot Storage Duration Web Archive API now offers 72h of hot storage data instead of 96h. For more details see here

​

Scrapers:

​

🆕 New features

• Output Schema Change Monitoring We have released our “output schema change monitoring system” to improve detection accuracy and transparency. Schema change notifications are now consolidated into a single email for customer updates.
• New discovery method for “Walmart products” Users can now discover and collect product information based on the SKU.

​

🚀 Improvements

• External Storage Option for Large Files If the compressed file still exceeds 5GB, users will be prompted with an option to send the file to an external storage location for easier access and management.
• Custom scrapers - modify the customs scraper request flow We are now routing calls based on customer type (Community, Named, Key, Strategic), and only the relevant personnel are added to the invite to increase efficiency
• Custom scrapers schema changes agility If there are any changes to the schema defined by the customer during the call, the team can now update the schema before it is sent to the partners.

​

Proxy products:

​

🚀 Improvements

• Enable moving datacenter and ISP proxies from legacy pay per GB zones to new datacenter and ISP zones with unlimited plans.

​

🐞 Bug Fixes

• Fix wrong country assignment for default country in Residential proxy network.

​

Datasets:

​

🆕 New features

• eCommerce Datasets now have Daily & Weekly Subscriptions

  You can now get daily or weekly updates for 41 out of 73 eCommerce datasets. This makes it easier to track new products in a category or monitor price changes more frequently. We’re working on adding support for more datasets soon!

  🚀 Improvements

• Improved Filtering Options

  • See Matching Records Before Applying Filters – Now you can see how many records match your criteria before applying filters.

  • Filter by Date – You can now filter datasets using relative dates for more precise data selection.

• Estimated Cost Label on Dataset Page view

  • We’ve added an estimated cost label below the purchase button so you can see pricing details before buying.

  • A new “Filter Dataset” button next to the cost label lets you quickly refine your dataset selection, helping you get the exact data you need while managing costs.

• Better Checkout Experience

  • Clearer Navigation – Breadcrumbs now guide you through the checkout process.

  • Smarter Updates – We’ve improved how we explain delta subscriptions (smart updates) so you can make better decisions.

  • Promo Codes – You can now apply discount codes directly at checkout.

• Better Marketplace Experience

  • Breadcrumbs in Marketplace – Easily track where you are while browsing datasets.

  • Redesigned Marketplace UI – We’ve refreshed the header, category list, and dataset cards grid to make browsing easier.

​

🐞 Bug Fixes

• Faster Quote Calculation

  removed unnecessary calculations in the checkout process, improving performance.

• Subscription Update Fix

  Updating a dataset subscription no longer redirects you to a different page, it will keep you on the same page.

• Improved Dataset Preview Speed

  The dataset preview now loads faster for a smoother experience.

​

Scrapers:

​

🆕 New features

• Cancel snapshot from the No-Code Scraper – Users can now cancel a running snapshot directly from the No-Code Scraper interface.

• New logs functionality – New buttons were added to the logs section to provide better interaction options and debugging capabilities to the customers

• New discovery method for “Airbnb Properties” – Allowing users to retrieve Airbnb property information based on filtered URLs.

• New discovery method for “Glassdoor Companies” – Users can now collect company information based on filtered URLs.

​

🚀 Improvements

UI Enhancements to Scrapers Home Page We’ve improved the usability and navigation of the Scrapers home page enabling faster search, displaying smaller cards for better display and explore more scrapes to users, enhanced Custom Scraper suggestions, and added search within category pages

• Unified the API and No-Code delivery options – We have fixed inconsistencies between the API and No-Code Scraper delivery settings. Now, both methods follow the same logic, making it easier for users to configure and manage their data delivery preferences.

​

🐞 Bug Fixes

• Fixed broken delivery settings – Resolved an issue that was causing delivery settings to not function correctly. This fix ensures that users can reliably configure and receive their scraped data without disruptions.
• Filter functionality - Filtering my “My scrapers” table according to product is now working
• URL structure - The URL structure were inconsistent across the scrapers pages (e.g, some api_data and some scrapers) now all URLs are consist in structure and contain the word scrapers as part of the URL
• Fixed search results page on Scraper home page – Corrected an issue where search results on the Scraper home page were not displaying correctly. This fix ensures that users can find relevant scrapers more easily.

​

Proxy products:

​

🚀 Improvements

• Added option to breakdown zone statistics by domain in zone overview page:

​

Web Unlocker:

​

🆕 New features

• You can now get HTML automatically converted into Markdown Language. It’s supported both in API and native proxy interfaces. See https://docs.brightdata.com/scraping-automation/web-unlocker/features#scrape-as-markdown

​

Datasets:

​

🆕 New features

• Support for Date Expressions in Dataset Filters:You can now use date expressions in dataset filters (Relative Date), making it easier to refine your data selection based on time-based criteria. This feature is supported in both the Filter API and Dataset Subscriptions, allowing for more flexible and dynamic filtering. Example Use Case: Suppose you want to filter records to include only those from the last 7 days. You can use the “Last” expression to retrieve all records from the past week while excluding older data.

​

🚀 Improvements

• We have enhanced the way Personally Identifiable Information (PII) is masked to improve data security and compliance.
• Improved Quote Generation Time on Checkout Page - loading time is now faster to get quote data and quote the final price.
• Released a new checkout page. Users can now see added and updated records directly on the checkout page.

​

🐞 Bug Fixes

• Fixed Preview Table Errors:Resolved an issue that caused slow loading times in the dataset preview table, ensuring faster and smoother data previews.

​

Scrapers:

​

🆕 New features

• Perplexity scraper - Introducing a brand-new Perplexity Scraper, similar to our chatGPT scraper! This scraper allows customers to interact with Perplexity programmatically, retrieving responses as structured data. Now, you can automate queries and extract insights from Perplexity effortlessly. Try it here

• Monitoring system for schema changes – The first release of a monitoring and alerting system for schema changes was implemented to improve data accuracy and stability. It detects changes and sends automatic notifications to ensure seamless data operation. Expect improvements to this system in the coming weeks.

• New delivery feature for the No-Code Scraper – We’ve added delivery methods, giving no-code users more flexibility in exporting and accessing their scraped data.

​

🚀 Improvements

• Enhanced search option to include new result types – The search functionality now displays data points along with their respective scrapers, making it easier to identify relevant scrapers according to their outputs.

• “Deliver to external storage” is now enabled by default – To streamline workflows, external storage delivery is now automatically enabled on the web scraper API, reducing manual setup steps.

• Custom scrapers request - We simplified the request process. Now, it only requires two URLs, and the rest will be handled automatically, to try and streamline the request process

​

🐞 Bug Fixes

• Resolved issues affecting custom scrapers and implemented an alert system to notify partners when action is required.
• Fixed undefined URL structure – Improved URL handling to ensure consistent and well-defined structures across different pages.

​

Proxy products:

​

🆕 New features

• New API will provide the list of proxies pending replacement in advance and prepare for. Try it here: Get proxies pending replacement API.

​

🚀 Improvements

• Integrate faster with pre-made connect strings for proxy access details (host, port, user and password) accompanied with new set of configuration guides Integration guides.
• We added error codes to our error catalog. Find and resolve issues faster by yourself Proxy errors catalog.
• We have increased our servers deployment in Asia, and added some performance boosts.

​

SERP API:

​

🆕 New features

• Google’s Generative AI Overviews - you can now increase the likelihood of this overview appearing in your SERP results:
  • https://docs.brightdata.com/scraping-automation/serp-api/query-parameters/google#ai-overview
• Google Trends - new parameters to get more accurate widget specific data:
  • `brd_trends = timeseries,geo_map`
  • https://docs.brightdata.com/scraping-automation/serp-api/query-parameters/google#trends

​

Datasets:

​

🆕 New features

• Delta Data for Dataset Subscriptions - Customers can now enable delta logic to receive only new or updated records instead of full datasets for every collection cycle. This improves data relevance and allows flexible snapshot comparisons. No more manually identifying changes, as our system does it for you.

​

🚀 Improvements

• New Pricing Table - A redesigned pricing table provides clearer information, making it easier for customers to understand pricing tiers and make informed decisions.
• Enhanced Dataset Checkout Confirmation Page - The confirmation page has been fixed to ensure a smoother and more reliable checkout experience.
• Improved Error Messaging for Missing Payment Details - Customers attempting to purchase without adding payment and billing details will now receive clearer and more informative error messages.
• Refined Delivery Methods - The order of delivery options has been reorganized, and UI improvements have been made for better usability.
• Keyboard Shortcuts for “Update Dataset” - New keyboard shortcuts have been implemented to streamline dataset updates and UI enhancements for a better user experience.
• Better Data Sample Preview Table - Users can now intuitively open table rows and view object data types in a popover for improved clarity and visibility.

​

🐞 Bug Fixes

• Fixed Routing Bug - Unnecessary query parameters no longer persist when navigating the dataset page.
• Resolved Promotions Card UI Issue - The promotions cards now display correctly according to the intended UI layout.

​

Scrapers:

​

🆕 New features

• AI Scraper Agents – Our first AI scraper agent allows users to extract data from any website by simply providing its domain. It automatically retrieves all URLs under that domain and extracts content in HTML, Markdown, or plain text. Ideal for AI companies, researchers, and developers needing high-quality text data for training models, analysis, or content generation. Try it here

• ChatGPT scraper - Introducing a brand-new ChatGPT Scraper, similar to our SERP scrapers! This scraper allows customers to interact with ChatGPT programmatically, retrieving responses as structured data. Now, you can automate queries and extract insights from ChatGPT effortlessly. Try it here

• “Run Again” Button in Logs Page - We’ve added a “Run Again” button to the Logs page, allowing users to quickly re-run the same request with a single click. making it easier to retry failed or past jobs without manually re-entering parameters.

​

🚀 Improvements

• My scrapers table- Scrapers that are currently running or the most recent scraper that was in use will now appear at the top of the “My Scraper” table. This makes it easier to track active jobs without having to scroll through completed ones.
• WSA API- To improve navigation and organization, we’ve moved the “Delivery Options” section to the “Management API” tab. This change makes it more intuitive for users managing API-based data deliveries.

​

🐞 Bug Fixes

• If a collection job fails in the No-Code Scraper, users can now download error details to better understand what went wrong. Previously, the download button was disabled, making debugging difficult. This improvement ensures better transparency and troubleshooting.
• We’ve resolved an issue where the “Report an Issue” feature did not correctly associate snapshots with tickets. Now, when customers report a problem, the correct snapshot is automatically linked, making it easier to troubleshoot and resolve issues efficiently.

​

Proxy products:

​

🆕 New features

• HTTP3 beta is live - The proxy network was upgraded to handle HTTP3/QUIC requests. Contact Bright Data sales - sales@brightdata.com if you want to try it out!

​

🚀 Improvements

• It is now easier to copy your proxy credentials on the ‘overview’ tab, in different formats.

​

🐞 Bug Fixes

• Fix broken code examples on account management API doc playground
• Added missing documentation for endpoint and some old doc links not redirecting properly.

​

Datasets:

​

🆕 New features

• “AI Filter Assistant” allows users to filter datasets using Natural Language - just describe desired filters in English and get the relevant records.

• Payment Link Sharing – Dashboard users can now generate and share payment links, making the purchasing process easier for others (e.g. their CFO) without logging in to the control panel.

• Additional Dataset Statistics – Each Dataset now includes additional statistics regarding its content, for example: geographical distribution of records. This information is updated every month.

​

🚀 Improvements

• Enhanced Filtering Experience – You can now move the filter window while reviewing and refining your filters and datasets. No more hiding of the table!
• Better data sample preview – We’ve limited the cell size for better readability. You can now click on fields to expand and view the full data point in the sample dataset.

​

🐞 Bug Fixes

• Fixed incorrect parsing of CSV files uploaded as inputs to the filter component.
• Made sure the record count always gets updated, to show the effect of the filter.

​

Scrapers:

​

🆕 New features

• Custom scrapers - Allows users to request a custom scraper if the domain is unavailable in our marketplace. Just supply 2 URLs to get the process started - see the video!

​

🚀 Improvements

• An automatic email notification that is being sent after a customer triggers a scraper for the first time, for better onboarding experience.
• Added code examples in multiple languages to allow users to copy the API code in their preferred language on all of our API requests
• Reduced the number of examples from 10 to 5 in the “Overview” tab to provide a “clear” view to the users
• Bright AI: Users can now ask questions about specific use cases and receive relevant suggestions along with a direct link to the dataset.

​

🐞 Bug Fixes

• Fixed broken links across the Scraper pages.