Data validation for Customers
Overview
Bright Data’s automated dataset creation platform consists of a verification and approval phase before the dataset delivery. The platform facilitates error handling, validation checks, and customization, ensuring data accuracy and reliability. These validation checks are crucial in saving time, reducing errors in the data, and keeping the data quality at the desired level.
How does it work?
Once the dataset snapshot is ready:
The user will get the dataset with an indication in the platform that all tests passed.
The user will get the dataset with an indication in the platform that all tests passed.
The developer reviews the issues and will decide whether to:
- Fix the dataset according to the failed tests.
- Deliver the dataset to the user, explaining why the validation test failed but was overridden.
The user can then decide to:
- Approve the snapshot.
- Approve the snapshot for this time frame only.
- Reject the snapshot, and we will fix the scraper accordingly.
When the user approves the dataset snapshot, they proceed to the delivery phase.
Validation Rules
Main components and functionality
Overall view of dataset tests (All results, Passed, Failed)
Evaluating the validation test results
Once the dataset snapshot validation errors are handled, the user is notified to evaluate and choose whether to:
- Approve
- Approve temporarily
- Reject the snapshot.
Evaluation actions
For each failed validation test, the user has three options:
- Set a new threshold
- Set custom values - If the developer doesn’t reach the default set value, the user can choose a new threshold. The snapshot is returned to the developer once a new threshold is set.
- Set for X% - Accept the success rate the developer reached and set the threshold to the value the developer managed to extract.
- Ignore test (one time only) - Accept the value that the developer extracted just once (the default value will not change for the next dataset snapshot)
- Reject - The user doesn’t accept the adjustments to the failed tests; the issues will be returned to the developer to fix. The status will be marked as “Rejected” for additional fixes and will later on be re-send to the customer for approval.
In case all issues are ignored/approved, click on “Deliver dataset” to deliver the snapshot.
In case all/some issues are rejected, click on “send back to the developer” to send it back for additional fixes.
Communications and notifications
Users are notified of status updates through their account in the control panel and via email.