Data validation for Customers

The dataset must contain a certain percentage of unique values.

• Example: In the LinkedIn company profiles dataset, each company’s LinkedIn URL should be unique. If duplicate URLs exist, the same company is listed more than once, violating the uniqueness rule.

The dataset must contain a minimum percentage of filled values.

• Example: In a LinkedIn company profiles dataset, at least 90% of the profiles must have the ‘Industry’ field filled. If more than 10% of the profiles are missing this information (leaving the ‘Industry’ field blank), the dataset does not meet the required filling rate.

Certain fields must be filled; an error will occur if they remain empty.

• Example: Fields like ‘Company Name’ and ‘Headquarters Location’ might be mandatory in the LinkedIn dataset. Any profile lacking this information is flagged as an error.

Compared to previous values collected, the number value must not change by more than X.

• Example: If the dataset is updated regularly, the number of employees for a company shouldn’t drastically change between updates (e.g., a sudden jump from 50 to 5000 employees) unless there’s a known reason (like a merger).

Verifies each entry’s data type against its field type (e.g., string, number, date) and flags mismatches for correction.

• Example: The dataset should only accept date formats in the ‘Foundation Date’ field. If a text string like ‘unknown’ is entered, it should be flagged for correction.

Create a custom rule to validate whether a field exists and its value is valid, such as requiring the size string to be ‘S,’ ‘M,’ or ‘L.‘

• Example: The dataset might have a field for ‘Company Size’ with accepted values like ‘Small,’ ‘Medium,’ and ‘Large.’ A record must be flagged if it has a value outside these options.

The dataset must have X records (each URL should have X records from the total URL inputs).

• Example: If the dataset aims to represent companies in a specific sector, like technology, it must have a minimum number of company profiles from this sector to be considered complete and representative.

Determines whether dataset size fluctuations are within a +/- X% range.

• Example: For datasets updated monthly, the total number of companies listed shouldn’t fluctuate wildly (e.g., more than 10% increase or decrease) from one month to the next unless a specific event or trend is affecting the industry.

Checks each individual record in the dataset to ensure it doesn’t have a high percentage of empty or null fields. If a record has more than a predetermined threshold (e.g., 70%) of its fields empty or null, it triggers an error.

• Example: In the LinkedIn company profiles dataset, if a specific company’s profile has more than 70% of its fields (like industry, size, location, description) empty, this rule would flag it as incomplete.

Detects and resolves issues that lead to multiple duplicate records in the dataset due to improper identity assignment or entry errors. It ensures each record is distinct and accurately represents a unique data point.

• Example: In the LinkedIn dataset, this rule would identify cases where the same company is listed multiple times due to errors in assigning unique identifiers. For instance, if slight variations in company profiles (like different spellings of a company name) result in the same company appearing as separate entries, this rule would flag them for correction.