🚩 Create a free WhyLabs account to get more value out of whylogs!

Did you know you can store, visualize, and monitor whylogs profiles with the WhyLabs Observability Platform? Sign up for a free WhyLabs account to leverage the power of whylogs and WhyLabs together!

Data Validation at Scale - Detecting and Responding to Data Misbehavior¶

In this tutorial, we'll introduce the concept of data logging and discuss how to validate data at scale by creating metric constraints and generating reports based on the data's statistical profiles using the whylogs open-source package.

We will also walk through steps that data scientists and ML engineers can take to tailor their set of validations to fit the specific needs of their business or project, take actions when their rules fail to be met, and debug and troubleshoot cases where data fails to behave as expected.

Agenda¶

• Session 1: Introduction to Data Logging with whylogs
• Session 2: Data Validation with Metric Constraints
• Session 3: Per-value constraints with Condition Count Metrics
• Session 4: Auto-constraints generation
• Session 5: Debugging Failed Conditions

What is Data Validation?¶

Data validation is the process of ensuring that data is accurate, complete, and consistent. It involves checking data for errors or inconsistencies, and ensuring that it meets the specified requirements or constraints. Data validation is important because it helps to ensure the integrity and quality of data, and helps to prevent errors or inaccuracies in data from propagating and causing problems downstream.

In whylogs, you validate data by creating Metric Constraints and validating those constraints against a whylogs profile.

Session 1 - Data Logging¶

Installing whylogs¶

To install the whylogs library, you can use the following command:

Loading a Pandas DataFrame¶

Before showing how we can log data, we first need the data itself. Let's create a simple Pandas DataFrame:

Profiling with whylogs¶

To obtain a profile of your data, you can simply use whylogs' log call, and navigate through the result to a specific profile with profile():

Analyzing Profiles¶

Once you're done logging the data, you can generate a Profile View and inspect it in a Pandas Dataframe format:

This will provide you with valuable statistics on a column (feature) basis, such as:

• Counters, such as number of samples and null values
• Inferred types, such as integral, fractional and boolean
• Estimated Cardinality
• Frequent Items
• Distribution Metrics: min,max, median, quantile values

Writing to Disk¶

You can also store your profile in disk for further inspection:

Reading from Disk¶

You can read the profile back into memory with:

Session 2 - Data Validation with Metric Constraints¶

Constraints are a powerful feature built on top of whylogs profiles that enable you to quickly and easily validate that your data looks the way that it should. There are numerous types of constraints that you can set on your data (that numerical data will always fall within a certain range, that text data will always be in a JSON format, etc) and, if your dataset fails to satisfy a constraint, you can fail your unit tests or your data pipeline.

You can create Metric Constraints in two different ways:

• By directly using out-of-the-box helper constraints, such as greater_than_number, no_missing_values, or is_non_negative.
• By assembling your own set of tailored constraints.
• By automatically generating constraints from a reference profile

Once you have a set of constraints, you can use it to validate or generate a report of a Profile.

In this session, we'll show how to create constraints through the first approach: by using out-of-the-box helper constraints.

The above diagram depicts the relationship between the concepts presented so far:

As its name suggest, a metric constraint uses metrics in a profile to perform the validation. With no additional customization, the constraint will have at its disposal standard metrics captured in the logging process, such as Distribution, Frequent Items and Cardinality metrics.

Loading the Data¶

To begin, let's load the data into our environment. We will be using two datasets: df_reference and df_target.

We will create our set of constraints assuming the existence of previous domain knowledge and experience with the data. We will use df_reference as our reference dataset to create the constraints, and df_target as the dataset to be validated. In an ML scenario, df_reference will usually be the training dataset, and df_target the data you encounter in production on the system's day-to-day operation.

Defining the Constraints¶

Let's define a list of constraints to apply to our data in the tables below.

For each of these constraints, we will show the feature it will be applied to, used parameters, and also a brief explanation of what it checks for.

Completeness Constraints¶

Consistency Constraints¶

Statistics Constraints¶

Profiling and Building the Constraints¶

To build our constraints, let's first profile our target dataset:

Now, we create our constraints with the help of ConstraintsBuilder. That will allow us to iterate over the constraints we wish to build. Let's first show how to add a single constraint. Let's begin with the no_missing_values constraint for the id feature:

Now, we can call constraints.validate to validate our constraints against the target profile:

validate will return True if all the constraints passes, and False otherwise. In this case, we can see that the constraint passes, as there are no missing values in the id column.

We can also call generate_constraints_report to generate a report of the constraints against the target profile:

Let's continue adding constraints as defined previously:

We can also visualize the constraints report with the viz module. With it, you can filter the displayed constraints by name or status (pass or fail).

If you hover on each constraint's status, it will provide you with additional context that was used to determine the constraint's status.

Session 3 - Per-value constraints with Condition Count Metrics¶

whylogs profiles contain summarized information about our data. This means that it’s a lossy process, and once we get the profiles, we don’t have access anymore to the complete set of data.

This makes it impossible to create some types of constraints from the standard metrics themselves. For example, suppose you need to check every row of a column to verify that there is no textual information that matches a credit card number or email information. Or maybe you’re interested in ensuring that there are no even numbers in a certain column. How do we do that if we don’t have access to the complete data?

The answer is that you need to define a Condition Count Metric to be tracked before logging your data. This metric will count the number of times the values of a given column meet a user-defined condition. When the profile is generated, you’ll have that information to check against the constraints you’ll create.

• A whylogs profile contains a set of metrics that summarize the original data.
• Metric Constraints validate Metrics present on the Profile to generate a Report. This report will tell us whether the data meets the data quality constraints we defined.
• Those metrics can be standard metrics, such as Distribution, Frequent Items, or Types metrics. They can also be Condition Count Metrics.
• Condition Count Metrics count the number of times a certain relation passed/failed. For example, the number of times the rows for the column Name was equal to Bob
• For a profile to contain Condition Count Metrics, you first have to specify a Condition before logging your data, and pass that to your Dataset Schema.
• The Dataset Schema configures the behavior for tracking metrics in whylogs.

Defining the Conditions¶

The conditions are defined through a whylogs' Condition object. There are several different ways of assembling a condition. In the following example, we will define a custom function that will verify if the string is properly formatted as a date. We'll also define a second relation that will validate the text against a defined regex pattern to verify the url formatting.

Since we can define multiple conditions for a single column, we'll assemble the conditions into dictionaries, where the key is the condition name. Each dictionary will later be attached to the relevant column.

Creating the Schema and Logging the Data¶

whylogs must be aware of those conditions while profiling the data. We can do that by creating a Standard Schema, and then simply adding the conditions to the schema with add_resolver_spec. That way, we can pass our enhanced schema when calling why.log() later.

Generating constraints¶

Session 4 (Bonus) - Constraints Auto Generation¶

Constraints can be built:

1. From domain knowledge
2. Based on a Reference Dataset

Until now, we have demonstrated how to build constraints based on the user's experience and knowledge about the data. Another way to get us started is by automatically generating a set of constraints based on a Reference Profile. This is useful when we have a dataset that we know is of good quality, and we want to ensure that future data is similar to it.

Let's do it all on one block: We will ask for suggested constraints based on a reference profile with generate_constraints_from_reference_profile, and then we will validate those constraints against our target profile. Then, we will visualize our constraints report by calling constraints_report:

Note. The constraints generated by generate_constraints_from_reference_profile are not guaranteed to be the best constraints for your data. They are just a starting point. You should always review the constraints and adjust them to your needs. The feature is also currently experimental, and we are working on improving it.

Session 5 - Debugging Failed Conditions¶

With Condition Validators, the user is able to evaluate conditions on individual values on real-time scenarios. These checks are done while data is being logged, and can trigger one or multiple actions when these conditions fail to be met. With Condition Validators, you are able to define actions where an immediate response is required, such as emiting an alert to key stakeholders, logging specific failures or throwing exceptions. Validators are designed with flexibility in mind, so you are free to customize your actions as well as the conditions that trigger those actions.

Unlike metrics, validators will not log properties into profiles. They are meant only to evaluate conditions and trigger actions while logging is under way.

Additionally, the validator retain contextual information about the data that failed the conditions. This is done through the process of Reservoir Sampling. This means that the validator will retain a sample of the data that failed the conditions, and will be able to provide that information when the user requests it.

Data Validators¶

Note: Actions can also be executed using Conditions. In upcoming releases, our goal is to unify all the functionalities introduced in Sessions 4 and 5 within the Conditions framework.

Defining the Actions¶

The action to be triggered when a contidion fails is created by simply defining a regular function.

We should just remember to define the arguments: validator_name, condition_name and value. You can use these values to help with logging and debugging the failures.

To create a Condition Validator, we need a name, a set of conditions, and a list of actions.

Let's make a Validator for the date format validator and another Validator for the url validation. Each validator has a single condition to be evaluated, and also a single action.

Now we're set to log our data. We'll use the same schema as before, but this time we'll pass our validators to the log method.

You can see that the validators are being triggered when the conditions fail. The validators will also retain a sample of the data that failed the conditions. We can access that data by calling get_samples on the validator.

Additionally, if you call to_summary_dict on the validator, you'll get a summary of the validator's status, as well as the number of times the conditions were evaluated, and the number of times the conditions failed.