Tutorial | ML assertions#

ML assertions are checks that help you to systematically verify whether your model predictions align with the experience of your domain experts. Learn how to use assertions with this tutorial.

Create the project#

If you have already created the Machine Learning 102 project in your Dataiku instance, open it.

Otherwise, from the Dataiku Design homepage, click +New Project > DSS tutorials > ML Practitioner > Machine Learning 102.

Note

The Machine Learning 102 project is shared by the following tutorials:

  • Feature generation

  • Model overrides

  • ML assertions

  • ML diagnostics

  • What if analysis

  • Causal prediction

Double click on the ML diagnostics & assertions Flow zone.

Starting Flow for the tutorial.

Add assertions to a model#

This tutorial uses a dataset of a fictional company’s customers and some demographic and business information about those customers. It includes two models, already trained in the Lab, that predict whether each customer is a high-revenue customer, as indicated in the high revenue column.

Let’s add assertions based on domain knowledge and retrain the models to check our assertions.

To access the models:

  1. From the top navigation bar, click on Visual Analyses.

  2. Select the model Quick modeling of high_revenue on customers labeled.

Access the model under Visual Analyses.

Defining assertions#

There are no default assertions; you must define them based upon your domain knowledge. After defining assertions, we’ll need to retrain the models so they can evaluate the assertions.

For example, we may know from experience that while 10% of all customers are considered high revenue, those whose first order was made before they turn 30 years old are almost never high revenue.

To create an assertion that captures this knowledge:

  1. From the model Result page, click on the Design tab, then the Basic > Debugging panel on the left.

  2. In the Assertions section, click + Add an Assertion.

  3. Change the name of the assertion from Assertion 1 to Under 30 not high revenue.

  4. Define a condition where age_first_order < 30.

  5. Leave the Expected class as False and change With valid ratio to 100% to reflect that we expect none of these customers to be high revenue.

    The under 30 not high revenue assertion.
  6. Click Save then Train, then Train again in the info window.

After training is complete, a summary of the diagnostics is available. We can see that there are four diagnostics warnings for the logistic regression model, including one that our assertion failed for that model.

Summary results of quick model for high_revenue column, with diagnostics tooltip
  1. Click Diagnostics for the logistic regression model. This opens the model to the Training Information section.

  2. Navigate to the Performance > Metrics and assertions section.

Here we can see that 324 customers in the test set made their first order before the age of 30, and the model predicts that 1 will be a high revenue customer. (Your results may vary depending on your model.) This contradicts our domain knowledge that no such customers should exist and makes us pause before deploying the model.

Metrics and assertions section of model results.

Assertions in deployed model metrics#

Assertions are incorporated into metrics and checks for models deployed to the Flow. To see these:

  1. In the logistic regression model, click Deploy in the top right, then Create in the info window.

  2. In the Flow, open the deployed model and navigate to the Metrics & Status tab. By default, Build duration is the only metric displayed.

  3. Click the Display button that currently reads 1/13 Metrics.

    Model metrics & status tab.

    For each assertion, there are three available metrics: the number of dropped rows, the number of rows matching the assertion condition (in this case, customers whose age at first purchase is under 30), and the proportion of rows that are members of the expected class, according to the definition of the assertion.

  4. Add each of these metrics to the display:

    • Dropped rows: Under 30 not high revenue

    • Rows matching criteria: Under 30 not high revenue

    • Valid ratio: Under 30 not high revenue

    Model metrics display settings.
  5. Click Save.

The metrics match what we saw in the original analysis; 324 customers in the test set made their first order before the age of 30, and the model predicts that 323/324 = .9969 will not be high-revenue customers.

Like any other metrics, you can create checks for these metrics and use those checks in scenarios that rebuild this model. That way, when this project is put into production, you can be automatically notified when the rebuilt model does not pass these checks.

Model metrics

Assertions in evaluation metrics#

ML assertions can also be computed as one of the metrics from an Evaluate recipe. To see this:

  1. From within the deployed model, click Actions and then Evaluate.

  2. In this project, we don’t have an external validation dataset, so for now simply select customers_labeled as the input dataset.

  3. Create customer_scores as the Output dataset and customer_metrics as the Metrics, then click Create Recipe.

    By default, the Evaluate recipe computes any ML assertions as part of its output.

    Evaluate recipe outputs.
  4. Leave the Compute assertions box checked and click Run.

The resulting metrics dataset has a column called assertionsMetrics, which contains JSON of the form:

{
  "Under 30 not high revenue":
  {
    "result":false,
    "validRatio":0.9993939393939394,
    "nbMatchingRows":1650,
    "nbDroppedRows":0
  }
}

You can parse this with your own code, or use the Unnest object processor in a Prepare recipe to work with this data.