Automate model monitoring

Of course, you don’t want to manually build the model evaluation stores every time new data is available. You can automate this task with a scenario.

In addition to scheduling the computation of metrics, you can also automate actions based on the results. For example, the goal may be to retrain the model if a metric such as data drift exceeds a certain threshold. Let’s create the bare bones of a scenario to accomplish this kind of objective.

Note

This case monitors a MES metric using a check. You can also monitor datasets with data quality rules.

Create a check on a MES metric

The first step is to choose a metric important to the use case. Since it’s one of the most common, let’s choose data drift.

Tip

This example deliberately chooses a data drift threshold to demonstrate an error. Defining an acceptable level of data drift is dependent on your use case.

1. From the Flow, open the mes_for_input_drift, and navigate to the Settings tab.

2. Go to the Status checks subtab.

3. Click Metric Value is in a Numeric Range.

4. Name the check Data Drift < 0.4.

5. Choose Data Drift as the metric to check.

6. Set the Soft maximum to 0.3 and the Maximum to 0.4.

7. Click Check to confirm it returns an error.

8. Click Save.

Now let’s add this check to the display of metrics for the MES.

1. Navigate to the Status tab.

2. Click X/Y Metrics.

3. Add both the data drift metric and the new check to the display.

4. Click Save once more.

Design the scenario

Just like any other check, you now can use this MES check to control the state of a scenario run.

1. From the Jobs () menu in the top navigation bar, open the Scenarios page.

2. Click + New Scenario.

3. Name the scenario Retrain Model.

4. Click Create.

First, you need the scenario to build the MES.

1. Navigate to the Steps tab of the new scenario.

2. Click Add Step.

3. Select Build / Train.

4. Name the step Build MES.

5. Click Add Item > Evaluation store > mes_for_input_drift > Add.

Next, the scenario should run the check you’ve created on the MES.

1. Still in the Steps tab, click Add Step.

2. Select Verify rules or run checks.

3. Name the step Run MES checks.

4. Click Add Item > Evaluation store > mes_for_input_drift > Add.

Finally, you need to build the model, but only in cases where the checks fail.

1. Click Add Step.

2. Select Build / Train.

3. Name the step Build model.

4. Click Add Item > Model > Predict authorized_flag (binary) > Add.

5. Change the Run this step setting to If some prior step failed (that step being the Run checks step).

6. Check the box to Reset failure state.

7. Click Save when finished.

Tip

For this demonstration, you’ll trigger the scenario manually. In real life cases, you’d create a trigger and/or a reporter. Try these features out in Tutorial | Automation scenarios and Tutorial | Scenario reporters.

Run the scenario

Let’s introduce another month of data to the pipeline, and then run the scenario.

1. Return to the new_transactions dataset in the Model Scoring Flow zone.

2. On the Settings tab, switch the data to the next month as done previously.

3. Return to the Retrain Model scenario.

4. Click Run to manually trigger the scenario.

5. On the Last Runs tab, observe its progression.

6. Assuming your MES check failed as intended, open the saved model in the Flow to see a new active version!

Note

This goal of this tutorial is to cover the foundations of model monitoring. But you can also think about how this specific scenario would fail to meet real world requirements.

• For one, it retrained the model on the original data!

• Secondly, model monitoring is a production task, and so this kind of scenario should be moved to the Automation node.