Run more model evaluations#

Before diving into the meaning of these metrics, let’s add more data to the pipelines for more comparisons between the model’s training data and the new “production” data found in the new_transaction_data folder.

Get a new month of transactions#

  1. In the Model Scoring Flow zone, navigate to the Settings tab of the new_transactions dataset.

  2. In the Files subtab, confirm that the Files selection field is set to Explicitly select files.

  3. Click the trash can to remove /transactions_prepared_2017_04.csv.

  4. On the right, click List Files to refresh.

  5. Check the box to include /transactions_prepared_2017_05.csv.

  6. Click Save.

  7. Refresh the page to confirm that the dataset now only contains data from May.

Dataiku screenshot of the Settings tab of a dataset.

Rebuild the MES for input drift monitoring#

We can immediately evaluate the new data in the Input Drift Monitoring Flow zone.

  1. In the Input Drift Monitoring Flow zone, select the mes_for_input_drift.

  2. In the Actions tab of the right panel, select Build.

  3. Click Build Model Evaluation Store with the Build Only This setting.

Rebuild the MES for ground truth monitoring#

For ground truth monitoring, we first need to send the data through the Score recipe to maintain consistency.

  1. In the Ground Truth Monitoring Flow zone, select the mes_for_ground_truth.

  2. In the Actions tab of the right panel, select Build.

  3. Select Build Upstream.

  4. Click Preview to confirm that the job will run first the Score recipe and then the Evaluate recipe.

  5. Click Run.

  6. Open the mes_for_ground_truth to find a second model evaluation when the job has finished.

Dataiku screenshot of a model evaluation store with two evaluations.

Tip

At this point, both model evaluation stores should have two rows (two model evaluations). Feel free to repeat the process above for the months of June and beyond so that your model evaluation stores have more data to compare.