# Hands-On: Explain Your Model¶

In the previous hands-on lesson, you modified various settings that improved the model’s performance. Before putting this model into production, we want to make sure we understand its predictions. It is advisable to investigate whether its predictions are biased, in this case for example, by age groups or gender.

With this goal in mind, we return to a few of the options first introduced in the lesson on evaluating your model.

In order to check the model for bias, we’ll perform three kinds of tests:

Once again, return to the Models tab of the *High revenue analysis*. Open the report of the best performing model, the random forest model from Session 3.

## Partial Dependence¶

First, let’s use a partial dependence plot to understand the effect of a **numeric** feature (*age_first_order*) on the target (*high_revenue*).

Click

**Partial dependence**in the left panel to open the partial dependence page of the output.Specify

**age_first_order**as the variable.Click

**Compute**.

The partial dependence plot shows the dependence of *high_revenue* on the *age_first_order* feature, computed on the test set (2177 rows).

A negative partial dependence value represents a negative dependence of the predicted response on the feature value.

A positive partial dependence value represents a positive dependence of the predicted response on the feature value.

For example, the partial dependence plot shows that *high_revenue* being “True” has a negative relationship with *age_first_order* for ages below 39 years. The relationship reaches a plateau between ages 50 and 70, but then drops off until age 78.

Underneath the trend line, the plot also displays the distribution of the *age_first_order* feature. From the distribution, you can see that there is sufficient data to interpret the relationship between the feature and the target.

Let’s see what the partial dependence plot looks like for a **categorical** feature *gender*.

Select

**gender**as the variable.Click

**Compute**.

The partial dependence plot shows that *high_revenue* being “True” has a negative relationship with *gender* being “F”. In the cases where *gender* is “M” or has no value, then the relationship is positive. The *gender* distribution (on the right) is roughly equal between males and females, and it accounts for about 90% of the data.

## Subpopulation Analysis¶

Another useful tool to better understand the model is subpopulation analysis. Using this tool, we can assess if the model behaves identically across subgroups or if the model shows biases for certain groups.

Let’s use a subpopulation analysis to understand how our model behaves across different gender groups.

Click

**Subpopulation analysis**in the left panel to open the subpopulation analysis page of the output.Specify

**gender**as the variable.Click

**Compute**.

The table shows a subpopulation analysis for *gender*, computed on the test set. The overall model predicted that *high_revenue* was true 18% of the time when it was actually true only 9% of the time.

For the “F” subgroup, the model predicted that

*high_revenue*was true 16% of the time when the actual number was 8%.Similarly, the model predicted “True” 18% of the time for the “M” subgroup, when the actual number was 10%.

The predicted probabilities for male and female are close, but not identical. We can investigate whether this difference is significant enough by displaying more metrics in the table and more detailed statistics related to the subpopulations represented by the “F” and “M” rows.

Click the

**Displayed Metrics**dropdown at the top right of the table, and select**F1 Score**. Note that this metric considers both the precision and recall values. The best possible value is one, and the worst is zero.Click anywhere on the “F” and “M” rows to expand them.

In this analysis, the male subgroup has the highest F1 score (0.39) of all the groups, even though this score is quite low. Also, the confusion matrices (displaying **% of actual classes**) for both the male and female groups show that the male subgroup does better (54%) at correctly predicting *high_revenue* to be true than the female subgroup (42%).

## Individual Explanations¶

Apart from exploring the effects of features on the model, it can also be useful to understand how certain features impact the prediction of specific rows in the dataset. The individual prediction explanations feature allows you to do just this!

Note that this feature can be computed in two ways:

From the

**Individual explanations**tab in the model results page.Within a Scoring recipe (after deploying a model to the flow), by checking the option

**Output explanations**. See the**Scoring**course for an example.

Let’s use the Individual explanations tab in the model results page to visualize the five most influential features that impact the prediction for specific samples in the dataset.

Click

**Individual explanations**in the left panel to open the page.Specify

`5`

most influential features to use for the explanation.Keep the default

**ICE**method.Click the gear icon in the top right corner to see more settings, such as the sampling details. Keep the “Sample size” as

`1000`

. This sample is drawn from the test set because the model implemented a simple train/test split. If the model implemented K-Fold validation, then the sample would be drawn from the entire dataset.

Move the left slider close to

`0.10`

(corresponding to “~109 rows”) and the right slider close to`0.70`

(corresponding to “~74 rows”) to specify the number of rows at the low and high ends of the predicted probabilities. Note that the probability density function in the background is an approximation based on the test set. The boxes to enable low and high probabilities are checked by default, so that you can move the sliders.

Click

**Compute**.

Note

Depending on the exact location of the sliders, your values may be different from the ones shown in this analysis.

DSS returns 137 rows for which the output probabilities are less than `0.10`

, and 87 rows for which the output probabilities are greater than `0.70`

. Each row explanation is represented in a card below the probability density plot. Notice that DSS has selected *customerID* as the identifier, but you can change this selection.

On the left side of the page, the cards have low probabilities of *high_revenue* being “True” and are sorted in order of increasing probabilities. In contrast, the cards on the right side of the page have high probabilities of *high_revenue* being “True” and are sorted in order of decreasing probabilities. For each card, the predicted probability is in the top right, and the “customerID” (card identifier) is in the top left.

For cards on the left side of the page, observe that all of the bars are red and oriented to the left. This reflects that the predicted probability is below average and the features negatively impact the prediction. In some cases, some of the bars may have counter effects (green and oriented to the right), even so, the net effect of the features will still be negative for cards on the left side of the page.

The opposite observation can be made for the cards on the right side of the page, where the bars are mostly — if not always — green and oriented to the right to reflect the positive impact of the features on the outcome.

You can click **Features** in a card to display the full list of all its features and their values.

## What’s next?¶

Congratulations! In an effort to seek out unintended biases, you have performed three different kinds of analyses that can help you interpret your model.

If you want to see how far you can go with your model, or how you can actually use it in an automated way to score new data as it arrives, complete the **Scoring** course.