Concept: Model Summary Overview

The Summary panel of the Report page displays general information about the model, such as the algorithm and training date. In addition, the report page also contains sections relating to the model interpretation, performance, and detailed information about the model.

Model summary page of an XGBoost model.

Model Interpretation (Explainability)

In the Explainability section, we can see that the Variable importance tab displays the global feature importance of our model. In this chart, the Discharge Disposition feature has the strongest relationship with hospital readmission rates.

Variable importance chart showing global feature importance of the model.

Partial dependence plots help us understand the effect an individual feature has on model predictions. For example, computing the Partial dependence plot of the Age feature reveals the likelihood of hospital readmission increases approximately from age 60 to 80.

Partial dependence plot showing dependence of the predicted response on a single feature.

Subpopulation analysis allows us to assess the behavior of the model across different subgroups. For example, we can analyze the model performance based on Gender. The results show similar model behavior across genders, with a slight decrease in performance for male patients.

Subpopulation analysis showing behavior of the model across different subgroups.

While global variable importance can be a useful metric in determining overall model behavior, it does not provide insights into individual model predictions. Dataiku allows users to generate individual prediction explanations. For example, we can see which features most influenced our model’s prediction for individual patients.

Individual prediction explanations feature.

Some of the panels in the Explainability section are algorithm-dependent. For example, a linear model, such as Logistic Regression, will display information about the model’s coefficients, instead of variable importance, as in the case of an XGBoost model.

Model Performance

The Confusion matrix compares the actual values of the target variable with the predicted values. In addition, Dataiku displays some associated metrics, such as “precision”, “recall”, and the “F1-score”. For example, the Confusion matrix shows that our model has a 44% false-positive rate, and a recall of 84%.

Confusion matrix of an XGBoost model.

By default, Dataiku displays the Confusion matrix and associated metrics at the optimal threshold (or cut-off). However, manual changes to the cut-off value are reflected in both the Confusion matrix and associated metrics in real-time.

The Decision chart tab displays a graphical representation of the model’s performance metrics for all possible cut-off values.

Decision chart representing the model's performance metrics.

The Decision chart also shows the location of the optimal cut-off (based on the F1 score), which is 0.35 for our XGBoost model.

The Lift charts and ROC curve are visual aids that can be used to assess the performance of a machine learning model. The steeper the curves are at the beginning of the graphs, the better the model.

Lift chart visual aid used to assess performance of a model.

Finally, the Density chart illustrates how the model succeeds in recognizing and separating the classes. While in a perfect model the probability densities for the different models would not overlap, this is almost never the case for models trained on real data. For our XGBoost model, we are able to observe two distinct distributions, with their medians separated by a 12% predicted probability.

Density chart with medians indicated by vertical lines.

Model Information

Finally, let’s explore the Model Information section. The Features tab includes information on feature handling, as well as a list of all preprocessed features. For our XGBoost model, we can see that the Encounter ID feature was rejected, while the Number of Inpatient Visits was processed as a numeric feature and standardized.

Model information page with feature handling information.

The Grid-search optimization panel shows a history of all models trained during the hyperparameter grid-search. In our case, we can see that Dataiku trained two XGBoost models, varying the “max_depth” hyperparameter from 2 to 3.

Model information page with grid-search optimization information.

The Algorithm panel contains information on the optimum model resulting from the hyperparameter grid search. For our XGBoost model, we can see that Dataiku selected the XGBoost model with a “Max tree depth” of 5.

Model information page with algorithm information.