Tutorial | API endpoint monitoring#

Get started#

After deploying an API service with one or more endpoints, the next logical step is to set up a monitoring system to centralize the logs from the API node and to monitor the responses of endpoints. Let’s see how this can be done in Dataiku!

Objectives#

In this tutorial, you will:

Store API endpoint prediction logs into an usable data storage.
Use these logs to build a feedback loop monitoring data drift.

Prerequisites#

Dataiku 13.2 or later to use the streamlined method shown here. Users on older versions can use a manual method.
A Full Designer user profile on the Dataiku for AI/ML or Enterprise AI packages.
An active deployment of an API service including a prediction endpoint. If you don’t already have this, see Tutorial | Real-time API deployment to create one.

Create the project#

This tutorial can be completed with any project that includes an active API deployment including a prediction endpoint. For demonstration, we’ll reuse the project created in Tutorial | Real-time API deployment.

Locate API audit logs#

You can use the audit logs of the API node for several different use cases:

Understand the user base of the API service through information such as the IP address and timestamp for each query.
Check the health of the API, for instance, by monitoring its response time.
Monitor the API predictions for data drift (shown in this tutorial).
Implement an ML feedback loop. In the case of a machine learning model trained on a few data points, the queries sent to the API can be used, after manual relabelling, to retrain the original model with more data.

The first step is finding where the audit logs of the API node are stored.

As explained in Audit trail on Dataiku Cloud, these logs will be in an S3 connection called customer-audit-log.

Send test queries to the API endpoint#

The next step is to confirm that the API endpoint is an active deployment.

Send at least one test query from the Run and Test tab of the API deployment or submit Sample code in a terminal as done in the Query the API endpoint section of the API deployment tutorial.
If these tests fail, return to Tutorial | Real-time API deployment for guidance on deploying an API service.

Create a monitoring feedback loop#

Our goal is to import the logs from these queries into the Design node for analysis, and in particular, use them to monitor the model’s predictions.

Return to the API service on the Design node project.
For the prediction endpoint inside this service, navigate to the Monitoring panel, where you should see details of current deployments.
Click Configure to set up the monitoring loop.
Click OK.
Return to the Flow.
If using Dataiku Cloud, move the API log data and the model evaluation store to a new Flow zone.

Let’s review what this action did. The Flow now contains a new zone dedicated to monitoring this API deployment, including three objects:

API log data, partitioned by day (stored in the connection detailed above)
An Evaluate recipe with the API log data and saved model as inputs
A model evaluation store as output to the Evaluate recipe

Inspect the API log data#

Each row of the API log data corresponds to one actual prediction request answered by the model.

The exact columns may vary depending on the type of endpoint. For example, here we are using a prediction model, but it won’t be exactly the same for an enrichment or a custom Python endpoint.

Scrolling through the columns, you’ll find:

Basic information like the ID of the service, endpoint, its generation, deployment infrastructure, and model version.
Timing details.
All of the features that were requested.
The prediction results.
Additional information like IP address and the server timestamp.

Tip

Return to the sample queries you submitted to the endpoint and confirm for yourself that these are the same queries. In this example, in addition to queries from Tutorial | Real-time API deployment, we also have queries from Tutorial | Model monitoring in different contexts.

Inspect the Evaluate recipe#

Before running it, take a moment to compare this particular Evaluate recipe to the others you would have seen in Tutorial | Model monitoring with a model evaluation store.

In the Evaluation dataset tile of the Settings tab, note the presence of an option to Handle dataset as API node logs. Even though the API node data has a very different schema than the input data to the other Evaluate recipes, Dataiku is able to automatically recognize API node logs and ingest it without additional configuration.
The Model tile also has additional options to filter rows on model version and deployment ID. This setting matches model versions between the active model version in the Flow and the model version used when scoring the rows retrieved from the API node.

Note

If using a version of Dataiku prior to 11.2, you need to add a Prepare recipe that keeps only the features and prediction columns and renames them to match the initial training dataset convention.
Finally, in the Output tile, note that the box is checked to skip computation of performance metrics — just as was done for the recipe monitoring data drift. Remember that the ground truth is also missing here as this data is straight from the API endpoint.
After observing these settings, click Run to execute the Evaluate recipe.
Open the output MES to find the same set of data drift metrics that you would normally find.

What’s next?#

Congratulations! You’ve set up a feedback loop to monitor data drift on an API endpoint deployed in production.

If you have components of your monitoring pipeline outside of Dataiku, see Tutorial | Model monitoring in different contexts.