Hands-On Tutorial: Manage Multiple Versions of an API Service¶

Once you’ve successfully deployed an API service to production, it is important to monitor the service and update it as needed. You’ll also want to monitor the different versions of your API service so that you can roll back to previous versions, when needed, or even use the multiple versions for performing A/B testing.

For this tutorial, you’ll continue working with the project from the Hands-On Tutorial: Create Endpoint and Test Queries. You’ll also work with the API service that you deployed to production in the Hands-On Tutorial: Deploy Real-Time API Service.

Let’s Get Started!¶

In this tutorial, you will:

create a scenario with steps to retrain the prediction model and update the API service if the new version of the model performs better; and
deploy multiple versions of the prediction endpoint to use for A/B testing.

Prerequisites¶

This tutorial assumes that you satisfy the technical prerequisites for the MLOps Practitioner learning path.

Create a Scenario To Update the API Service¶

Return to your Dataiku project.

From the top navigation bar, go to the Jobs menu and click Scenarios.
Create a scenario called Update_Endpoint.

We’ll begin adding steps to the scenario. First, we’ll add a step to build/train the prediction model.

Click the Steps tab of the scenario.
Add a first step: Build/Train.
Click Add Model to Build and add the Predict authorized_flag (binary) model. This model should be the only available model. Keep the default values for the other parameters.

Next, we’ll add a step to create a new version of the API service from the API Designer.

Add a second step: Create API service version.
Select fraud_detection as the API service.
Name the “New version id.” retrained_model.
Check the box to “Make version id. unique”. This will suffix the version id with a unique identifier every time a new version is created.
Name the “Target variable” version_id.
Check the box to “Publish on Deployer”. The “Target service” should automatically select your previously deployed fraud_detection API service.

Add a scenario step to create API service version.

After creating a new version of the API service, we’ll want to update the deployment so that the new version gets activated.

Add a third step: Update API deployment.
Select the “Deployment id” that resembles fraud_detection-on-[INFRA_NAME]. We’re deploying to the dev infrastructure; therefore, we select fraud_detection-on-api-dev-v10, but yours may differ.
Specify the “New version id” as ${version_id} to reference the target variable that we created in the second scenario step. Recall that this variable stores the name of the latest version of the endpoint.

Add a scenario step to update the API deployment.

After adding the scenario steps, we’ll schedule the scenario to run at a specified time automatically.

Go to the Settings tab of your scenario.
Create a time-based trigger that will automatically run the scenario once a month, at midnight.
You can also add a reporter to your scenario if you like.

Add a trigger to automatically run the scenario.

Click the Run button to run the scenario manually.
Wait for the run to finish (it might take a few minutes). You can view the logs of the scenario run by switching to the Last runs tab.
Once the scenario run is done, return to the API Deployer node (you may have to refresh the page if your API Deployer is already open).

In the API Deployer node, notice that the active version of the model on the development infrastructure is the one that was just deployed by the scenario.

Active version of the deployed model is the retrained model.

From the “Versions to deploy” panel (left-hand side of the page), click retrained_model in the fraud_detection API service.

See the version history of the API endpoint.

Here, you can see that all versions of the API endpoint remain in the history. If needed, you can easily roll back to a previous version by simply clicking Deploy on the version you want.

Deploy Multiple Versions of the Endpoint for A/B Testing¶

When you’ve deployed multiple versions of your prediction endpoint, you may decide to run multiple generations of the endpoint at once. This allows the multiple versions of your prediction model to be used at random for scoring requests to the API service.

In the API deployer, return to the Deployments tab.
Click the fraud_detection API service that is deployed on the development infrastructure to open it.
Click the Settings tab of the service.
Change the “Active version mode” from “single generation” to Multiple generations.

In the “Entries” field, we will define a mapping to enable multiple versions of the endpoint. The mapping is in JSON format and specifies one or more generations and their corresponding probabilities. The probabilities indicate the likelihood that calls to the API node will be served to each of the generations. Therefore, the sum of the probabilities of all the generations must equal one.

Specify the value of the “Entries” field as follows:

[
   {
      "generation": "retrained_model",
      "proba": 0.7
   },
   {
      "generation": "v1",
      "proba": 0.3
   }
]

This mapping specifies that calls to the API endpoint will be served to the retrained_model version 70% of the time and to the v1 version the remaining 30% of the time.

Specify settings for multiple generations of an endpoint.

Note

Setting up multiple generations must be done manually and cannot be done through the automated deployment of API services.

Click the Save and Update button to update the API service with the new settings.
Select the Full update option.
Click the Status tab and then the Run and test tab.
Click Run All to run a few test queries.
Click the Details of each response to see which version of the API endpoint generated the response.

Test queries and see which version of the API endpoint generated the response.

What’s Next?¶

Congratulations! You successfully used a scenario to retrain the prediction model and update the API service with a new model that performs better. You also saw how to use multiple versions of the prediction endpoint in your API service to implement A/B testing.

The product documentation provides more information on managing versions of your endpoint.

To continue learning, see the next lesson, Monitor Output of API Endpoints, to learn how to set up a monitoring system that centralizes the logs from the API node and monitors the responses of endpoints.