Leverage test scenarios in deployment pipelines#

Running test scenarios on a Design node is only the beginning. To derive the maximum amount of value from test scenarios, you need to incorporate them into a production workflow.

An organization might leverage test scenarios into the overall framework of their deployment processes in many possible ways. Rather than outline detailed procedures, for the remainder of this tutorial, we’ll illustrate one possibility, which may inspire you to consider how test scenarios can be incorporated into Dataiku’s existing MLOps ecosystem.

Deploy to a QA infrastructure#

The Deployer allows organizations to structure their deployments into lifecycle stages. By default, these stages may be named Dev, Test, and Prod. Organizations though may use their own terminology. The key point is that multiple stages allow for additional levels of scrutiny in the deployment lifecycle.

This is important as development and production environments may have important differences when it comes to settings like connections, code environments, plugins, recipe engines, and user permissions.

In our setup, we have two Automation nodes: one for QA and one for production. Accordingly, our first step was to deploy the bundle to the QA infrastructure.

We previously published a bundle from the Design node to the Project Deployer.
From the Project Deployer, we deployed the bundle to a QA Automation node called xops-qa.
In doing so, a post-deployment hook running the test scenarios failed.

Use a post-deployment hook to run test scenarios#

Deployment hooks are custom Python actions executed before or after a deployment to an infrastructure.

In this case, our QA infrastructure has a post-deployment hook. Accordingly, the deployment of the failing bundle was still able to succeed. In this setup, we allow failures on the QA infrastructure. We just want to know about them.

Let’s take a look at where the post-deployment hook warning comes from.

On the Project Deployer, go to Infrastructures.
Select your infrastructure.
Navigate to the Settings tab.
Go to the Deployment hooks panel.
View the Post-Deployment Hooks.
The one of interest in our case is called Running all test scenarios.

Below is a sample of this kind of hook.

import time

def execute(requesting_user, deployment_id, deployment_report, deployer_client, automation_client, deploying_user, deployed_project_key, **kwargs):

    project = automation_client.get_project(deployed_project_key)
    cnt = 0
    time.sleep(3)
    for scn in project.list_scenarios(as_type="objects"):
        if scn.get_settings().get_raw()['markedAsTest'] :
            print(f"Execution test scenario {scn.id}")
            cnt +=1
            run_handler = scn.run_and_wait()
            if run_handler.outcome != "SUCCESS" :
                HookResult.error("Failure when running test scenario {scn.id}")
            else :
                print(" --> Execution successful")

    return HookResult.success(f"All test scenarios have been executed successfully ({cnt} scenarios)")

Run test scenarios on the QA infrastructure#

Now that we have a bundle running on the QA infrastructure, the next step was to execute the test scenarios.

From the QA Automation node, we executed one of the test scenarios.
We checked the results in the test dashboard.
On an Automation node, this test dashboard enables us to switch between bundles of the same project.
Note that you can download XML or HTML versions of the report.

Tip

One possible use of the report available for download is as an input to sign-off processes in the Govern node.

Prevent a deployment with a pre-deployment hook#

Although this bundle has failing tests, we attempted to deploy it to a production infrastructure. This will demonstrate how a pre-deployment hook can be used as an extra layer of security to prevent an unwanted deployment.

From the Project Deployer, we attempted to deploy the failing bundle to a production infrastructure.
This time, a pre-deployment hook caused the project update to fail. The bundle was not deployed to the production infrastructure.

If we look at the deployment hooks attached to the production infrastructure, we find a pre-deployment hook called Validate Test Status with the following code:

import dataikuapi

QA_INFRA_ID = "xops-qa"

def execute(requesting_user, deployment_id, deployment_report, deployer_client, automation_client, deploying_user, deployed_project_key, **kwargs):
    print("----------CONTROLLING TEST STATUS-----------------")
    # Fetch deployment details
    pdpl = deployer_client.get_projectdeployer()
    deplyt = pdpl.get_deployment(deployment_id).get_settings()

    # Identify the bundle that is being deployed
    bundle_id = deplyt.get_raw()['bundleId']
    pub_prj = pdpl.get_project(deplyt.get_raw()['publishedProjectKey'])

    #Find data on the corresponding deployment on QA node
    qa_dpl = pub_prj.get_status().get_deployments(infra_id = QA_INFRA_ID)
    if not qa_dpl:
        return HookResult.error(f"Cannot find deployment of this project on QA instance, please deploy and test first")

    qa_status = qa_dpl[0].get_testing_status(bundle_id)
    print(f"Found test report for bundle '{bundle_id}' on infra '{QA_INFRA_ID}':\n{qa_status}")

    if qa_status['nbScenariosPerOutcome']['FAILED'] > 0 :
        return HookResult.error(f"Deployment not allowed as test status of bundle {bundle_id} is not OK ({qa_status})")

    return HookResult.success("All tests are OK, validating deployment to production")

Fix the failing test scenarios on the Design node#

Unless the source of the failing tests is the production environment itself, when faced with failing test scenarios, it is important to correct them in the Design node project and create a new bundle to insert into your deployment pipeline.

On the Design node, fix any problems leading to failures in test scenarios and re-execute them.
Confirm that the test dashboard on the Design node shows no failures.
Create a new bundle.
Deploy the new bundle to the QA infrastructure. The post-deployment hook should report no errors.
Check that the test dashboard on the QA Automation node shows no errors.
Deploy the same bundle to the production infrastructure.
Without a failing test, the deployment to the production infrastructure should pass, with the pre-deployment hook succeeding.

Automate the deployment pipeline#

Once you understand how test scenarios can fit into a deployment pipeline, you can automate all steps as your processes mature.

As a demonstration, the project contains a scenario called Automate Deployment that aims to achieve this objective. When executed, this scenario:

Creates a new project bundle and publishes it to the Deployer.
Updates the deployment on the QA infrastructure (which also runs the post-deployment hook to automatically run all test scenarios, and stop the scenario’s execution if one test fails).
Updates the deployment on the production infrastructure (after running the pre-deployment hook, which ensures all test scenarios have been run successfully on the QA automation node).

Tip

If testing this scenario yourself, be sure to update the deployment IDs to match your project key and infrastructure names.

Integrate test scenarios with Govern workflows#

Many organizations also use a Govern node as a key part of their overall MLOps strategy. Test scenarios can be incorporated into Govern workflows.

For example, some organizations may find that the reports generated by test scenarios on a QA infrastructure (as shown above) can be a vital input to deployment sign-off processes occurring in the Govern node.

For an organization keeping a human in the loop of their deployment processes, these kinds of test reports can help build trust and awareness as they try to scale their efforts.