Tutorial | Implement pre-deployment checks with Project Standards#

Get started#

For users in many enterprises, building an AI project that first adds value, and is also deployment-ready, is a significant challenge. To name just a few:

Those building projects might be unclear as to what constitutes a deployment-ready project and face late rejections when seeking to move to a production environment.
Those reviewing projects may find themselves overwhelmed with the complexity and responsibility for approving the deployment of a new AI initiative.
Those maintaining deployment infrastructure have a sincere need to mitigate risk, particularly as the number of deployments scale.

For organizations on the Dataiku for Enterprise AI package, Project Standards is a way to address the challenge of scaling high-quality deployments. These organizations can define their own pre-deployment quality checks, which project designers, reviewers, and managers can then use to smooth the deployment journey.

Objectives#

Utilizing Project Standards incorporates many different roles across an organization. In this tutorial, you will switch hats between:

Instance administrators who define the pre-deployment quality checks (the actual project standards) and the scopes which determine where they should be applied.
Flow designers who run the report, which includes these quality checks, and then act on its findings to accelerate their path to deployment.
Ops managers who set policies to gate deployments with respect to the results of this report.

Prerequisites#

Dataiku 14.1 or later.
The Dataiku for Enterprise AI package.
Membership in a group with the Administrator permission.
For the last section, the ability to deploy a bundle to an Automation node. See the prerequisites in Tutorial | Deploy a project bundle to a production environment for details.

Tip

If you lack any of the required permissions above, work through the sections relevant to your role.

Create the project#

From the Dataiku Design homepage, click + New Project.
Select Learning projects.
Search for and select Project Standards.
If needed, change the folder into which the project will be installed, and click Install.
From the project homepage, click Go to Flow (or type g + f).

Note

You can also download the starter project from this website and import it as a zip file.

Create instance-wide project quality checks#

Your organization may already have its own unwritten production guidelines. Project Standards provides an avenue to codify such conventions. To get started, an instance administrator needs to design and enable pre-deployment quality checks relevant to their organization.

Add a pre-deployment quality check#

Start with the most basic kind of check, the default checks available to all instances.

From the waffle () menu of the top navigation bar, select Administration.
Go to the Project Standards tab.
Within the Checks library panel, click + Add Checks.
For the source plugin, select Project Standards Built-in Checks.
For checks, click Select All to add all available built-in checks.
Click Add to confirm.

Tip

These default checks are generic demonstrations. For Project Standards to play a meaningful role in your deployment processes, you’ll want to design your own or work with Dataiku’s data science teams to import commonly used checks.

Edit quality checks#

After adding the built-in checks, you can customize them further. Certain checks may have parameters you can configure based on your needs.

In the Checks library panel, select the first check you’ve added named Project has an acceptable number of datasets. It should have 0 scopes.
To more easily identify it as your own, as well as to force an error, select it, and rename it <YOUR INITIALS>: Project has a max of 2 datasets.
Under Check parameters, set the maximum number of datasets to 2.
Click Save.
Repeat this kind of process for the other checks on descriptions, Flow zones, and test scenarios. Recognize which have parameters, and insert your initials into the name of each check.

Explore the source of a check#

All checks come from plugins. This could be the built-in checks plugin found on all instances, those built by Dataiku, or ones specific to your organization.

From the Checks library panel, for any of the checks you’ve added, click Project Standards Built-in Checks in the Source column.
Explore the components and documentation for this plugin.

Tip

See Dataiku DSS - Developer Guide for how to get started working with code in Dataiku.

Define a scope#

The checks you just created aren’t yet applied to any projects. The checks library should report that your checks are used in 0 scopes.

The scope defines which checks apply to which projects. You can select projects by name, folder, or tag. For example, you may have Dataiku projects with an academy tag. You could create a scope to apply specific checks to only Dataiku projects with this specific tag.

In this first example, make the scope even more narrow. Apply it to one specific project name.

From the Project Standards tab of the Administration page, go to the Scopes panel.
Click + Add Scope.
Name it <YOUR-INITIALS>-test-scope.
Give any description such as A scope for testing out Project Standards.
Select projects by Name.
Select your version of the Project Standards Dataiku project.
Select all four checks belonging to you.
Click Save.

Tip

If you return to the Checks library panel, you should see your checks now used in 1 scope.

Run project quality checks before deploying#

Now you’ve set up some basic checks for your organization. Take a moment to see how they work from the perspective of project designers who will use such checks.

Create a Project Standards report#

To test if their current project meets the deployment standards defined by their organization, users can build the Project Standards report at any time from the Flow.

From your version of the Project Standards project, go to the Flow (g + f`).
Click Flow Actions > Check Project Standards.
Click Run Project Standards.
Explore the checks you need to review and those that succeeded.
Close the window when finished reviewing.

Tip

These simple built-in checks are only of medium severity. For your own checks, you’ll be able to define statuses ranging from lowest to critical, should the checks be violated.

Act on a project standard check#

Users can follow the results of the Project Standards report to smooth their deployment journey. For example, according to the report, one check to review was the presence of a test scenario.

Add an empty test scenario to demonstrate how to move a check into the success state.

From the Jobs () menu of the top navigation bar, select Scenarios.
Click + New Scenario.
Name it Test Scenario.
Click Create.
Within the Settings tab of the scenario, check the box to Mark as a test scenario.
Click Save.

The Project Standards report is in fact a report. It doesn’t recompute automatically.

Return to the Flow.
Go to Flow Actions > Check Project Standards, and confirm that the test scenario check still needs review.
Click Rerun to generate a new report.
Confirm the test scenario check moves into the Success tab.

Create a bundle#

Normally, users will want to have passed all checks before trying to deploy. In this case, you’ll use the checks still needing review to demonstrate how to gate deployment.

From the More Options () menu of the top navigation bar, select Bundles.
Click + New Bundle.
Name it v1.
Click Create.
Click on the name of the v1 bundle to view its contents.
Review the Project Standards tab of the bundle details.

Tip

The status of the Project Standards checks shown here are “bundled” with the bundle. Both represent a snapshot. If you were to make changes to the project, existing bundles wouldn’t include those changes. Similarly, the Project Standards report would also not reflect any new changes.

Use project quality checks to gate deployment#

You can use sign-offs in the Govern node to warn about or even prevent unapproved deployments. You might consider that a human-in-the-loop answer to the problem of deployment quality assurance.

Alongside those hard-working humans, you can use the Project Standards report to automatically enforce your organization’s quality checks.

Set a deployment control policy#

Similar to the Govern policies for deployment, you define how permissive or restrictive you wish to be with respect to Project Standards, according to each infrastructure.

For example, consider a user that tries to deploy a project with unresolved issues in its Project Standards report. Their ability to deploy can depend on:

The severity of the issue — You might prevent a deployment with critical issues, but only present a warning for those with lesser violations.
The infrastructure — You may have a test infrastructure that produces only a warning and a production infrastructure that blocks the deployment entirely.

In this case, set the deployment control to warn users before deploying for any reported issues from quality checks in the Project Standards report.

From the waffle () menu of the top navigation bar, select Local/Remote Deployer (depending on your instance’s setup).
Select Deploying Projects.
From the top navigation bar, select Infrastructures.
Select an infrastructure where you can deploy a project.
Navigate to the Settings tab for that infrastructure.
Go to the Deployment policies panel.
Under Project Standards control, select Warn as the deployment control.
Leave the Maximum severity allowed to No issues.
Click Save.

Tip

It’s up to your organization to find the correct balance between restrictive and permissive deployment controls. As you begin implementing Project Standards, consider taking a more permissive approach initially. Then, gradually tighten controls as you find the quality checks most important for your organization and the scale of deployments begins to increase.

Attempt to deploy a bundle with unresolved quality checks#

Having set up this policy, try to deploy your bundle, knowing it has unresolved quality checks. Follow the routine deployment process:

Return to the Bundles page of your project on the Design node.
Select your bundle, and click Publish on Deployer. Click Publish on Deployer again to confirm.
Click Open in Deployer.
Review the Project Standards section of the bundle on the Deployer.

Despite a report including an issue of medium severity, proceed with deploying the bundle.

On the bundle found on the Deployer, click Deploy.
For the Target infrastructure field, select the infrastructure on which you set the deployment control, and immediately see a warning flag.
Ignoring it, click Create.
Click Deploy and Activate.
Confirm you receive a warning message that a Project Standards check is above the allowed severity.
Click Cancel to abandon the deployment attempt.

Tip

Project Standards are also surfaced in the Govern node. In a page such as Governable items or the Bundle registry, go to this bundle’s Source Objects () tab of the right panel. You’ll find the same results from the Project Standards report.

Next steps#

Congratulations! You should now have an understanding of how your organization can implement pre-deployment quality checks, giving users a clearer path to successful deployments.

Before you finish, take a moment to delete work that you no longer need as a courtesy to those sharing your instance.

Within the Project Standards tab of the Administration page, return to the Scopes panel.
Click the bin () next to your scope to delete it.
Navigate to the Checks library panel.
Search for your checks, and delete them as well.
Return to the infrastructure settings on the Deployer, and revert the deployment control settings.
On the Deployer, delete your project and bundle.

Tip

To start using Project Standards in real projects, you’ll need to go beyond the simple checks shown here. Reach out to Dataiku’s data science account teams to learn how to import existing and/or design your own custom quality checks.