Tutorial | Governance lifecycle#

Get started#

Once you understand the basics of the Govern node, you’re ready to walk through the governance of the an end-to-end lifecycle of an AI initiative — from the initial ideation phase to a deployment in a production environment.

Objectives#

In this tutorial, you will:

Explore and qualify an AI initiative in the Govern node before building any Dataiku assets.
Walk through a sign-off process within a governance workflow, requesting and submitting feedback on a governed child item.
Inject human oversight into deployment processes for more secure MLOps.

Prerequisites#

To reproduce the steps outlined in this tutorial, you will need:

Dataiku 14.0 or later.
A Dataiku Govern node connected to a Design node.
You should be comfortable with all concepts covered in Tutorial | Governance basics.
To complete the last section, you’ll also need all prerequisites in Tutorial | Deploy a project bundle to a production environment, including an Automation node connected to your Design node.

Caution

By its nature, effective AI governance requires close collaboration between many individuals of diverse responsibilities. Completing governance tutorials may require you to perform actions outside your own day-to-day activities. If you find yourself lacking the permissions to complete certain actions, reach out to your instance administrator. Otherwise, read along to simulate the suggested steps.

Create a governed project#

Although the Govern node surfaces Dataiku items, your governance efforts might begin before any development in a connected Design node. Rather than governing existing Dataiku items, your governance journey might begin in the Govern node.

Open the Govern node. You can navigate there from the Overview panel of the Cloud Launchpad or, from a Design node, go to the waffle () menu > Dataiku Govern.
Go to the Governed projects () page.
Click Create.
If necessary, select the Dataiku Standard template.
Name the governed project as <YOUR INITIALS> Governance Lifecycle.
Click Create.
In the governed project, navigate to the Source Objects panel to confirm that it has no Dataiku source item.

Tip

Hovering over the information () icon on the governed project title also reports the absence of a Dataiku source item.

Use case summary#

Governance is a team sport! Many different parties may share responsibilities for an AI initiative over the course of its lifecycle. For example:

One group may originally sponsor an initiative.
A second group may build the actual assets, such as a bundle or model.
A third group may approve or reject those assets.
A final group may deploy approved assets to production environments.

For the purpose of this tutorial however, you’ll switch between all these hats. Given your own organizational structures, you’ll need to imagine how to adapt the workflow demonstrated here.

Important

Here you’ll move through the standard workflow. It demonstrates one possible scenario for governing an AI initiative. Rather than focusing on the exact details, try to see how the components used here can meet your own governance needs. Remember, organizations with advanced licenses, can design their own items, including their own workflows.

Explore and qualify initiatives before building#

The governance journey demonstrated in Tutorial | Governance basics started by surfacing existing Dataiku items from a connected Design node.

However, it’s possible that you may institute a governance framework before building any Dataiku assets. For example, you can use the Govern node to guide exploration and qualification stages to ensure that your team is working on fully validated initiatives.

Add reviewers to a governed project#

One important component of a standard Govern project is the space to designate reviewers. You can designate which groups or individuals can provide feedback on an item, or, eventually, approve it.

From your governed project, navigate to the Overview panel.
Click Edit in the item header.
Scroll down to the Sign-off reviewers and approvers section.
Next to Business reviewers, click + Add, and select yourself.
Also, next to Final approvers, click + Add, and select yourself.
Click Save in the item header.

Validate project plans#

It’s possible that the exploration and qualification steps of a Govern project occur without any connected Dataiku source items. Simulate walking through these steps.

Navigate to the Exploration step of the workflow.
Click Edit in the item header.
Click Set as Ongoing to begin the Exploration step.

In the notes field, copy-paste the sample below.

We need to start researching the risk, value, cost, and feasibility of this project.

Click Set as Finished, thereby advancing to the Qualification step.
Fill out the Qualification step as much as desired, and then set the Resulting decision field to Go.
Click Set as Finished to move to the In Progress step.
Click Save in the item header.

Build Dataiku items in tandem with a governance workflow#

Having completed the exploration and qualification steps, you’ve just green-lit the start of a new project. The next step would be to create a Dataiku project in the Design node linked to the existing Govern project. Then, as development progresses in the Design node, you’d manage the corresponding governance workflow steps in the Govern node.

Link a Dataiku project to an existing Govern project#

In this scenario, a validated Govern project would mean starting a blank Dataiku project. However, for the purpose of this tutorial, you can skip over the actual building stage and just create an existing project. The actual specifics of the Dataiku project are unimportant.

Open the homepage of a Design node connected to your Govern node.
Click + New Project > Learning projects.
Search for and select Governance Lifecycle.
Click Install, and OK.
On the project homepage, click the pencil () next to the title.
Rename it <YOUR INITIALS> Governance Lifecycle.

Note

You can also download the starter project from this website and import it as a zip file.

Tip

For convenience, the name of your Dataiku project should match the name of your Govern project. It’s fine if they don’t! You’ll just see an information () icon on the title hinting at the Dataiku source project. You can edit the titles at any time as shown in Tutorial | Governance basics.

Once you have a Dataiku project, you can link it to the existing (and validated) Govern project.

Navigate to the Governable items () page of your Govern node.
Find your Governance Lifecycle project in the Projects accordion menu.
At the end of its row, click the gavel () to govern it.
In the Project definition, choose Select an existing Govern project.
Select your governed project. Its name should be <YOUR INITIALS> Governance Lifecycle.
For existing items, select the Dataiku Standard template.
Do the same for future synced items, overriding the instance rules if necessary.
Click Govern.

You should now have a Govern project in progress, attached to a Dataiku source project. Confirm these facts first.

Open your governed project from the Governed projects () page.
Navigate to the Source Objects panel.
Confirm the presence of your Dataiku project as the source object.
Under Related objects, recognize that it doesn’t have a bundle yet.

Tip

Keep your Dataiku and Govern projects open in separate tabs to make it easier to switch back and forth throughout this tutorial!

Automatically govern synced Dataiku items#

Having confirmed a successful link between the Govern node and your Design node project, now you can simulate the work of building the actual Dataiku project. The final aim of the project could be to create an output such as a dashboard, Dataiku app, or a webapp.

Often though, the most important output from a production standpoint could be a bundle or a model. For example, at some point, the team building the project may package their work as a bundle.

Return to your Governance Lifecycle project on the Design node.
From the top navigation bar, select More Options () > Bundles.
Click + New Bundle.
Name it v1.
Click Create.
After creating the bundle, select it to open it, and review its Governance status.

Your bundle should be synced and governed according to the standard template. However, it’s not yet validated or in production.

Confirm that the Govern node automatically governs the new bundle.

Return to (and refresh) the Source Objects panel of the Govern project.
Expand the Bundles accordion menu.
Select the v1 bundle.
In the right Details panel, click Open to view the Govern bundle.

Sign-off on Dataiku items#

You now have a qualified Govern project automatically governing new child items, such as project bundles and saved model versions. In this section, you’ll play the roles of both builder and reviewer, switching between the Design and Govern nodes as you advance a project through a governance workflow.

Start the workflow of a governed child item#

Your Govern project is in progress, but recall that governed child items have their own workflow steps. In particular, the workflow of Govern bundles and Govern model versions, according to the standard template, includes a step with an attached sign-off.

Next, walk through the workflow for a project bundle to see an example of a sign-off in action.

From the governed bundle, select the Review workflow step.
Click Edit in the item header.
Click Set as Ongoing to initiate this step.
Click Save in the item header.

Tip

Here we’re demonstrating the standard governance workflow of a project bundle, but the principles are the same for a saved model version.

Request feedback#

Recall that after creating the governed project, you added reviewers in different areas. You can use review steps to insert humans in the loop of your governance processes.

For example, once on the Review step, notice that you can’t set it as finished. It’s impossible to advance the workflow step without a final approval. Work towards that approval!

From within the Review step of the governed bundle, click Request Feedback.
Choose Select users to notify.
Display the Business reviewers.
Select yourself.
Click Request Feedback to complete the dialog.

Submit feedback#

Remember that you’re playing all roles in this situation. Having requested feedback, now it’s your turn to submit feedback!

From within the Review step of the governed bundle, click Review for the business division.
Set the status to Approved.
Add the comment LGTM.
Click Submit Review.

Reject a bundle#

Various teams can provide feedback during the review process, but one party is responsible for a final approval, according to the standard template.

From within the Review step of the governed bundle, click Request Final Approval.
Choose Select users to notify.
Select yourself.
Click Request Final Approval once more to complete the dialog.

Now put on the hat of the final reviewer, but reject it for the purpose of demonstration.

From within the Review step of the governed bundle, click Review to partake in the final approval process.
Set the status to Rejected.
Add a comment like Fails to meet requirements.
Click Submit Review.

Tip

Confirm for yourself that you’re unable to advance to the next workflow step with a rejected approval status!

Grant final approval#

A reviewer might reject a bundle or model version for many possible reasons. Faced with this rejection, the builders must take any feedback they’ve received and work on a new bundle. First, create a (presumably) better bundle.

Return to the Bundles page of the Design node project.
Click + New Bundle, name it v2, and click Create.
Return to the Govern node, and open the new governed v2 bundle.

Now request another review.

Repeat the same process as before, beginning the Review step for the v2 bundle.
In the Review step, once again click Request Feedback.
Select yourself among the Business reviewers.
Skip the actual business review this time. Instead, go ahead to click Request Final Approval.
In the dialog, choose Select users to notify, and select yourself.
Click Request Final Approval.

Although the bundle is exactly the same as the previous one, this time approve it.

In the Final approval tile of the governed v2 bundle, click Review.
Set the status to Approved.
Click Submit Review.
Click Edit on the item header of the governed bundle.
Click Set as Finished on the Review step, advancing the bundle to the Signed-off step.
Click Save on the item header to confirm.

Tip

Having seen the review process for a bundle, challenge yourself to do the same for a saved model version!

Deploy Dataiku items in tandem with a governance workflow#

You now have a bundle in a Design node project that has earned an explicit approval in the Govern node. It’s time to deploy it to a production environment!

Review governance details in the source project#

Before deploying, note that the sync between the Govern node and the Design node works both ways. For example, you can find the governance status of child items like bundles and saved model versions in the Dataiku source project.

Return to the Bundles page of the Design node project.
Open the v2 bundle.
In the Governance status, confirm it now reports an approved validation status.

Tip

You’ll find the same governance status details for a saved model version in its Summary panel!

Publish a bundle to the Deployer#

Other resources provide detailed accounts for deploying project bundles and API services, respectively. For that reason, we won’t cover those processes again here, and instead assume you’re comfortable deploying either kind of package to a production environment.

Caution

Remember that governance is a team sport! It’s possible that deployment isn’t your responsibility, and so you won’t have the required permissions to complete the remainder of this tutorial yourself.

If you’re unable to publish to the Deployer, you may need to request permissions from your instance administrator. See Tutorial | Deploy a project bundle to a production environment for a discussion of prerequisites. Otherwise, read along to follow how you’d govern an actual deployment.

In the Design node project, select the v2 bundle.
Click Publish on Deployer, and Publish on Deployer again to confirm.
Don’t actually deploy it to an infrastructure just yet!

Review your infrastructure’s Govern policy#

If you previously deployed a project bundle or API service, as long as you had the correct permissions, you may not have faced any restrictions to deploy your specific bundle or API service.

The Govern node’s tight integration with the Deployer can provide just this type of intervention. In fact, as defined in the reference documentation on Governance features, your organization has three options to define how the Deployer should behave when a user tries to deploy a bundle or an API service:

Always deploy without checking.
Warn and ask for confirmation before deploying unapproved packages.
Prevent the deployment of unapproved packages.

Instance administrators can define this policy with respect to each infrastructure.

If you have permission, on the (Project or API) Deployer, navigate to the Infrastructures tab in the top navigation bar.
Select an infrastructure.
Go to the Settings tab.
In the General settings panel, view the selection and options for Dataiku Govern policy, but don’t make any changes.

It’s entirely up to your organization to determine what role the Govern node should (or shouldn’t) play in deployment processes!

To give just one possible example, many organizations designate their infrastructures according to lifecycle stage. For example, they may have one infrastructure for development, another for testing, and a final one for production.

Accordingly, an organization might use all three policies:

Always deploy without checking on the development infrastructure.
Warn before deploying unapproved packages on the test infrastructure.
Prevent the deployment of unapproved packages on the production infrastructure.

Deploy an approved bundle#

Your bundle already has received an approval in the Govern node. Therefore, no matter what Govern policy your organization has chosen, there should be no blocker to deploying it.

Return to your v2 bundle on the Project Deployer.
Complete the usual deployment routine (clicking Deploy, selecting an infrastructure, clicking Create, and finally Deploy and Activate).
When it succeeds, navigate to the Status tab of the deployment.
Go to the Health panel to review the governance status. You may need to refresh the page after a few minutes for it to register.

Govern a bundle in production#

You now have an approved bundle in production! That’s a big step, but it’s not the end of the governance story.

Complete the following three tasks in the Govern node on your own.

In the Bundle registry (), filter for deployments (), and confirm your bundle remains.
Advance the governed bundle from the Signed-off stage to Production.
Advance the governed parent project from the In Progress stage to Validation and Roll-out.

Next steps#

Congratulations! You’ve now seen one complete lifecycle of an AI initiative, from its initial exploration to an approved deployment in a production environment.

Before you finish, take a moment to delete work that you no longer need as a courtesy to those sharing your Govern node.

From the Governed projects () page, open your project. From the vertical dots () menu, click Delete Govern Project.
From the Design node, open your project. From the project homepage, click Actions > Delete this project.
From the waffle () menu of your Design node, go to Local/Remote Deployer > Deploying Projects. Select your deployment.
From the deployment page, open the Automation Project in a new tab. From the Automation project’s homepage, click Actions > Delete this project.
From the deployment page, click Actions > Delete this deployment.

This was only one possible workflow: the standard workflow. Advanced license holders have the ability to create their own workflows, including review stages as needed. You’ll learn more about customization in the Academy course on Advanced Dataiku Govern.