Tutorial | Governance lifecycle

Get started

Once you understand the basics of governing Dataiku items in the Govern node, you’re ready to walk through the entire end-to-end lifecycle of an AI initiative — from the initial ideation phase to the governance of a deployment in a production environment.

Objectives

In this tutorial, you will:

• Explore and qualify an AI initiative in the Govern node before building any Dataiku assets.

• Walk through a sign-off process within a governance workflow, requesting and submitting feedback on a governed child item.

• Demonstrate ways to inject human oversight into deployment processes in Dataiku.

• Govern a deployment in a production environment.

Prerequisites

To reproduce the steps outlined in this tutorial, you will need:

• Dataiku 14.0 or later.

• A Dataiku Govern node connected to a Design node.

• A Full Designer user profile.

• You should be comfortable with all concepts covered in Tutorial | Governance basics.

• To complete the last section, you’ll also need all prerequisites in Tutorial | Batch deployment, including an Automation node connected to your Design node.

Create a governed project

Although the Govern node surfaces Dataiku items, your governance efforts might begin before any development in a connected node. Rather than governing existing Dataiku items, your governance journey might begin directly in the Govern node.

1. Open the Govern node. You can navigate there from the Overview panel of the Cloud Launchpad or, from a Design node, go to the waffle () menu > Dataiku Govern.

2. Go to the Governed projects () page.

3. Click Create.

4. If necessary, select the Dataiku Standard template.

5. Name the governed project as <YOUR INITIALS> Governance Lifecycle.

6. Click Create.

   

7. In the governed project, navigate to the Source Objects panel to confirm that it has no Dataiku source item.

Tip

Hovering over the information () icon on the governed project title also reports the absence of a Dataiku source item.

Use case summary

Governance is a team sport! Many different parties may share responsibilities for an AI initiative over the course of its lifecycle. For example:

• One group may originally sponsor an initiative.

• A second group may build the actual assets, such as a model or bundle.

• A third group may test and ultimately approve or reject it.

• A final group may deploy approved items to production environments.

For the purpose of this tutorial however, you’ll wear all these hats simultaneously. Given your own organizational structures, you’ll need to imagine how to adapt the workflow demonstrated here.

Important

Here we’ll walkthrough the standard workflow. It demonstrates one possible scenario for governing an AI initiative. Rather than focusing on the exact details, try to see how the components used here can meet your own governance needs. Remember, if you have an advanced license, you’ll be able to design your own items, including their workflows.

Explore and qualify initiatives before building

The governance journey demonstrated in Tutorial | Governance basics started by surfacing existing Dataiku items from a connected Design node. However, it’s possible that you may institute a governance framework before building any Dataiku assets. For example, you can use the Govern node to guide exploration and qualification stages to ensure that your team is working on fully validated initiatives.

Add reviewers to a governed project

One important aspect of a governed project is the ability to designate reviewers. You can designate which individuals can provide feedback on an item, or, eventually, approve its status.

1. From your governed project, navigate to the Overview panel.

2. Click Edit in the item header.

3. Scroll down to the Sign-off reviewers and approvers section.

4. Next to Business reviewers, click + Add, and select yourself.

5. Also, next to Final approvers, click + Add, and select yourself.

6. Click Save in the item header.

Validate project plans

It’s possible that the exploration and qualification stages of a governed project occur without any attached Dataiku items. Simulate walking through these stages.

1. Navigate to the Exploration stage of the workflow.

2. Click Edit in the item header.

3. Click Set as Ongoing to begin the Exploration stage.

4. In the notes field, copy-paste the sample below.

   We need to start researching the risk, value, cost, and feasibility of this project.
   

5. Click Set as Finished, thereby advancing to the Qualification stage.

6. Fill out the Qualification stage as much as desired, and then set the Resulting decision field to Go.

7. Click Set as Finished to move to the In Progress stage.

8. Click Save in the item header.

Build Dataiku items in tandem with a governance workflow

Assuming the exploration and qualification stages result in a green-lit project, the next step would be to create a Dataiku project in the Design node. Then, as development progresses in the Design node, you’d manage the corresponding governance workflow stages in the Govern node.

Link a Dataiku project to an existing governed project

In this scenario, a validated governed project would mean starting a blank project. However, for the purpose of this tutorial, you can skip over the actual building stage and just create an existing project. The actual specifics of the Dataiku project are unimportant.

1. Open the home page of a Design node connected to your Govern node.

2. Click + New Project > Learning projects.

3. Search for and select Governance Lifecycle.

4. Click Install, and OK.

5. On the project homepage, click the pencil () next to the title.

6. Rename it <YOUR INITIALS> Governance Lifecycle.

Note

You can also download the starter project from this website and import it as a zip file.

Tip

For convenience, the name of your Dataiku project should match the name of your governed project. It’s fine if they don’t! You’ll just see an information () icon on the title hinting at the Dataiku source project. You can edit the titles at any time as shown in Tutorial | Governance basics.

Once you have a Dataiku project, you can link it to the existing (and validated) governed project.

1. Navigate to the Governable items () page of your Govern node.

2. Find your Governance Lifecycle project in the Projects accordion menu.

3. At the end of its row, click the gavel () to govern it.

4. In the Project definition, choose Select an existing Govern project.

5. Select your governed project. Its name should be <YOUR INITIALS> Governance Lifecycle.

6. For existing items, select the Dataiku Standard template.

7. Do the same for future synced items, overriding the instance rules if necessary.

8. Click Govern.

You should now have a governed project in progress, attached to a Dataiku source project. Confirm these facts first.

1. Open your governed project from the Governed projects () page.

2. Navigate to the Source Objects panel.

3. Confirm the presence of your Dataiku project as the source object.

4. Under Related objects, recognize that it doesn’t have a bundle yet.

Tip

Keep your Design and Govern node projects open in separate tabs to make it easier to switch back and forth throughout this tutorial!

Automatically govern synced Dataiku items

Having confirmed a successful link between the Govern node and your Design node project, now you can simulate the work of building the actual Dataiku project. For example, at some point, the team building the project may package their work as a bundle.

1. Return to your Governance Lifecycle project on the Design node.

2. From the top navigation bar, select More Options () > Bundles.

3. Click + New Bundle.

4. Name it v1.

5. Click Create.

6. After creating the bundle, select it to open it, and review its Governance status.

Your bundle should be synced and governed according to the standard template. However, it’s not yet validated or in production.

Confirm that the Govern node automatically governs the new bundle.

1. Return to (and refresh) the Source Objects panel of the governed project.

2. Expand the Bundles accordion menu.

3. Select the v1 bundle.

4. In the right Details panel, click Open to view the governed bundle.

Sign-off on Dataiku items

You now have a qualified Dataiku project automatically governing new child items, such as project bundles and saved model versions. In this section, you’ll play the roles of both builder and reviewer, switching between the Design and Govern nodes as you advance a project through a governance workflow.

Start the workflow of a governed child item

Your governed project is in progress, but recall that governed child items have their own workflow stages. Let’s walk through the workflow for a project bundle.

1. From the governed bundle, select the Review workflow stage.

2. Click Edit in the item header.

3. Click Set as Ongoing to initiate this step.

4. Click Save in the item header.

Tip

Here we’re demonstrating the standard governance workflow of a project bundle, but the principles are the same for a saved model version.

Request feedback

Recall that when creating the governed project, you added reviewers in different areas. You can use review stages to insert humans in the loop of your governance processes.

For example, once on the Review stage, notice that you can’t set it as finished. It’s impossible to advance the workflow stage without a final approval. Let’s work towards that approval!

1. From within the Review stage of the governed bundle, click Request Feedback.

2. Choose Select users to notify.

3. Display the Business reviewers.

4. Select yourself.

5. Click Request Feedback to complete the dialog.

Submit feedback

Remember that you’re playing all roles in this situation. Having requested feedback, now it’s your turn to submit feedback!

1. From within the Review step of the governed bundle, click Review for the business division.

2. Set the status to Approved.

3. Add the comment LGTM.

4. Click Submit Review.

Reject a bundle

Various teams can provide feedback during the review process, but ultimately one party is responsible for a final approval.

1. From within the Review step of the governed bundle, click Request Final Approval.

2. Choose Select users to notify.

3. Select yourself.

4. Click Request Final Approval once more to complete the dialog.

Now put on the hat of the final reviewer, but reject it for the purpose of demonstration.

1. From within the Review step of the governed bundle, click Review to partake in the final approval process.

2. Set the status to Rejected.

3. Add a comment like Fails to meet requirements.

4. Click Submit Review.

Tip

Confirm for yourself that you’re unable to advance to the next workflow stage with a rejected approval status!

Grant final approval

A reviewer might reject a bundle or model version for many possible reasons. Faced with this rejection, the builders must take any feedback they’ve received and work on a new bundle. First, create a (presumably) better bundle.

1. Return to the Bundles page of the Design node project.

2. Click + New Bundle, name it v2, and click Create.

3. Return to the Govern node, and open the new governed v2 bundle.

Now request another review.

1. Repeat the same process as before, beginning the Review stage for the v2 bundle.

2. In the Review stage, once again click Request Feedback.

3. Select yourself among the Business reviewers.

4. Skip the actual business review this time. Instead, go ahead to click Request Final Approval.

5. In the dialog, choose Select users to notify, and select yourself.

6. Click Request Final Approval.

Although the bundle is exactly the same as the previous one, this time approve it.

1. In the Final approval tile of the governed v2 bundle, click Review.

2. Set the status to Approved.

3. Click Submit Review.

4. Click Edit on the item header of the governed bundle.

   

5. Click Set as Finished on the Review stage, advancing the bundle to the Signed-off stage.

6. Click Save on the item header to confirm.

Tip

Having seen the review process for a bundle, challenge yourself to do the same for a saved model version!

Deploy Dataiku items in tandem with a governance workflow

You now have a bundle in a Design node project that has earned an explicit approval in the Govern node. It’s time to deploy it to a production environment!

Review governance details in the source project

Before deploying, note that the sync between the Govern node and the Design node works both ways. For example, you can find the governance status of child items like bundles and saved model versions in the Dataiku source project.

1. Return to the Bundles page of the Design node project.

2. Open the v2 bundle.

3. In the Governance status, confirm it now reports an approved validation status.

Tip

You’ll find the same governance status details for a saved model version in its Summary panel!

Create and publish a bundle to the Deployer

Other resources provide detailed accounts for deploying project bundles and API services, respectively. For that reason, we won’t cover those processes again here, and instead assume you’re comfortable deploying either kind of package to a production environment.

Important

Remember that governance is a team sport! It’s possible that deployment isn’t your responsibility, and so you won’t have the required permissions to complete the remainder of this tutorial yourself.

If you’re unable to publish to the Deployer, you may need to request permissions from your instance administrator. See Tutorial | Batch deployment for a discussion of prerequisites. Otherwise, read along to follow how you’d govern an actual deployment.

1. In the Design node project, select the v2 bundle.

2. Click Publish on Deployer, and Publish on Deployer again to confirm.

3. Don’t actually deploy it to an infrastructure just yet!

Review your infrastructure’s Govern policy

If you previously deployed a project bundle or API service, as long as you had the correct permissions, you may not have faced any restrictions to deploy your specific bundle or API service.

The Govern node’s tight integration with the Deployer can provide just this type of intervention. In fact, as defined in the reference documentation on Governance features, your organization has three options to define how the Deployer should behave when a user tries to deploy a bundle or an API service:

• Always deploy without checking.

• Warn and ask for confirmation before deploying unapproved packages.

• Prevent the deployment of unapproved packages.

Instance administrators can define this policy with respect to each infrastructure.

1. If you have permission, on the (Project or API) Deployer, navigate to the Infrastructures tab in the top navigation bar.

2. Select an infrastructure.

3. Go to the Settings tab.

4. In the General settings panel, view the selection and options for Dataiku Govern policy, but don’t make any changes.

It’s entirely up to your organization to determine what role the Govern node should (or shouldn’t) play in deployment processes!

To give just one possible example, many organizations designate their infrastructures according to lifecycle stage. For example, they may have one infrastructure for development, another for testing, and a final one for production.

Accordingly, an organization might use all three policies:

• Always deploy without checking on the development infrastructure.

• Warn before deploying unapproved packages on the test infrastructure.

• Prevent the deployment of unapproved packages on the production infrastructure.

See also

The Govern policy isn’t the only way to inject oversight into deployment processes. See the reference documentation on Deployment infrastructures to learn more about topics such as deployment policies and hooks.

Deploy an approved bundle

Your bundle already has received an approval in the Govern node. Therefore, no matter what Govern policy your organization has chosen, there should be no blocker to deploying it.

1. Return to your v2 bundle on the Project Deployer.

2. Complete the usual deployment routine (clicking Deploy, selecting an infrastructure, clicking Create, and finally Deploy and Activate).

3. When it succeeds, navigate to the Status tab of the deployment.

4. Go to the Health panel to review the governance status. You may need to refresh the page after a few minutes for it to register.

Govern a bundle in production

You now have an approved bundle in production! That’s a big step, but it’s not the end of the governance story.

Complete the following three tasks in the Govern node on your own.

1. In the Bundle registry (), filter for deployments (), and confirm your bundle remains.

2. Advance the governed bundle from the Signed-off stage to Production.

3. Advance the governed parent project from the In Progress stage to Validation and Roll-out.

See also

Here you have to manually move the governed bundle from the Signed-off stage to Production. However, it’s possible you could automate this with a deployment hook. Learn more in the reference documentation on Deployment infrastructures.

Govern the Automation node bundle as a related project

Recall that the Govern node syncs Dataiku items from all connected nodes. That includes Automation nodes! Therefore, after deploying a bundle, you should now have an ungoverned Automation node project. You can also group this project into the existing governed project, along with its Design node sibling.

1. Return to the Governable items () page of the Govern node.

2. Find your Automation node project. Unless changed, it has the same title as the Design node project. However, it will have a different instance node ID.

3. Click the gavel () to govern it.

4. For the Project definition, confirm you’ve chosen Select a related Govern Project.

5. Select your Governance Lifecycle project as that related project.

6. Make any desired template selections for existing and synced items.

7. Click Govern.

   

8. Find your governed project in the Governed Projects () page.

9. Note the presence of a 2 next to the title, indicating the governed project has two Dataiku source projects: one from the Design node and one from the Automation node.

Tip

Feel free to continue experimenting. For example, walk through the same procedures using a saved model version instead of a bundle!

Next steps

Congratulations! You’ve now seen one complete lifecycle of an AI initiative, from its initial exploration as a governed project to an approved deployment in a production environment.

Before you finish, take a moment to delete work that you no longer need as a courtesy to those sharing your Govern node.

1. From the Governed projects () page, open your project. From the vertical dots () menu, click Delete Govern Project.

2. From the Design node, open your project. From the project homepage, click Actions > Delete this project.

3. From the waffle () menu of your Design node, go to Local/Remote Deployer > Deploying Projects. Select your deployment.

4. From the deployment page, open the Automation Project in a new tab. From the Automation project’s homepage, click Actions > Delete this project.

5. From the deployment page, click Actions > Delete this deployment.

This was only one possible workflow: the standard workflow. Advanced license holders have the ability to create their own workflows, including review stages as needed. You’ll learn more about customization in the Academy course on Advanced Dataiku Govern.

See also

For more information, see the reference documentation on Governance.