Tutorial | Governance lifecycle#

Get started#

Once you understand the basics of governing Dataiku items in the Govern node, you’re ready to walk through the entire end-to-end lifecycle of an AI initiative — from the initial ideation phase to the governance of a deployment in a production environment.

Objectives#

In this tutorial, you will:

  • Explore and qualify an AI initiative in the Govern node before building any Dataiku assets.

  • Walk through a sign-off process within a governance workflow, requesting and submitting feedback on a governed child item.

  • Demonstrate ways to inject human oversight into deployment processes in Dataiku.

  • Govern a deployment in a production environment.

Prerequisites#

To reproduce the steps outlined in this tutorial, you will need:

  • Dataiku 14.0 or later.

  • A Dataiku Govern node connected to a Design node.

  • A Full Designer user profile.

  • All prerequisites in Tutorial | Batch deployment, including an Automation node connected to your Design node.

  • You should be comfortable with all concepts covered in Tutorial | Governance basics.

Create a governed project#

Although the Govern node surfaces Dataiku items, your governance efforts might begin before any development in a connected node. Rather than governing existing Dataiku items, your governance journey might begin directly in the Govern node.

  1. Open the Govern node. You can navigate there from the Overview panel of the Cloud Launchpad or, from a Design node, go to the waffle (Waffle icon.) menu > Dataiku Govern.

  2. Go to the Governed projects (Governed projects icon.) page.

  3. Click Create.

  4. If necessary, select the Dataiku Standard template.

  5. Name the governed project as <YOUR INITIALS> Governance Lifecycle.

  6. Click Create.

    Dataiku screenshot of the dialog to create a Govern project.
  7. In the governed project, navigate to the Source Objects panel to confirm that it has no Dataiku source item.

Tip

Hovering over the information (Information icon.) icon on the governed project title also reports the absence of a Dataiku source item.

Use case summary#

Governance is a team sport! Many different parties may share responsibilities for an AI initiative over the course of its lifecycle. For example:

  • One group may originally sponsor an initiative.

  • A second group may build the actual assets, such as a model or bundle.

  • A third group may test and ultimately approve or reject it.

  • A final group may deploy approved items to production environments.

For the purpose of this tutorial however, you’ll wear all these hats simultaneously. Given your own organizational structures, you’ll need to imagine how to adapt the workflow demonstrated here.

Important

Here we’ll walkthrough the standard workflow. It demonstrates one possible scenario for governing an AI initiative. Rather than focusing on the exact details, try to see how the components used here can meet your own governance needs. Remember, if you have an advanced license, you’ll be able to design your own items, including their workflows.

Explore and qualify initiatives before building#

The governance journey demonstrated in Tutorial | Governance basics started by surfacing existing Dataiku items from a connected Design node. However, it’s possible that you may institute a governance framework before building any Dataiku assets. For example, you can use the Govern node to guide exploration and qualification stages to ensure that your team is working on fully validated initiatives.

Add reviewers to a governed project#

One important aspect of a governed project is the ability to designate reviewers. You can designate which individuals can provide feedback on an item, or, eventually, approve its status.

  1. From your governed project, navigate to the Overview panel.

  2. Click Edit in the item header.

  3. Scroll down to the Sign-off reviewers and approvers section.

  4. Next to Business reviewers, click + Add, and select yourself.

  5. Also, next to Final approvers, click + Add, and select yourself.

  6. Click Save in the item header.

Dataiku screenshot of the sign-off reviewers section of a governed project.

Validate project plans#

It’s possible that the exploration and qualification stages of a governed project occur without any attached Dataiku items. Simulate walking through these stages.

  1. Navigate to the Exploration stage of the workflow.

  2. Click Edit in the item header.

  3. Click Set as Ongoing to begin the Exploration stage.

  4. In the notes field, copy-paste the sample below.

    We need to start researching the risk, value, cost, and feasibility of this project.
    
  5. Click Set as Finished, thereby advancing to the Qualification stage.

  6. Fill out the Qualification stage as much as desired, and then set the Resulting decision field to Go.

  7. Click Set as Finished to move to the In Progress stage.

  8. Click Save in the item header.

Dataiku screenshot of a qualified project.

Build Dataiku items in tandem with a governance workflow#

Assuming the exploration and qualification stages result in a green-lit project, the next step would be to create a Dataiku project in the Design node. Then, as development progresses in the Design node, you’d manage the corresponding governance workflow stages in the Govern node.

Automatically govern synced Dataiku items#

Having confirmed a successful link between the Govern node and your Design node project, now you can simulate the work of building the actual Dataiku project. For example, at some point, the team building the project may package their work as a bundle.

  1. Return to your Governance Lifecycle project on the Design node.

  2. From the top navigation bar, select More Options (Horizontal dots icon.) > Bundles.

  3. Click + New Bundle.

  4. Name it v1.

  5. Click Create.

  6. After creating the bundle, select it to open it, and review its Governance status.

Dataiku screenshot of the governance status of a bundle.

Your bundle should be synced and governed according to the standard template. However, it’s not yet validated or in production.

Confirm that the Govern node automatically governs the new bundle.

  1. Return to (and refresh) the Source Objects panel of the governed project.

  2. Expand the Bundles accordion menu.

  3. Select the v1 bundle.

  4. In the right Details panel, click Open to view the governed bundle.

Dataiku screenshot of the Source Objects panel of a governed project.

Sign-off on Dataiku items#

You now have a qualified Dataiku project automatically governing new child items, such as project bundles and saved model versions. In this section, you’ll play the roles of both builder and reviewer, switching between the Design and Govern nodes as you advance a project through a governance workflow.

Start the workflow of a governed child item#

Your governed project is in progress, but recall that governed child items have their own workflow stages. Let’s walk through the workflow for a project bundle.

  1. From the governed bundle, select the Review workflow stage.

  2. Click Edit in the item header.

  3. Click Set as Ongoing to initiate this step.

  4. Click Save in the item header.

Dataiku screenshot of a bundle workflow.

Tip

Here we’re demonstrating the standard governance workflow of a project bundle, but the principles are the same for a saved model version.

Request feedback#

Recall that when creating the governed project, you added reviewers in different areas. You can use review stages to insert humans in the loop of your governance processes.

For example, once on the Review stage, notice that you can’t set it as finished. It’s impossible to advance the workflow stage without a final approval. Let’s work towards that approval!

  1. From within the Review stage of the governed bundle, click Request Feedback.

  2. Choose Select users to notify.

  3. Display the Business reviewers.

  4. Select yourself.

  5. Click Request Feedback to complete the dialog.

Dataiku screenshot of the dialog to request feedback on a bundle.

Submit feedback#

Remember that you’re playing all roles in this situation. Having requested feedback, now it’s your turn to submit feedback!

  1. From within the Review step of the governed bundle, click Review for the business division.

  2. Set the status to Approved.

  3. Add the comment LGTM.

  4. Click Submit Review.

Dataiku screenshot of the dialog for submitting a review.

Reject a bundle#

Various teams can provide feedback during the review process, but ultimately one party is responsible for a final approval.

  1. From within the Review step of the governed bundle, click Request Final Approval.

  2. Choose Select users to notify.

  3. Select yourself.

  4. Click Request Final Approval once more to complete the dialog.

Dataiku screenshot of the dialog to request final approval.

Now put on the hat of the final reviewer, but reject it for the purpose of demonstration.

  1. From within the Review step of the governed bundle, click Review to partake in the final approval process.

  2. Set the status to Rejected.

  3. Add a comment like Fails to meet requirements.

  4. Click Submit Review.

Dataiku screenshot of a rejected final review of a bundle.

Tip

Confirm for yourself that you’re unable to advance to the next workflow stage with a rejected approval status!

Grant final approval#

A reviewer might reject a bundle or model version for many possible reasons. Faced with this rejection, the builders must take any feedback they’ve received and work on a new bundle. First, create a (presumably) better bundle.

  1. Return to the Bundles page of the Design node project.

  2. Click + New Bundle, name it v2, and click Create.

  3. Return to the Govern node, and open the new governed v2 bundle.

Now request another review.

  1. Repeat the same process as before, beginning the Review stage for the v2 bundle.

  2. In the Review stage, once again click Request Feedback.

  3. Select yourself among the Business reviewers.

  4. Skip the actual business review this time. Instead, go ahead to click Request Final Approval.

  5. In the dialog, choose Select users to notify, and select yourself.

  6. Click Request Final Approval.

Dataiku screenshot of the dialog to request final approval.

Although the bundle is exactly the same as the previous one, this time approve it.

  1. In the Final approval tile of the governed v2 bundle, click Review.

  2. Set the status to Approved.

  3. Click Submit Review.

  4. Click Edit on the item header of the governed bundle.

    Dataiku screenshot of an approved bundle in the Signed-off stage.
  5. Click Set as Finished on the Review stage, advancing the bundle to the Signed-off stage.

  6. Click Save on the item header to confirm.

Tip

Having seen the review process for a bundle, challenge yourself to do the same for a saved model version!

Deploy Dataiku items in tandem with a governance workflow#

You now have a bundle in a Design node project that has earned an explicit approval in the Govern node. It’s time to deploy it to a production environment!

Review governance details in the source project#

Before deploying, note that the sync between the Govern node and the Design node works both ways. For example, you can find the governance status of child items like bundles and saved model versions in the Dataiku source project.

  1. Return to the Bundles page of the Design node project.

  2. Open the v2 bundle.

  3. In the Governance status, confirm it now reports an approved validation status.

Dataiku screenshot of the governance status of a validated bundle.

Tip

You’ll find the same governance status details for a saved model version in its Summary panel!

Create and publish a bundle to the Deployer#

Other resources provide detailed accounts for deploying project bundles and API services, respectively. For that reason, we won’t cover those processes again here, and instead assume you’re comfortable deploying either kind of package to a production environment.

  1. In the Design node project, select the v2 bundle.

  2. Click Publish on Deployer, and Publish on Deployer again to confirm.

  3. Don’t actually deploy it to an infrastructure just yet!

Tip

If you’re unable to publish to the Deployer, you may need to request permissions from your instance administrator. See Tutorial | Batch deployment for a discussion of prerequisites.

Review your infrastructure’s Govern policy#

If you previously deployed a project bundle or API service, as long as you had the correct permissions, you may not have faced any restrictions to deploy your specific bundle or API service.

The Govern node’s tight integration with the Deployer can provide just this type of intervention. In fact, as defined in the reference documentation on Governance features, your organization has three options to define how the Deployer should behave when a user tries to deploy a bundle or an API service:

  • Always deploy without checking.

  • Warn and ask for confirmation before deploying unapproved packages.

  • Prevent the deployment of unapproved packages.

Instance administrators can define this policy with respect to each infrastructure.

  1. If you have permission, on the (Project or API) Deployer, navigate to the Infrastructures tab in the top navigation bar.

  2. Select an infrastructure.

  3. Go to the Settings tab.

  4. In the General settings panel, view the selection and options for Dataiku Govern policy, but don’t make any changes.

Dataiku screenshot of the general settings of an infrastructure.

It’s entirely up to your organization to determine what role the Govern node should (or shouldn’t) play in deployment processes!

To give just one possible example, many organizations designate their infrastructures according to lifecycle stage. For example, they may have one infrastructure for development, another for testing, and a final one for production.

Accordingly, an organization might use all three policies:

  • Always deploy without checking on the development infrastructure.

  • Warn before deploying unapproved packages on the test infrastructure.

  • Prevent the deployment of unapproved packages on the production infrastructure.

See also

The Govern policy isn’t the only way to inject oversight into deployment processes. See the reference documentation on Deployment infrastructures to learn more about topics such as deployment policies and hooks.

Deploy an approved bundle#

Your bundle already has received an approval in the Govern node. Therefore, no matter what Govern policy your organization has chosen, there should be no blocker to deploying it.

  1. Return to your v2 bundle on the Project Deployer.

  2. Complete the usual deployment routine (clicking Deploy, selecting an infrastructure, clicking Create, and finally Deploy and Activate).

  3. When it succeeds, navigate to the Status tab of the deployment.

  4. Go to the Health panel to review the governance status. You may need to refresh the page after a few minutes for it to register.

Dataiku screenshot of the governance status of a deployment.

Govern a bundle in production#

You now have an approved bundle in production! That’s a big step, but it’s not the end of the governance story.

Complete the following three tasks in the Govern node on your own.

  1. In the Bundle registry (Bundle icon.), filter for deployments (Deployments icon.), and confirm your bundle remains.

  2. Advance the governed bundle from the Signed-off stage to Production.

  3. Advance the governed parent project from the In Progress stage to Validation and Roll-out.

See also

Here you have to manually move the governed bundle from the Signed-off stage to Production. However, it’s possible you could automate this with a deployment hook. Learn more in the reference documentation on Deployment infrastructures.

Next steps#

Congratulations! You’ve now seen one complete lifecycle of an AI initiative, from its initial exploration as a governed project to an approved deployment in a production environment.

This was only one possible workflow: the standard workflow. Advanced license holders have the ability to create their own workflows, including review stages as needed. You’ll learn more about customization in the Academy course on Advanced Dataiku Govern.

See also

For more information, see the reference documentation on Governance.