Tutorial | Deploy a Dataiku instance to Cloud Stacks on Azure#

In this article, we’ll walk through a step-by-step process to set up and deploy an instance of Dataiku using Dataiku Cloud Stacks for Azure.

Deploying a Dataiku instance is a three-step process:

  • Gather information

  • Deploy Fleet Manager

  • Deploy a first instance

We’ll use a Microsoft Azure Resource Manager (ARM) template for deploying Dataiku Fleet Manager. Then, we’ll use Fleet Manager to deploy our first Dataiku instance.

Fleet Manager handles the entire lifecycle of Dataiku instances, freeing you from most administration tasks. A fleet is the collection of Dataiku resources such as nodes deployed together. A blueprint is a set of pre-configured instances or complete fleets.

At the end of this user guide, you’ll have set up a Dataiku Cloud Stack for Azure with Dataiku Fleet Manager and a single Dataiku Design node without elastic AI capabilities.

Completing these steps will help you understand the basics of Fleet Manager so that you’ll be able to deploy more complex instances, including complete fleets.

Step 1. Gather information#

Gathering the necessary information up front will make the process run smoother. You can find the following information in your Microsoft Azure environment or by contacting your cloud administrator. You’ll need this information to complete the ARM template:

  • Landing zone information for Dataiku DSS (Dataiku), including the subscription and region

  • An Azure resource group for which you are an owner

  • A user-assigned managed identity dedicated to managing the Dataiku nodes through Fleet Manager.

  • A user-assigned managed identity dedicated to Dataiku resources to manage and interact with Azure Kubernetes Services (AKS) or Azure Container Registry (ACR).

For help creating the required managed identities, visit the user guide.

  • A virtual network range with a CIDR /16 size

  • A SSH key pair

  • Determine if you want a public or private IP address to access Fleet Manager

Step 2. Set up and deploy Fleet Manager#

In this section, we’ll use an ARM template to deploy a cloud stack that contains the following infrastructure:

  • A Dataiku Fleet Manager VM including its storage and NIC

  • A CIDR /16 virtual network

The resources shown in the diagram are needed to set up and deploy Fleet Manager:

../../_images/azure-cloudstack-01.png

Complete the custom ARM template#

To set up and deploy Fleet Manager, we’ll use a custom ARM template.

To open the template:

  1. Visit the Dataiku documentation to find the link.

  2. Select Deploy to Azure.

    The Azure portal displays the template.

  3. Complete the template using the guidelines.

Template guidelines#

Project Details

  1. Choose the Azure Subscription.

  2. Choose an existing Resource group dedicated to the cloud stack or create a new one.

Instance Details

  1. Select the Azure Region where you want to deploy the cloud stack.

  2. Set the Virtual Machine Size for Fleet Manager.

    Note

    A size of 2 vCPUs and 8 GB of RAM is recommended. You can always increase the machine size later.

  3. In Instance identity, enter the fm-id-name.

    • This user-assigned managed identity is dedicated to Fleet Manager and manages the Dataiku nodes.

    • Enter the name of the managed identity. Use the short format of the name. Do not enter the full ID that displays under the managed identity properties. For example, do not include “/subscription/…./resource”.

    • This information is found in the Managed Identity properties in the Azure console.

  4. In Username, choose a username for signing in to Fleet Manager.

    This is the Fleet Manager administrator username.

  5. In Password, enter a strong password for signing in to Fleet Manager.

    This is the Fleet Manager administrator password.

  6. In SSH Username, enter the username for the SSH user that is permitted to connect to the Fleet Manager VM.

    This sudo Linux user will connect to Fleet Manager VM via SSH.

  7. In SSH public key source, choose a source for the key.

    For example, you can choose to use your existing public key.

  8. In SSH Public Key, enter an RSA public key for signing into the Fleet Manager VM.

Network Parameters

  1. In Virtual Network CIDR, enter the address of the CIDR /16 virtual network that is allocated to the cloud stack, for example, 10.0.0.0/16.

  2. Enter the Subnet CIDR you want to create. Use the X.X.X.X/X format.

  3. Enter the Private Ip Address for Fleet Manager.

  4. In Associate Public Ip Address, select true if you want a public IP address for connecting to Fleet Manager; otherwise, select false.

    Note

    If using a public IP address, you may need to consider how to secure it.

  5. In Allow CIDR, enter the IP addresses permitted to connect to Fleet Manager.

  6. Leave Snapshot empty. This is the default setting for a new installation.

    Note

    • To migrate or upgrade from a snapshot, you’ll first need to create a snapshot of the data disk for the Fleet Manager instance you want to migrate or upgrade.

    • You’ll then reference the Resource ID in the Snapshot field.

      For example: /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxx/providers/Microsoft.Compute/snapshots/my-snapshot.

You are now ready to review and deploy the template.

  1. Select Review + create.

  2. Verify the creation information, and then select Create.

Azure deploys the resources, including the new virtual network resulting in a new Fleet Manager VM on your cloud stack.

Note

To visualize the deployed resources, choose Visualize in the Azure console.

Azure displays an error message if it is not able to validate the information provided in the template or if rights to create the resources are insufficient.

You are now ready to deploy your first instance of Dataiku.

Step 3. Deploy your first Dataiku instance#

Log in to Fleet Manager#

To connect to Fleet Manager, you’ll need to determine the Fleet Manager IP Address. To do this:

  1. In the Azure console, select Go to resource group.

  2. Choose the VM instance to view its details.

  3. Copy the Public IP address.

    Note

    If you selected private IP, you’ll need to expose the application.

  4. Type the address in your browser to launch Fleet Manager.

  5. Wait while Fleet Manager starts.

  6. Enter the administrator username and password you specified in the template.

The Dataiku Fleet Manager homepage displays.

Configure your Dataiku license#

To deploy a Dataiku instance, you’ll first need to configure your Dataiku license. One way to do this is by using the settings under Cloud setup.

  1. Under Settings, choose Cloud setup.

  2. Select Edit.

  3. In License mode, choose Manually entered.

  4. In License file, enter your license file. Be sure to copy the entire contents of the JSON file including the final ‘}’.

  5. Save your changes.

The license file now displays the license expiration date.

Deploy a Fleet blueprint#

Fleet blueprints allow you to deploy pre-configured instances or complete fleets with minimal setup. Fleet blueprints automatically create instances, instance setting templates, and virtual networks. Using Fleet blueprints you can deploy everything from a minimal design with a single Design node to a full fleet with elastic AI capabilities.

In this section, we’ll deploy a single Design node without Kubernetes (Elastic AI) capabilities.

  1. Under Quick Start, choose Fleet blueprints.

  2. Choose Deploy Minimal Design.

    ../../_images/deploy-minimal-design.png

    Fleet Manager displays the fleet configuration page.

  3. Enter a descriptive name for the fleet.

    • Follow your organization’s naming convention for network resources.

    • For example, design-admin-learning-dss-01.

    • This name is added to all deployed instances, instance templates and virtual networks.

  4. Optional: Add tags to tag your Azure resources.

  5. Optional: Enter the name of the Azure resource group in which all resources will be created.

    By default, all resources will be created under the Azure Virtual Machine resource group (this is the resource group where Fleet Manager is deployed).

  6. Enter an authorized SSH key. This allows Azure to sign in to the Dataiku instances.

  7. Optional: In Design node data volume name, assign a unique name to the storage volume.

    Caution

    This cannot be renamed later.

    Review and update your settings before performing the actual deployment.

  8. Select Deploy.

    Fleet Manager creates the fleet and lets you know that the fleet is ready for provisioning.

  9. Select OK and return to the Fleet Manager homepage to view All instances.

You are now ready to provision the instance.

Provision the instance#

The Dataiku instance(s) you deployed using the blueprint are not yet available to Dataiku users. In this section, we’ll provision the instance.

Before provisioning any instance(s), you can specify configuration options, including the VM type and the name of the Azure objects, and add tags.

  1. From Instances, choose All and then locate the new instance.

  2. Select the name of the instance to view its settings.

    The status of the instance is Not provisioned.

  3. Navigate to the Settings tab.

    You can configure the settings, including the VM type and the name of the Azure objects. You can also add tags. Each field is labeled to let you know if the change requires reprovisioning.

  4. If you made any changes, select Save.

You can now provision the instance to make it available to Dataiku users.

  1. Select the Save menu arrow to display the list.

  2. Choose Provision.

    ../../_images/provision-button.png

  3. Wait while Fleet Manager provisions the instance.

In the Fleet Manager homepage, you can see that the status of the instance has changed from Not provisioned to Running.

The resources deployed in the Azure resource group now look like this:

../../_images/azure-cloudstack-02.png

You are now ready to launch the Dataiku instance.

Launch Dataiku#

In this section, we’ll launch the Dataiku instance.

  1. From Instances, choose All and locate the newly provisioned instance.

  2. Select the name of the instance to view its settings.

The dashboard displays the following information:

  • The instance details, including the storage capacity

  • Settings templates

  • The VM’s Virtual network connection

  • Basic VM information

  • Agent logs

To launch Dataiku:

  1. Select Retrieve to retrieve the initial admin password.

  2. Copy the password. It will display only once.

  3. Select Go to DSS.

    ../../_images/instance-status-running.png

  4. Sign in with your admin username and password.

  5. Change the initial password.

Note

You can manage the admin password by visiting the Administration menu in your Dataiku instance. Visit the Security tab and then select the admin user. In Change password, enter a new password.