Tutorial | Project variables in visual recipes#

Project variables enable you to replace hard-coded values in many places throughout the Flow of a Dataiku project.

Get started#

Objectives#

In this tutorial, you will:

  • Define project variables.

  • Insert them into formulas and visual recipes using both the variables[“variable_name”] and ${variable_name} syntax.

Prerequisites#

To reproduce the steps in this tutorial, you’ll need:

  • Access to an instance of Dataiku 12+.

  • Basic knowledge of Dataiku (Core Designer level or equivalent).

  • You may also want to review this tutorial’s associated concept article.

Create the project#

  1. From the Dataiku Design homepage, click + New Project > DSS tutorials > Advanced Designer > Project Variables.

  2. From the project homepage, click Go to Flow.

Note

You can also download the starter project from this website and import it as a zip file.

You’ll next want to build the Flow.

  1. Click Flow Actions at the bottom right of the Flow.

  2. Click Build all.

  3. Keep the default settings and click Build.

Define project variables#

The first step is actually defining variables in JSON format so that they will be accessible throughout the project.

  1. In the More Options menu of the top navigation bar, click Variables.

  2. In the Global variables section, copy-paste the following JSON object.

    {
  "mean_purchase_amount" : 232,
  "chosen_category" : "B",
  "tx_month" : "2017-01"
}

  3. Click Save.

Discover the two syntaxes for retrieving variables#

The reference documentation details two syntaxes for retrieving variables. There are two important questions to consider when choosing a variable type:

  • How does Dataiku evaluate the variable expression?

  • What does Dataiku return when there is no variable with the supplied name?

Syntax

Evaluation

Variable not found

variables[“variable_name”]

JSON

Empty cell

${variable_name}

Verbatim replacement

Error

Retrieve a variable in a formula#

Let’s first look at how these two syntaxes can be used within formulas.

Return to a visual analysis#

This project has a working Flow that we may not wish to disturb. We can safely experiment with adding variables to the Prepare recipe script by using the existing visual analysis attached to the Prepare recipe.

  1. From the top navigation bar, open the Visual Analyses page (or use the shortcut g + a).

  2. Open the Analyze tx_joined Lab object.

  3. Navigate to the Script tab to find the same steps found in the Prepare recipe of the Flow.

Dataiku screenshot of a visual analysis in the Lab.

Retrieve an integer variable#

Let’s start with an integer example.

  1. In the script, click + Add a New Step.

  2. Choose a Formula.

  3. Name the output column int_variable_example.

  4. Click Open Editor Panel.

  5. Provide the formula purchase_amount > variables["mean_purchase_amount"], and view the sample output in the preview.

  6. Now to demonstrate the difference between variable syntaxes, switch to the formula purchase_amount > ${mean_purchase_amount}, and compare the sample output.

  7. Click Apply, and recognize no discernible difference between the output of these two syntaxes in this example.

Dataiku screenshot of an integer variable in a formula.

Retrieve a variable that does not exist#

Now let’s try to retrieve a variable that does not exist.

  1. Click Open Editor Panel to reopen the step that creates the int_variable_example column.

  2. Change the formula to reference a variable that does not exist, for example: purchase_amount > ${avg_purchase}.

  3. Note how Dataiku warns the formula is invalid, and click Apply. The entire step will return an error.

Dataiku screenshot of an invalid formula.

The entire step returns an error. Try the other syntax.

  1. Once again, click Open Editor Panel to reopen the step that creates the int_variable_example column.

  2. Try the other syntax with a variable that does not exist, for example: purchase_amount > variables["avg_purchase"].

  3. Note how the formula is valid, but the sample output displays errors.

  4. Click Apply, and observe that there is an empty int_variable_example column instead of an error in the script.

Dataiku screenshot of an expression error for an integer variable.

Retrieve a string variable#

Now let’s try retrieving a string variable in a formula.

  1. In the script, click + Add a New Step.

  2. Choose a Formula.

  3. Name the output column str_variable_example.

  4. Click Open Editor Panel.

  5. Provide the formula item_category == variables["chosen_category"], and view the sample output of true and false values in the preview.

  6. Now to demonstrate the difference between variable syntaxes, switch to the formula item_category == ${chosen_category}, and compare the sample output.

Dataiku screenshot of a string variable in a formula.

This is a valid formula, but all of the output values are false. What happened to the true values?

Recall that this syntax performs a verbatim replacement. Therefore, it gets evaluated as item_category == B. What we really want though is item_category == "B". We need to provide these quotes ourselves.

  1. Edit the formula to item_category == "${chosen_category}".

  2. Observe the return of true values in the sample output, and click Apply.

Dataiku screenshot of a string variable in a formula.

Retrieve a variable in a visual recipe condition#

Formulas are not the only location where we can use variables. For example, many visual recipes have pre- and post-filter steps, and variables are often helpful here.

Take the Join recipe in this Flow for example. Before the Join step executes, the Join recipe pre-filters the tx dataset on a particular month. In the current project, this value is hard-coded so let’s replace it with a variable.

Variables inside a conditional filter#

These kinds of pre- and post-filtering conditions lend themselves to a verbatim replacement offered by the ${variable_name} syntax.

  1. Go back to the Flow, and open the Join recipe.

  2. Navigate to the Pre-filters step.

  3. Click on the filter name to edit the filter on the tx dataset.

  4. Delete 2017-01, and replace it with ${tx_month}.

  5. Click Apply.

  6. Click Run to observe that the variable still applies the same rule. (The Run Only This option is sufficient for all builds in this tutorial.)

Dataiku screenshot of the pre-filter step of a visual recipe including a variable.

Variables inside a formula#

We may want the freedom to filter using a formula instead of a GUI-based condition. In this case, both syntaxes are again available, following the same set of rules established above.

  1. Return to the Pre-filters step of the Join recipe.

  2. Click the filter name again to edit the filter on the tx dataset.

  3. Switch from a condition to a formula.

  4. For the second argument of the contains function, provide either variables["tx_month"] or "${tx_month}". Remember to wrap the latter option in quotes so it is read as a string.

  5. Click Apply.

  6. Run the recipe once more to confirm it also works.

Dataiku screenshot of the pre-filter step of a visual recipe including a variable using formula syntax.

Tip

To gain further practice, try replacing the hard-coded pre-filter in the Pivot recipe found in this Flow with a variable!

What’s next?#

In this tutorial, we learned how to define project variables as JSON objects and insert them into visual recipes, demonstrating both available syntaxes.

If you are working with variables, you’re likely getting closer to a production workflow. The next Dataiku feature you may wish to explore is Flow zones.