Reporting Scenario Activities¶
Once you’ve created a scenario, you may be interested in receiving updates about scenario activities, such as model training or changes in data quality.
Dataiku DSS provides the ability to add reporters which can send these updates, and actionable messages about scenario activities to users. Reporters can be set up to auto-trigger and send messages to email addresses or messaging channels such as Slack, Hipchat, Microsoft Teams, and Twilio.
Before beginning the hands-on portion of the tutorial, you can watch the following video, which walks through the outline of the steps.
Let’s Get Started!¶
In this tutorial, you will learn the basics of creating reporters that send updates about scenario activities. These reporters will send messages:
to a channel in Microsoft Teams
at the beginning of a scenario run
at the end of a scenario run
We will work with the fictional retailer Haiku T-Shirt’s data.
Prerequisites¶
This tutorial assumes that:
you have completed the Metrics, Checks, & Scenarios lesson.
you have an account with Microsoft Teams and an existing channel called Customer Churn Analysis.
Create Your Project¶
From the DSS homepage, navigate to the project you started in the previous lesson. For this tutorial, we will continue to work with the scenario Rebuild data and retrain model that was created in the Automation tutorial.
From the Jobs menu in the top navigation bar, select Scenarios and open the Rebuild data and retrain model scenario.
Create a Reporter to Send a Message When the Scenario Starts¶
We will create a reporter that sends messages to a channel in Microsoft Teams when the scenario begins. To do this, first obtain the URL of an incoming webhook connector for your channel in Microsoft Teams. You may have to set up the incoming webhook if one doesn’t already exist.
In this tutorial, you will connect DSS to the Customer Churn Analysis channel, by means of an incoming webhook connector called DSS Scenario Reporter. Using this connection, DSS can post notifications about the scenario’s activities in the channel.
You create reporters on the Settings tab of the scenario. Say you want to create a reporter that sends messages to the Customer Churn Analysis channel at the beginning of a scenario run. In the Settings tab:
Click Add Reporter and select Microsoft Teams.
Name the reporter
Scenario Started.Set the value of “Send on scenario” to Start.
Set the “Run condition” to Off. The run condition controls whether the reporter runs at the end of the scenario, therefore, we want it off in this reporter.
Paste the URL of the webhook connector in the “URL” section. This allows DSS to connect to the Customer Churn Analysis channel.
Now, let’s customize the message that the reporter will send when triggered. The “Body” of the reporter contains some default text: { "text" : "DSS scenario ${scenarioName}: ${outcome}"} that consists of message content and variables. You can see a list of the available variables and their descriptions in the “Available variables” section of the reporter.
Edit the “Body” to make it more descriptive, by specifying the trigger name:
{ "text" : "DSS scenario ${scenarioName} has begun. Triggered by ${triggerName}"}.
The reporter is now ready to use. To test it, click Save and Run Now.
When you return to the Customer Churn Analysis channel of Microsoft Teams, notice that DSS has sent a message to the channel, stating: “DSS scenario Rebuild data and retrain model has begun. Triggered by Manual trigger”.
Create a Reporter to Send a Message When the Scenario Ends¶
It is also useful to send notifications of scenario results, for example: to send a message if a scenario fails. Let’s configure a reporter to do just this.
Configure the Reporter Settings¶
To create a reporter that runs when the scenario ends:
Click Add Reporter and select Microsoft Teams.
Name the reporter
Scenario Failed.Keep the default value of “Send on scenario” as End.
Set the “Run condition” to ON. Recall that the run condition further controls whether the reporter runs at the end of the scenario. Keep the default value of the run condition:
outcome != ‘SUCCESS’. This tells the scenario to run at the end of the scenario activity, if the scenario fails.Paste the URL of the webhook connector in the “URL” section, so that DSS can connect to the Customer Churn Analysis channel.
Customize the message that the reporter will send when triggered, by editing the “Body” of the reporter to read:
{ "text" : "DSS scenario ${scenarioName}: ${outcome}. Failed on ${firstFailedStepName}"}. Notice that the “Available variables” section of the reporter consists of a different set of variables that are unique to this kind of reporter (one that sends messages at the end of a scenario activity).Save the reporter settings.
Trigger the Reporter¶
Now that you’ve configured the settings for the reporter, the next thing to do is trigger the reporter This way, you can test that it works as it should, when the scenario fails.
In the monitoring with metrics and checks section of the previous Automation tutorial, we enabled the Records count metric to automatically compute after building the Orders_filtered dataset.
We also created a check that issued a warning if the Record count (representing the number of new customers in the dataset) was less than 170.
Now we will change the Checks settings so that the job fails (instead of merely giving a warning), when the number of new customers in the dataset is less than 170. To do this:
Navigate to the Status tab of the Orders_filtered dataset, then click the Edit tab and then the Checks panel.
Edit the settings for the Minimum new customers check to ensure that the software returns an error if the number of new customers is less than 170:
Set the “Minimum” value to
170(this is a hard minimum).Turn off the setting for the soft minimum.
Save
The scenario has a trigger that checks the input datasets: Customers and Orders, and rebuilds the Orders_filtered dataset whenever the input data changes. If you proceed to build the Orders_filtered dataset with the “Build mode” set to Build required datasets (the current setting), DSS will check to see if the source datasets have changed, and will update each dataset in the flow as appropriate.
Because we have not changed the input datasets, rebuilding the Orders_filtered dataset in the current build mode would not actually update the dataset. As a result, the Records count metric would not auto compute, and the Minimum new customers check would not run.
To ensure that the metrics and check run, we must ensure that the Orders_filtered dataset is re-built, by changing the build mode for the dataset to Force-rebuild the dataset and dependencies. Specifically:
Return to the scenario page and navigate to the Steps tab.
Change the “Build mode” in the Build Filtered Orders step to Force-rebuild dataset and dependencies. This ensures that DSS also rebuilds all datasets leading to the Orders_filtered dataset in the Flow. The metrics and checks also automatically compute and run, once the Orders_filtered dataset is built.
Test It All¶
Let’s now test the reporters in our scenario by hitting the Save and Run Now buttons.
Notice that the scenario Rebuild data and retrain model and the job Build Orders_filtered start and soon fail. This is because the checks run as part of a build, and having the check return an ERROR status causes the build to fail.
Upon returning to the Customer Churn Analysis channel in Microsoft Teams, we notice that DSS Scenario Reporter has sent a couple of messages to the channel. The first is to notify members that the scenario has begun, and soon after, the second message arrives to inform members that the scenario has failed.
Next Steps¶
Apart from sending simple messages to a Teams channel, DSS supports Microsoft cards via the scenario reporter. These cards are created with JSON text, and they contain components like images, graphs, and more, to present richly formatted content directly into a Microsoft Teams conversation.
We have now created our first couple of scenario reporters. Thanks for following this tutorial!