Plugin Development

In these topics, you will learn how to write your own plugins with programming languages, such as Python and R.

Tip

You can find this content (and more) by registering for the Dataiku Academy course, Plugin Development. When ready, challenge yourself to earn a certification!

Concepts

• Concept | Plugin development
• Concept | Development plugins
• Concept | Git integration for plugins

Tutorials

• Tutorial | My first plugin recipe and processor
• Tutorial | Plugin version management with Git

Reference | Plugin naming policies and conventions

Naming is an important area of plugin development that is often overlooked. Dataiku requires that plugin names and identifiers meet certain policies.

Benefits of a naming policy

It can be difficult to tell which of your installed plugins are developed internally, externally, or from Dataiku. One way to keep all your ducks in a row is to apply a plugin naming convention. In addition, when you share your plugins, your colleagues will be able to understand the purpose of the plugin.

Plugin display name and identifier

The plugin summary displays information about the plugin and its components. You can edit the display name and identifier for the plugin in the Edit tab.

Component display name and identifier

You can also edit the component display name and component identifier in the Edit tab. Adding a description is a good idea because it documents the reason why you created the component and lets others understand its usefulness and purpose.

Policies and guidelines

A naming error happens when the name of the plugin, or any of its components, labels, or tags, does not meet plugin naming policies. Luckily, you can avoid these errors by following the plugin naming policies below. Most of the policies are required while a few are recommended.

Plugin.json description

Reference | IDE setup to develop Dataiku plugins

Though the Dataiku interface has a built-in code editor, you may have a preferred IDE. You can set up your IDE for interactive editing of your plugin code.

This enables you to develop plugins in your favorite coding environment. You will also be able to test that your plugin is working as expected by interacting with Dataiku through its APIs.

Prerequisites

• This guide covers setup with Atom and Sublime Text 3.

• This guide covers only interactive python execution.

Note

All steps in this tutorial happen outside of Dataiku.

Installing packages

You must start by setting up a virtual environment in your workspace. In Python 3 you can use python3 -m venv dataiku_dev_env.

Activate your environment by running source dataiku_dev_env/bin/activate.

To install the dataikuapi package you can run pip install dataiku-api-client.

You must now install the dataiku package, please follow the instructions on our page dedicated to the installation of this package.

Set up the kernels

We will now take care of the interactive environment. Run pip install ipykernel to install Jupyter.

You need to create a kernel specification in your user space so that your IDE can find it later. python -m ipykernel install --user --name dataiku --display-name "Python (Dataiku venv)"

Set up your code editor

In atom

Install the package Hydrogen by going to Settings (ctrl+,) -> + Install.

You can now open any Python source file and use shift+enter. This will start a kernel in the current file.

In Sublime Text 3

Install the package Hermes (use ctrl+shift+p to open Package Control).

When you start a Hermes kernel, an output tab will appear. It is recommended to split your view in two panes and have that tab in the right pane.

Open your Python source file in the left pane and add # %% to define your cells. A “Run cell” phantom button will appear next to them!

Develop a plugin

File structure

Within plugins, it is recommended to develop libraries that will provide the core functionality you are looking to add. Additionally, you should provide some recipes that use your libraries.

When developing locally you can use a test file to check that your library is working as expected. Here is a typical file structure:

custom-recipes/my-recipe        Recipe folder
    recipe.json                   Recipe definition
    recipe.py                     Recipe contents, runs within DSS

python-lib                      Library folder
    mylib.py                      Library contents

README.md
plugin.json
test.py                         A local file to test your library

How-to | Clone a plugin from a remote git repository

Installing an existing plugin is useful if you simply want to use the plugin. If you want to contribute to the development of an existing plugin, you’ll need to clone the remote Git repository (repo) where the plugin source code is stored.

Prerequisites

• Familiarity with plugins.

• Familiarity with the basics of Git.

Technical requirements

• Access to a Dataiku instance that has been set up to work with remote Git repositories.

Connect to a remote Git repository

• Navigate to the Plugins Development area and click +New Dev Plugin.

• Select Retrieve plugin from Git as the source.

• Enter https://github.com/dataiku/dss-plugin-sample-correlations as the URL of the remote.

• Click Create

• Click on the change tracking indicator and select Add remote.

• Pull the changes from the remote

In the plugin editor, you should now see the contents of the remote repo in your local repo. If you have write privileges on the remote repository, you can push changes as described in Tutorial | Plugin version management with Git.

How-to | Share a plugin as a zip archive

To share your dev plugin as a zip archive, you simply select Download this plugin from the actions menu of the plugin. Alternatively you can compress the contents of the plugin folder to a zip file (the contents of the plugin folder must be at the root of the zip file, i.e. there must not be any leading folder in the zip file). The plugin.json file should be at the root of the zip file.

A typical zip file would therefore have this structure:

./plugin.json
./python-connectors/myapi-connector/connector.py
./python-connectors/myapi-connector/connector.json
./python-lib/myapi.py

You can then install this zip file in any Dataiku instance by following these instructions.

How-to | Edit a plugin

You can convert a plugin from the Plugin Store to a Dev Plugin. Use caution. You will not be able to get updates from the store or from your own repositories afterward.

Tip | Never use “import dataiku” in a plugin

One of the golden rules of plugins is to never use “import dataiku” in a library. With few exceptions, plugins do not depend on Dataiku unless it is an administrative plugin.

FAQ | Why should I create plugins?

Plugins turn custom logic into components that can be used by anyone, giving less technical users the ability to collaborate and technical users the ability to standardize the way data is processed. Plugins can easily be shared by the community of users in an organization and even shared across organizations. Since most plugins are open source, anyone can view the underlying code and make contributions to it or adapt it.

FAQ | What are some examples of plugins?

Plugins range from the very simple to the very complex. For example, plugins can be used to create a connector to read or write data to databases or connect to external APIs. Some examples are connecting with Salesforce data, or get weather forecast by location. You can find more plugins in the plugin store. The possibilities are limitless!

FAQ | Where can I find the code for a plugin?

You can find the code for some but not all of the public plugins on GitHub.