Concept | Shared code#

Reusable code enhances collaboration between users on a Dataiku instance, reduces the time spent coding, and helps improve consistency.

Dataiku offers several ways to share code:

  • Project libraries store reusable code within a project. Other projects can import them, and they can include code from Git repositories.

  • Global shared code makes code available to all projects on the instance.

  • Code samples store frequently used code snippets that users can insert into their code.

Project libraries#

Project libraries store reusable code, such as functions or modules. This code is available within the same project, for example in recipes and notebooks, and other projects on the same instance can import it.

Dataiku screenshot of a project library.

Library editor#

The library editor, available from the Code (Code icon.) > Libraries page, stores the project’s code files. It lets you create folders and upload or add files.

Modules written in a Python source folder, such as the default python folder, are automatically available in all Python code of the project. For example, the module below defines a function that writes data from a Dataiku dataset to an Excel workbook.

Dataiku screenshot of a Python code library.

Note

The code in this screenshot uses writer.save() and the encoding parameter of to_excel(), which were removed in pandas 2.0. With recent pandas versions, use a with pd.ExcelWriter(...) block instead, and drop the encoding parameter.

Any recipe or notebook of the project can then use the dataframe_to_xlsx function, with a regular Python import statement:

from to_xlsx import dataframe_to_xlsx

Tip

A running notebook doesn’t detect library changes automatically. After modifying the library, click Force Reload in the notebook header to restart the kernel and load the latest version of the library.

Libraries shared between projects#

A project can import the library of another project on the same instance. The external-libraries.json file, found in the library editor, lists in importLibrariesFromProjects the keys of the projects whose libraries are imported. For example, project B imports the library of project A with:

{
  "importLibrariesFromProjects": ["PROJECT_A_KEY"]
}
Slide demonstrating how to copy a project key to share code from another project.

The project key appears in the project’s URL, and differs from the project’s display name. Importing a library requires the Read project content permission on the source project and the Write project content permission on the importing project.

Note

A project library only needs to be imported once. If the parent project’s library is updated, those changes are available to all child projects. The only time a change in the parent project won’t be usable is if a user unintentionally changes the name of a shared file or function.

When projects share libraries and are deployed to an Automation node, the parent project must also be deployed so that its library is available.

Code imported from Git#

Project libraries can also include code developed outside of Dataiku and stored in a Git repository, such as a library created by another team. This mechanism is called Git references.

A Git reference imports a whole repository, or part of it, into the project library. In the library editor, the Git > Import from Git menu creates a Git reference. The imported code isn’t a frozen copy: the Git > Manage repositories menu updates the references to retrieve the latest changes from the remote repository.

You can also import notebooks from a Git repository and keep them in sync with it. See How-to | Import a notebook from Git.

See also

For more information, see Importing code from Git in project libraries in the reference documentation.

Global shared code#

In addition to project libraries, the Global Shared Code page, available from the waffle (Waffle icon.) menu, provides a library editor for the whole instance. Python modules written there are available in all projects.

Editing global shared code requires the Edit lib folders global permission. Because global code increases the risk of conflicts between projects, project libraries are generally preferable. Also note that global libraries aren’t included in project bundles.

Tip

The Documentation Portal page, available from the waffle (Waffle icon.) menu, displays auto-generated documentation for the Python code stored in project libraries and global shared code. It helps users discover the code available on the instance. See Documentation Portal for more information.

Code samples#

Code samples store frequently used code snippets, so that users can insert them when writing code in Dataiku.

Dataiku screenshot of a code recipe showing available code samples.

Dataiku stores code samples in collections of the Enterprise Asset Library. Users can access the code samples of any collection where they hold at least Reader rights.

Creation of code samples#

Users with Contributor rights on a collection can create code samples in it, from the New Asset > Create code sample menu of the collection or from the + Add Your Own button. A code sample includes:

  • A title and a description, which are both searchable.

  • Tags, which make the code sample easier to discover.

  • The type of code, such as Python, R, or SQL.

  • Optional documentation explaining the purpose and usage of the code sample.

  • One or more variations of the code.

Dataiku screenshot of the dialog for creating a code sample.

Variations make sense when a single code snippet can’t cover all use cases.

Users can also keep personal drafts in the My Assets collection, and then publish them to shared collections for broader reuse across teams.

See also

For more information, see Managing code samples in the reference documentation.

Usage of code samples#

Code samples make it easier to reproduce a Dataiku object or a simple component from one project to another, improving the reusability and reliability of the code. Wherever you write code in Dataiku, a Code Samples button gives access to them: for example, in notebooks, in the Code tab of code recipes, in the Settings tab of code webapps, or in the code editors of the visual ML tool.

This button opens a panel listing the code snippets available on the instance. For instance, a search on dataframe returns the code samples whose title or description contains this term, such as snippets to merge pandas DataFrames. Tags such as Pandas, Combine, and Dataframe help narrow down the results.

Dataiku screenshot of a code sample in a notebook.

Dataiku displays the code sample, including any variations, along with its documentation. Once inserted, the code becomes part of the script.

Next steps#

To go deeper into sharing code, practice with Using Git shared code in your projects in the Developer Guide, or see Reusing Python code in the reference documentation.