How to Automate Project Cleaning and Maintenance

Introduction and Definitions

Project cleaning and maintenance can quickly become repetitive tasks. Built-in maintenance macros allow you to automate these tasks across one or all projects on your instance. You automate project cleaning and maintenance across a Dataiku DSS (DSS) instance by creating an Admin Project.

At the end of this article, you’ll be able to perform the following tasks:

• Create an Admin Project and run maintenance macros.

• Understand the DSS project cleaning and maintenance macros and be able to describe them.

• Create a scenario to automate cleaning and maintenance tasks.

What is a Maintenance Macro?

With maintenance macros, you clean and maintain all projects on your DSS instance. Maintenance macros help you perform maintenance tasks such as deleting jobs and temporary files. For some maintenance macros, you can configure the steps in a scenario to execute the macro across one or all projects on the instance.

To view DSS maintenance macros, navigate to the More Options (“…”) menu and choose Macros.

Visit DSS Macros for technical details about macros.

What is an Admin Project?

An Admin Project is a blank project you create that is accessible only to admins on the instance. An Admin Project contains a scenario with one step for each macro you want to execute.

After following the steps described in this article, your Admin Project will look like this:

Create the Admin Project

To begin, let’s start with a new, blank project. You’ll need to create the project while signed in to the instance as a user with administrator privileges. This will become more obvious when you are creating steps in your scenario. Without administrator privileges, you will not have the option to apply macros to all projects on the instance.

• In your DSS instance, create a new, blank project and give it a name like, Admin Project.

Next, we’ll set the project visibility and permissions:

• From the top navigation bar in your project, navigate to the More Options (“…”) menu and choose Security.

• Set the Project visibility to Private.

• Set Project access requests to Disabled.

We’ll want administrators to have access to this project, so let’s grant access to the administrators group.

• Choose administrators and click Grant Access to Group.

• Select Admin to give administrators Admin permissions.

• Save your changes.

Our next step is to configure a scenario to run our macros.

Automate Project Cleaning and Maintenance

You can run macros manually or automatically from a scenario step. In this section, we’ll create a scenario to execute five maintenance macros. These five macros are recommended as part of project maintenance best practices:

• Clear job logs

• Clear scenario run logs

• Kill Jupyter sessions

• Clear internal database (if you are using initial internal database h2)

Note

For production environments, the use of an externally hosted PostgreSQL runtime database is recommended. Visit Externally hosting runtime databases for more information.

• Clear continuous activities

Note

Clearing continuous activities is recommended if you are using streaming features.

Let’s configure a scenario to run the recommended macros.

Create a Scenario

To create your scenario:

• From the Jobs menu, navigate to the Scenarios panel, and create a new scenario.

• Ensure Sequence of steps is selected, and name it Maintenance Scenario.

Add a Trigger

Let’s create a time-based trigger so that our scenario runs each hour.

• Within the Triggers panel of the Settings tab, click the Add Trigger dropdown menu.

• Add a Time-based trigger.

• Instead of the default “Time-based”, name it Every 1 hour.

• Change Repeat every to 1 hours.

• Make sure its activity status is toggled to ON.

Next, we’ll add steps, one for each macro we want to run. We’ll configure each step to execute the macro on all projects on the instance.

Add a Step to Clear Job Logs

Our first step will run a macro to clear job logs.

• Navigate to the Steps tab.

• From the Add Step dropdown, choose to add an Execute macro step.

• Name it Clear job logs then select Clear job logs as the macro.

• Set the Max age (days) to 9.

• Select Perform deletion.

• Select All projects.

This tells DSS to delete logs older than nine days for all projects on the instance.

• Save your changes.

Add a Step to Clear Scenario Run Logs

This step will clear scenario run logs.

• From the Add Step dropdown, choose to add an Execute macro step.

• Name it Clear scenario run logs then select Clear scenario run logs as the macro.

• Set the Max age (days) to 9.

• Select Perform deletion.

• Select All projects.

This tells DSS to delete all logs and temporary files of scenario runs that are older than nine days.

Add a Step to Kill Jupyter Sessions

Let’s add a step to kill Jupyter sessions to free up some memory on the instance when Jupyter sessions have been running for too long or when they have been idle for too long.

• From the Add Step dropdown, choose to add an Execute macro step.

• Name it Kill Jupyter sessions then select Kill Jupyter sessions as the macro.

• Leave the default settings.

This tells DSS to delete old and unused Jupyter sessions.

Add a Step to Clear Internal Databases

Adding a step to clear internal databases can help resolve performance degradation.

• From the Add Step dropdown, choose to add an Execute macro step.

• Name it Clear internal databases then select Clear internal databases as the macro.

• Select Clear for all projects.

• Set Max age to 9.

This tells DSS to truncate jobs, scenarios and metrics histories for all projects.

Add a Step to Clear Continuous Activities Logs

When working with continuous activities such as streaming features, the continuous activities logs and temporary files can grow very quickly. Therefore, we might want to delete the logs and temporary files older than a certain number of days.

• From the Add Step dropdown, choose to add an Execute macro step.

• Name it Clear continuous activities logs then select Clear continuous activities logs as the macro.

• Set the Max age (days) to 9.

• Select Perform deletion.

• Select All projects.

This tells DSS to delete all logs and temporary files of continuous activities older than nine days.

• Save your changes.

Trigger the Scenario to Run as Admin

Now that we have configured our scenario, let’s set it to run automatically by enabling the auto trigger. We’ll also tell DSS to run the scenario as admin. Admin is required for running maintenance macros on each project on the instance.

• Within the Run panel of the Settings tab, toggle the Auto-triggers to ON.

• Set the Run as option to admin.

• Leave the default settings.

• Save your changes.

You can now run the scenario.

Maintenance Macro Descriptions

This section describes the DSS maintenance macros.

Backup Internal Databases

This macro backs up the contents of the internal databases of DSS, so that they can be truncated.

Note

This macro is not working for externally hosted internal databases.

Warning

This macro will lock the databases while exporting their contents, potentially blocking usage of DSS. Usage of DSS could be blocked for seconds or minutes depending on the size of the databases.

When should I run this macro?

You can run this macro when you want to have a backup of your internal database, e.g., before externalizing your database.

Clear App Instances

This macro deletes all instances of an app whose last activity is older than a certain number of days.

When should I run this macro?

You can run this macro when you want to free up some memory on your instance and maintain control of instances of a deployed app.

Clear Continuous Activities Logs

This macro deletes all logs and temporary files of continuous activities older than a certain number of days. The operation also deletes the runs from the user interface.

When should I run this macro?

Run this macro when you are working with continuous activities such as streaming features.

Clear Deleted Experiments

This macro permanently deletes Experiment Tracking experiments and runs in the “deleted” lifecycle stage. It also deletes all artifacts and metadata.

This macro is equivalent to the MLflow “gc” command.

When should I run this macro?

Run this macro when you want to permanently delete project items.

Clear Internal Databases

You can configure this macro to clear jobs histories, metrics, checks, scenario runs history, and activity timelines.

When should I run this macro?

You can run this macro when your internal database becomes too large and DSS is experiencing performance degradation.

Clear Job Logs

This macro deletes all logs and temporary files of jobs older than a certain number of days.

The information about the job itself is not deleted. After running this macro, DSS still displays jobs in the jobs list, but all details about the old jobs will be removed.

When should I run this macro?

You can run this macro when you want to control disk space on your DSS server and control retention of jobs logs files.

Clear Model Versions

This macro deletes the oldest inactive model versions for the project.

When should I run this macro?

You can run this macro when you want to free up disk space.

Clear Scenario Run Logs

This macro deletes all logs and temporary files of scenario runs older than a certain number of days.

The information of the scenario run itself is not deleted. After running this macro, DSS displays the runs in the last runs tab of the scenario, but all details about the old runs will be removed.

When should I run this macro?

You can run this macro in order to retain control of your disk space on your DSS server and control retention of your scenario run logs files.

Drop Pipeline Views

This macro drops any views on a connection that remain after executing one or more SQL pipeline jobs. DSS drops the view from all projects on the instance and not just the project where you run the macro.

When should I run this macro?

Leaving views behind at the end of a pipeline can cause problems if you later try to drop the table from which the view was derived. Run this macro to clean up any old pipeline views on a connection.

List Datasets on Connection

This macro lists all datasets using a given connection.

When should I run this macro?

You can run this macro when you want to list all datasets using a particular connection. This is useful when you have planned to remove a connection.

Remove Old Container Images

This macro tries to remove all outdated container images that are no longer in use. It will not remove images that have dependent child images or images that are being used by containers.

When should I run this macro?

You can run this macro when you want to retain control of the container images deployed by DSS because it removes old and unused container images built inside DSS.

Remove Old Exports

This macro deletes exports such as datasets, notebook results, and script results that are older than a certain number of days.

When should I run this macro?

Run this macro to help control disk usage.

Kill Jupyter Sessions

This macro kills Jupyter sessions that have either been running for too long or have been idle for too long.

When should I run this macro?

You can run this macro when you want to free up some memory in your instance; this will delete your old and unused Jupyter sessions.