Project Cleaning and Maintenance

Explore resources here to understand how you can use scenarios and macros to automate the cleanup of projects across your instances.

Tutorials

• Tutorial | Create a scenario for automating project maintenance macros

Reference | Project maintenance macros

With maintenance macros, you clean and maintain all projects on your DSS instance. Maintenance macros help you perform maintenance tasks such as deleting jobs and temporary files. For some maintenance macros, you can configure the steps in a scenario to execute the macro across one or all projects on the instance.

To view DSS maintenance macros, navigate to the More Options (“…”) menu and choose Macros.

Note

Visit DSS Macros for technical details about macros.

Reference | Project maintenance macros glossary

This section describes the DSS maintenance macros.

Backup internal databases

This macro backs up the contents of the internal databases of DSS, so that they can be truncated.

Note

This macro is not working for externally hosted internal databases.

Warning

This macro will lock the databases while exporting their contents, potentially blocking usage of DSS. Usage of DSS could be blocked for seconds or minutes depending on the size of the databases.

When should I run this macro?

You can run this macro when you want to have a backup of your internal database, e.g., before externalizing your database.

Clear app instances

This macro deletes all instances of an app whose last activity is older than a certain number of days.

When should I run this macro?

You can run this macro when you want to free up some memory on your instance and maintain control of instances of a deployed app.

Clear continuous activities logs

This macro deletes all logs and temporary files of continuous activities older than a certain number of days. The operation also deletes the runs from the user interface.

When should I run this macro?

Run this macro when you are working with continuous activities such as streaming features.

Clear deleted experiments

This macro permanently deletes Experiment Tracking experiments and runs in the “deleted” lifecycle stage. It also deletes all artifacts and metadata.

This macro is equivalent to the MLflow “gc” command.

When should I run this macro?

Run this macro when you want to permanently delete project items.

Clear internal databases

You can configure this macro to clear jobs histories, metrics, checks, scenario runs history, and activity timelines.

When should I run this macro?

You can run this macro when your internal database becomes too large and DSS is experiencing performance degradation.

Clear job logs

This macro deletes all logs and temporary files of jobs older than a certain number of days.

The information about the job itself is not deleted. After running this macro, DSS still displays jobs in the jobs list, but all details about the old jobs will be removed.

When should I run this macro?

You can run this macro when you want to control disk space on your DSS server and control retention of jobs logs files.

Clear model versions

This macro deletes the oldest inactive model versions for the project.

When should I run this macro?

You can run this macro when you want to free up disk space.

Clear scenario run logs

This macro deletes all logs and temporary files of scenario runs older than a certain number of days.

The information of the scenario run itself is not deleted. After running this macro, DSS displays the runs in the last runs tab of the scenario, but all details about the old runs will be removed.

When should I run this macro?

You can run this macro in order to retain control of your disk space on your DSS server and control retention of your scenario run logs files.

Drop pipeline views

This macro drops any views on a connection that remain after executing one or more SQL pipeline jobs. DSS drops the view from all projects on the instance and not just the project where you run the macro.

When should I run this macro?

Leaving views behind at the end of a pipeline can cause problems if you later try to drop the table from which the view was derived. Run this macro to clean up any old pipeline views on a connection.

List datasets on connection

This macro lists all datasets using a given connection.

When should I run this macro?

You can run this macro when you want to list all datasets using a particular connection. This is useful when you have planned to remove a connection.

Remove old container images

This macro tries to remove all outdated container images that are no longer in use. It will not remove images that have dependent child images or images that are being used by containers.

When should I run this macro?

You can run this macro when you want to retain control of the container images deployed by DSS because it removes old and unused container images built inside DSS.

Remove old exports

This macro deletes exports such as datasets, notebook results, and script results that are older than a certain number of days.

When should I run this macro?

Run this macro to help control disk usage.

Kill Jupyter sessions

This macro kills Jupyter sessions that have either been running for too long or have been idle for too long.

When should I run this macro?

You can run this macro when you want to free up some memory in your instance; this will delete your old and unused Jupyter sessions.