Reference | Managing elastic AI compute capacity

Dataiku Cloud manages the infrastructure of your instance and provides elastic AI computing capabilities that can be used on containerized execution. These capabilities depend on your subscription. Three dimensions (CPU, GB of RAM, and parallel activities) define these quotas. These dimensions act as limits and define the maximum concurrent usage of those resources.

The Usage & Monitoring panel in your Launchpad reflects your quota. It’s a common pool of resources shared by all users. This means that the capacities used by a task won’t be available for others until that task finishes. If a new task requests more resources than those left available (on either one of the three quotas — CPU, RAM, and parallel activities), that task is queued until the resources it requests have been freed.

When a user starts a job requesting containerized execution, it launches one or several containers. This withdraws from your quota the CPU and RAM it uses, as well as one parallel activity. The quotas used for the containers will be freed when the job finishes. You must close (unload) webapps and notebooks to free the resources they’re using.

Note

Jobs on partitioned datasets launch several containers and count as several parallel activities. Dataiku will process as many partitions as available parallel activities in your quota simultaneously in a dedicated container. You can limit the maximum number of parallel activities requested by the recipe in the Advanced tab.