Reference | Managing elastic AI compute capacity

Dataiku Cloud manages the infrastructure of your instance and provides elastic AI computing capabilities that can be used on containerized execution. These capabilities depend on your subscription and are defined by quotas on three dimensions: CPU, GB of RAM, and parallel activities. These dimensions act as limits and define the maximum concurrent usage of those resources.

Your quota is reflected in the Usage & Monitoring panel in your Launchpad. It is a common pool of resources shared by all users. This means that the capacities used by a task won’t be available for others until that task is finished. If a new task requests more resources than those left available (on either one of the three quotas — CPU, RAM, and parallel activities), that task is queued until the resources it requests have been freed.

When a user starts a job requesting containerized execution, it launches one or several containers. This withdraws from your quota the CPU and RAM it uses, as well as one parallel activity. The quotas used for the containers will be freed when the job is finished. Webapps and notebooks need to be closed (unloaded) to free the resources they’re using.

Note

Jobs on partitioned datasets launch several containers and count as several parallel activities. As many partitions as available parallel activities in your quota will be processed simultaneously in a dedicated container. You can limit the maximum number of parallel activities requested by the recipe in the Advanced tab.