How-to | Set up Kubernetes and Spark-on-Kubernetes#

You can set up Kubernetes (K8S) and Spark-on-Kubernetes for each instance linked to the template. If your Dataiku nodes connect to Kubernetes clusters, you must enable this setting.

Fleet Manager automatically downloads and configures the required packages and configurations to leverage Kubernetes clusters. This includes the Kubernetes plugin, the ability to attach container images, and define configurations for Kubernetes and Spark to point to the correct Azure Container Registry.

To do this:

  1. Under Setup actions, choose Setup Kubernetes and Spark-on-Kubernetes.

  2. To use the host IP instead of hostname for communication between pods and Dataiku, toggle Use private IP to On.