Concept | Where computation happens in Dataiku#

In the Dataiku Flow, you take datasets from different sources (SQL, file-system, HDFS) and you seamlessly apply recipes (like SQL queries, preparation scripts or computing predictions from a model). But what happens under the hood, where does the computation actually take place?

In this article, we will present you succinctly the architecture behind Dataiku, and explain why it matters.


Computation in Dataiku can take four main forms:

  • in-memory

  • streamed

  • runs in database (SQL)

  • runs on-Hadoop.

Which strategy is used depends on the dataset and the operation you are applying to the dataset.

Why does it matter? Well, sometimes if you apply the wrong type of recipe to your dataset, your computation will be painfully long or run out of memory! These issues particularly arise with big data.

Here is a table summarizing where the computation takes place for each operation performed in Dataiku:

Locally in Dataiku In Hadoop / Spark In SQL Database In Kubernetes / Docker
Visual Preparation Design In-memory
Visual Preparation Execution YES
Visual Recipes (other than Prepare) YES
Streaming or disk-copy
Hive, Spark, Impala
Python and R recipes YES
In-memory or streaming
PySpark, SparkR, sparklyr
Custom code with DSS helper API YES
In-memory or streaming
Spark-Scala recipe N/A YES N/A N/A
Charts YES YES
Hive, Impala (most charts)
(most charts)
Machine Learning train YES
scikit-learn, XGBoost, Keras/Tensorflow
scikit-learn, XGBoost, Keras/Tensorflow
Machine Learning execution YES
scikit-learn, XGBoost, MLlib, Keras/Tensorflow
scikit-learn, XGBoost, MLlib
scikit-learn (partial), XGBoost, MLlib (some models)
scikit-learn, XGBoost, MLlib, Keras/Tensorflow
Python, R, Scala notebooks YES
In-memory or streaming
Spark-Scala, PySpark, SparkR, sparklyr
Custom code with Dataiku helper API YES
SQL-like recipe or notebook N/A YES
Hive, Impala, SparkSQL

Some rules are clear and easy to understand. Take a SQL recipe. First, remember that the input datasets of your SQL recipe must be in a SQL database. When you hit run on a SQL recipe, Dataiku will send a query to the SQL database: read the input datasets, perform the SQL query, and finally write the output dataset if it is a SQL dataset, or streams the output otherwise.

Diagram showing how the computation for a join of two data sources in an SQL database take place in-database.


So the computation actually runs in-database. The idea is the same for Hive recipes, but here is applied to HDFS datasets, and everything runs in-cluster.

Diagram showing how the computation for data sources in a Hadoop cluster take place in-cluster.


Dataiku does not solely act as a server, sending queries to a SQL-database or a Hadoop-cluster. It also performs computation, what we call the DSS engine, which uses two main strategies:

  • In-memory: the data is stored in RAM.

  • Streamed: Dataiku reads the input dataset as a stream of rows, applies computations to the rows as they arrive, and writes the output datasets row per row.


The DSS engine also uses a third strategy for charts, but this a particular topic that is well-explained in the documentation.

In-memory processing#

By concept, in-memory computation can lead to out-of-memory situations on large datasets.

When can this annoying situation actually happen in Dataiku? As you can see in the above table, in-memory computation can happen in the following situations:

  • Python and R recipes

  • Editing a preparation recipe (or analysis)

  • Training a machine learning model

Visual data preparation#

When you are editing a preparation script (a Prepare recipe), Dataiku actually samples the dataset to ensure that the design computations will fit in RAM.

In a preparation script, you can add any processor and the editor is always fast and responsive, no matter the size of the original dataset because the computation is done only on a small sample.

Prepare recipe with Design Sample panel open.

Diagram showing how the data source is sampled for use in a Preparation recipe.

Note that you can change the sampling, and you could even use the whole dataset. But you should really think twice before switching the sampling to the whole dataset when you have a large dataset.

First of all, it misses the point of having a fast and responsive editing mode, and you may even run out of memory!


In a preparation script, do not use a design sample that is too large.

Data preparation execution#

You may have noticed that the strategy used in preparation script execution (when you click run) is completely different from the one used in editing the script.

Diagram showing how Dataiku determines where preparation processing is done (streamed or in-cluster).


Dataiku will choose the best strategy available (streamed or in-cluster) to execute the script, but never in-memory. Dataiku will of course apply the computation to the whole dataset, which is why it does not use in-memory processing.

Python and R recipes#

When you are using the pandas Python package, or R, you should keep in mind that both require in-memory data manipulation. As such, R and pandas are not well-suited tools for large data manipulation. If the input or output dataset cannot fit in RAM, you may run out of memory.

Diagram showing how R and Python processing is done in-memory.



You cannot apply R recipes or Python recipes with pandas to datasets that are too big.

In Python, several workarounds are available:

  • Use streamed (per-row) processing instead of Dataframe processing

  • Use chunked processing (read dataframes by blocks)


The reference documentation has details on these implementations in Python recipes.

How to deal with big data?#

Python+Pandas or R remain useful when working with big data.

For example, sometimes, you can limit yourself to a sample of the dataset. It’s a good way to prototype your code. Dataiku allows you to stream a random sample of the data and import it directly as a Pandas dataframe (by passing the sampling and limit arguments in the get_dataframe() function) or R dataframe (by passing the sampling argument in the read.dataset() function).

But afterwards, if you want to apply the computation to the whole dataset, you will have to switch to another computing strategy.

Dataiku offers many alternatives:

  • As mentioned, you could use a streamed implementation of your Python code.

  • A Python UDF processor in a preparation recipe could work.

  • In some cases, using a SQL recipe may be a very good option.

However, when dealing with big data, you want to push the computation to the data as much as possible. Computing engines like Hadoop on HDFS follow this principle, and are specifically designed for big-data computing. Hive recipes, which run in-cluster, are well-suited for big data computing.

Dataiku allows you to easily switch between computing engines. For instance, suppose your current dataset is on the filesystem, but is too large for a Python recipe. You can use a Sync recipe to push your dataset to HDFS, and opt for a Hive or PySpark recipe instead.