Concept | SQL code recipes#

An SQL recipe is useful for maintaining legacy code in a Dataiku Flow or for executing complex transformations that cannot be done in a single visual recipe, all while using an underlying database execution engine.

SQL recipes in Dataiku are executed in two different ways — either as a Query or as a Script.

SQL query#

To create an SQL query, specify the input and output datasets, and the storage location for the output dataset. This storage location can use a different database connection than the connection used by the input dataset.

Creating the query recipe opens up a code editor that contains a SELECT statement which you can edit to build your query. You can then Validate your code to check for syntax errors.

Before running the query, note that Dataiku will use the primary or most encompassing SELECT statement to create and insert the query results into an output table.

When you Run the query, Dataiku writes this table into the storage location that you specified for the output dataset.

Because Dataiku handles the table creation or deletion, insertion into the output table, and the automatic detection of the table schema, an SQL query allows you to focus on writing the main query.

SQL script#

In the case of an SQL Script, however, Dataiku does not manage the input or output tables. This means that your code must include: DROP, CREATE, and INSERT statements, to ensure that your script is reproducible.

Furthermore, The output of an SQL script must be written to the same database where the input data resides.

Recommendation#

In general, we recommend that you use the SQL Query over the SQL Script for the reasons just discussed. There are two exceptions to this recommendation:

When your SQL code has Common Table Expressions or WITH statements that cannot be re-written
When you are working with a data type that is not natively supported by Dataiku.