How-to | Configure an external metastore (AWS Glue Data Catalog)#

DSS can interact with an AWS Glue Data Catalog when DSS is deployed on AWS. You’ll need an existing S3 connection from which you will retrieve credentials.

Step 1 - Enable the AWS Glue Data Catalog#

The first step is to enable the AWS Glue Data Catalog.

  1. From the Administration menu, navigate to the Settings tab.

  2. Under Compute & Scaling select Metastore catalogs.

  3. From the Metastore kind menu arrow select AWS Glue.

Metastore catalogs page in the Compute and Scaling section of the Settings tab within DSS settings.

Step 2 - Define an S3 Connection to the AWS Glue Data Catalog#

The next step is to configure an S3 connection to use the AWS Glue Data Catalog.

Access to the AWS Glue Data Catalog will be created through the credentials defined in the S3 connection, which may be per-user credentials. For this reason, Dataiku recommends authentication through an S3 connection.

When submitting Spark jobs, DSS will automatically configure Spark to use AWS Glue with the appropriate credentials.

To configure an S3 connection to the AWS Glue Data Catalog:

  1. Create an S3 connection or obtain the information of an existing S3 connection that you want to rely on.

  2. Set the Metastore kind option to AWS Glue.

  3. Set Glue Auth to Use AWS credentials from a connection.

  4. Enter the S3 connection name and then select Save.

Metastore catalogs page in the Compute and Scaling section of the Settings tab within DSS settings.

Step 3 - Configure the Connection to Sync with the Metastore#

In order to synchronize the datasets created through a specific connection to an external metastore, the connection must be configured.

  1. Select Keep datasets synced.

  2. Select a Fallback metastore DB value to point these metadata to the proper zone.

Once the connection is configured, every new dataset metadata will be enriched inside the external metastore, allowing the import of it from the catalog menu (via the Connections explorer tab).

For example, the following connection named “athena” is pointing to an AWS Glue catalog:

AWS Glue catalog connection in the Connections explorer tab of the Catalog.

External datasets (datasets that are generated outside of DSS) can also be imported this way.