Generative AI and Large Language Models (LLMs)¶
In these resources, you will learn how to unlock text insights in Dataiku through Generative AI technology, specifically large language models (LLMs).
Reference | LLM connections¶
Dataiku provides connections to a variety of Generative AI services, both hosted and self-hosted, that are required to build GenAI applications.
The connection can then be leveraged with Dataiku’s visual recipes, such as Prompt Studios, Prompt recipe, Natural Language Processing (NLP) recipes, and the Embed recipe, or via the Dataiku Python API for usage in code recipes, code studios, notebooks, and web applications.
Instance administrators can configure connections to LLMs in:
The Connections menu > Add a connection > Read Only Data Sources from the launchpad of Dataiku Cloud.
The Administration menu > Connections > New Connection > LLM Mesh in an on-premise installation.
Hosted LLM connections¶
Supported connections to Generative AI services include models such as OpenAI, Hugging Face, Cohere, etc.
Connection settings vary depending on the requirements of the providers.
Hugging Face connections¶
Users can host, manage, and use open source Hugging Face models locally with Dataiku’s Hugging Face connection. This connection includes containerized execution and integration with NVIDIA GPUs to make the model execution simpler. Users can start experimenting with Hugging Face models without downloading the model into the code environment’s resources.
Administrators can configure which Hugging Face models are available for different tasks, and other settings such as containerized execution, in the connection settings.
Custom LLM connections¶
Users who have developed their own LLMs with API endpoints can integrate these custom services into Dataiku via the Custom LLM connection. Setting up a Custom LLM connection requires developing a specific plugin using the Java LLM component to map the endpoint’s behavior.
FAQ | Are the LLM connections secure?¶
The LLM connections act as a secure API gateway between your applications and the underlying services. It sits between the Dataiku components and an API endpoint, transmitting the required requests to the relevant service.
This gateway is set by an administrator and allows you to:
Control the user access.
Enable security measures such as rate limiting, network, and tuning parameters (to define the number of retries, timeout, and maximum parallelism).
Apply security policies.
Centrally manage access keys, models, and permissions¶
When configuring the LLM connections, the administrator uses an API Key (or references an S3 connection for AWS Bedrock) and defines which models are made available.
Note
A connection to Hugging Face does not require any credential or API key.
This ensures that access keys are centrally managed and that only approved services are used.
The administrators also configure the security settings at the connection level to define which user groups can use the connection, thus ensuring the underlying services are properly managed.
Use filters to detect and manage sensitive data¶
Still at the connection level, the administrators can define a certain number of filters and moderation techniques to restrict the most sensitive data sent to self-hosted, private LLMs for example.
The LLM connections currently provide two main filters to detect and manage sensitive data:
PII (Personal Identifiable Information) detection
Forbidden terms
PII detection¶
The PII detection allows you to filter every query and define an action to take (reject the query, replace the PII with a placeholder, etc.).
It leverages the presidio library.
Forbidden terms¶
The administrator can add forbidden term filters for both queries and responses, to remove specific data from input or outputs.
The filter defines the source dataset that contains the forbidden terms and a matching mode.
FAQ | Can I detect toxic language?¶
Since LLMs work by predicting the next most likely word in a sequence, they can propagate toxic language or otherwise harmful biases that they have learned during the training process.
Toxicity detection is key to ensure that the model’s input or output does not contain inappropriate or offensive content, and define appropriate remedies if they do.
To mitigate this risk, the Dataiku administrators can set the LLM connection to moderate the content both for queries and responses, leveraging dedicated content moderation models to filter input/output.
If toxic content is spotted, whether in a query or a response, the API call fails with an explicit error.
FAQ | How can I assess the cost?¶
You can use the Compute resource usage (CRU) solution to keep track of the cost, thanks to the logs from the LLM connection (see FAQ | Can I audit the LLM connections?).
It enables you to monitor performance, diagnose issues, build fine-grained cost reports, aggregate costs per application, recipe, user or project, and define appropriate mitigation measures such as alerts.
FAQ | Can I use the cache to reduce the cost?¶
The LLM Mesh provides caching capabilities that an administrator can configure at the LLM connection level. By caching responses to common queries, the LLM Mesh avoids unnecessary re-generation of answers, reducing avoidable costs.
To enable the use of cache:
In the Administration panel > Connections, select your LLM connection or create a new one (New connection > LLM Mesh).
Under Usage Control, check the Caching option.
FAQ | Can I audit the LLM connections?¶
Dataiku administrators can configure the level of auditing and logging to keep a complete record of queries and responses. They can either:
Disable the audit.
Log only the metadata.
Log all the queries/responses.
Logs are then available for download from the administration panel or can be used directly in a Dataiku project for further analysis, for example, to ensure extensive cost control and monitoring. For further information, see the FAQ | How can I assess the cost?.