Tutorial | Agent Run Inspector

Get started

The Agent Run Inspector gives you real-time feedback and lets you dig into details of an agent test run, such as which tools it called, which inputs it passed, and what steps it took.

In this tutorial, you will learn how to examine these details and debug issues as you build and test an agent.

Objectives

In this tutorial, you will:

• Evaluate real-time feedback from Run Inspector as you build and test a Structured Visual Agent.

• View a failed run in Run Inspector and debug the issue.

• Use Run Inspector to understand how an agent stores and uses information in memory.

Prerequisites

To complete this tutorial, you will need:

• Dataiku 15.0 or later.

• An Advanced Analytics Designer or Full Designer user profile.

• A connection to at least one Generative AI model that supports tool calling and one that supports embedding. See supported models for tool calling and embedding (note that some models support both). Your administrator must configure the connection(s) in the Administration panel > Connections > New connection > LLM Mesh.

• An internal code environment for retrieval augmented generation and agents. Your administrator must set this up in the Administration panel > Code Envs > Internal envs setup > Retrieval augmented generation code environment.

• Knowledge of Structured Visual Agents in Dataiku.

Create the project

1. If needed, from the Dataiku Design homepage, use the () icon to switch to the Designer view.

2. Click + New Project > Learning projects.

3. Search for and select Agent Run Inspector.

4. If needed, change the folder into which the project will be installed, and click Create.

5. From the project homepage, click Go to Flow (or type g + f).

Note

You can also download the starter project from this website and import it as a ZIP file.

Use case summary

Your IT help desk receives hundreds of support tickets every week. You are tasked with building an agent to classify the tickets by urgency to help your team triage requests.

You will build a Structured Visual Agent that:

• Takes a ticket ID as input from the user and retrieves the ticket details from the it_tickets dataset.

• Saves the ticket details to temporary memory.

• Classifies the ticket’s urgency as Critical, High, Medium, or Low.

The project contains the dataset it_tickets and the agent tool ticket_lookup for looking up information in the dataset.

Debug a tool error

As you create the agent, you can send test chats and evaluate real-time feedback from the timeline in the chat and diagram. You can test each block as you build to make sure the components work.

1. Go to GenAI > Agents and click + New Agent.

2. Choose Structured Visual Agent.

3. Name it IT ticket triage agent.

4. Click Create.

Create block 1: Receive ticket ID

The first block obtains a ticket ID from the user and saves it to state. The next block uses that ID to retrieve the ticket details.

1. Click + Create Block and add an Agentic Loop block.

2. Name it receive_ticket_id and click Create.

3. Configure the following settings in the block:

• LLM: Choose from your configured options.

• Instructions:

Ask the user for a ticket ID if they have not provided one. Ticket IDs follow the format TKT-XXXX (for example, TKT-1042).

Once you have a valid ticket ID, save it to state as ticket_id and proceed. Do not proceed without a confirmed ticket ID.

• Tools: Check the box next to Read/Write State.

• Next block & Exit conditions: Add an exit condition of Type: State has keys and add the State keys of ticket_id.

4. Click Save.

Create block 2: Fetch ticket data

The next block takes the ticket ID and uses the agent tool to look up the rest of the data about that ticket.

1. Under Next block & Exit conditions > Next block, select + Create new block.

2. Choose a Manual Tool Call block, name it fetch_ticket_data, and click Create.

3. In the block, configure the following settings:

• Tool: Select ticket_lookup.

• Output: Change to Save to state, and add the Output state key of ticket_data.

4. Click Save.

Test and debug the blocks

Now test the blocks and track their steps, starting with the timeline in the chat.

1. Check the Show timeline box at the top of the chat.

2. Type TKT-1042 in the chat and send it.

3. The agent should return an error message. The second block appears red in the chat, and the timeline event displays a red X.

4. Review the agent diagram and note the red X on the fetch_ticket_data block. This means the agent couldn’t execute the block.

5. Click on the red fetch_ticket_data event in the chat response window. The Run Inspector panel appears on the right.

6. Check the Input/Output tab, which says that no input or output was captured.

The block failed before producing an output. The timeline and Inspector indicate that the failure occurred during the tool call, so the next step is to check the tool configuration.

1. To investigate, go to the GenAI menu () > Agent Tools.

2. Open the ticket_lookup tool.

3. Check the tool configuration and notice the tool isn’t connected to any dataset.

4. Set the Dataset to it_tickets.

5. Set the Lookup columns to ticket_id.

6. Click Save.

That should fix the tool error. Now return to the agent and test again.

1. Return to the agent (GenAI menu () > Agents, and open the agent).

2. Click Reset Chat and Confirm.

3. Make sure the Show timeline box is checked.

4. Send the message TKT-1042.

5. Review the timeline by clicking on the dropdown arrow next to the fetch_ticket_data block in the chat response. You should see a warning and another failure of the tool call.

6. Click on the failed tool call to open the Run Inspector window.

7. Review the Tool Output and note the message.

This time, the tool call has failed because there is no filter set in the block. The block needs a filter so the agent saves only the relevant data for the ticket at hand.

1. Return to the fetch_ticket_data block in the agent builder.

2. Next to Tool arguments, click + Add Argument.

3. Add the following CEL expression:

{"column": "ticket_id", "operator": "EQUALS", "value": state.ticket_id}

4. Click Save.

Tip

As you move between areas in the agent editor, you can resize the panes so it is easier to view and edit the agent. You can also use the Build, Test and View buttons to select only certain panes.

The filter means the agent will save only information about the given ticket. Now test the agent again.

1. Reset the chat and send TKT-1042.

2. As the agent processes the request, watch its reasoning in the timeline.

3. After the agent finishes, you should see green checkmarks indicating that both blocks and the tool have successfully executed.

4. Click the dropdown arrow on the fetch_ticket_data and click on the ticket_lookup tool to view the inspection panel.

5. In the panel, note the Input/Output tab showing the chat messages and input from the tool.

Note

Your results will vary depending on the model used and can also differ in every run.

Track memory across the agent

Each of the first two blocks saves some information about the given ticket into a temporary memory called state.

Run Inspector lets you track what information is saved to the state and another type of temporary memory called scratchpad. In this section, you’ll practice tracking memory changes through the agent.

View context

1. In the timeline from the last run, click on the receive_ticket_id block and view the inspector panel.

2. Click on the Context tab. You should see that the block added a value to state.ticket_id.

3. Next move in the timeline to the fetch_ticket_data block and view it in the inspector panel.

4. Click on the Context tab. You should see the ticket_data values added as a long block of key-value pairs.

The fetch_ticket_data block saved the entire ticket data into state. You can also save some individual column values into state to make it easier to view in the inspector and use those columns later in the agent.

Create block 3: Save ticket details to state

Adding a dedicated state-writing step makes the agent easier to debug because you can see the exact values stored at this point in the run before any further processing happens.

1. In the fetch_ticket_data block, under Next block, create a new next block.

2. Choose Set State Entries block, name it save_ticket_data, and click Create.

3. Add Entries for each of the following pairs:

State key	CEL expression
category	state.ticket_data.category
description	state.ticket_data.description
status	state.ticket_data.status

4. Click Save.

View context changes

Now you can view the values the agent has gathered for these columns using the Run Inspector.

1. Reset the chat, then send TKT-1042.

2. In the timeline, click on the save_ticket_data block.

3. In the inspector panel, click on Context. You should see the values added for each of the columns.

So far, you have reset the chat each time you use it. The Run Inspector can also track changes in the context if you look up multiple tickets in the same chat.

1. In the chat, send TKT-1001 (without resetting the chat).

2. In the timeline, click on the save_ticket_data block.

3. In the inspector panel, click on Context. This time, you should see that three of the values have changed.

4. Back in the timeline, click on the receive_ticket_id block and view the context in the inspector panel.

5. Note that the ticket_id saved to state has also changed.

Inspect agent execution

The final block in this agent will classify the urgency of the IT ticket. Then you can view the logic and steps the agent takes in a full test run.

Create block 4: Make a decision

This block will act on detailed instructions about classifying the IT tickets.

1. From the save_ticket_data block, create a next block.

2. Choose Agentic Loop, name it make_decision, and click Create.

3. Configure the following settings in the block:

• Choose an LLM. The agent will perform best if you use the same LLM as the other Agentic Loop block.

• In Instructions, copy and paste:

You are an IT support triage assistant. You will classify the urgency of a ticket based on the data saved in {state.ticket_data}.

Classify the ticket's urgency based on the description and any scope indicators present (such as number of users affected, revenue impact, security implications, or production system involvement).

Urgency levels:
- Critical: Affects many users, involves production systems, revenue loss, or
security breach. Requires immediate escalation. Do not attempt self-service.
- High: Significant disruption to a team or a time-sensitive individual issue.
Escalate to the assigned team.
- Medium: Individual issue with moderate impact. Provide KB guidance and
offer escalation if self-service fails.
- Low: Minor inconvenience. Provide KB self-service steps.

Save your classification to state as urgency_classification.

• Under Tools, check the box for Read/Write State.

• Under Next block & Exit conditions, add an exit condition of State has keys and add the State keys urgency_classification.

4. Click Save.

Test block 4

As you test the final block, use the Run Inspector to inspect the agent’s activity.

1. Reset the chat and send TKT-1116.

2. Verify in the agent diagram that all blocks and tools executed (have green checkmarks).

3. Click the dropdown arrows next to blocks in the timeline (receive_ticket_id, fetch_ticket_data, and make_decision).

4. Review the tool calls and steps under each block, noting the agent’s logical reasoning, such as Ticket TKT-1116 Captured or Let me look up the ticket data. These messages will depend on your LLM.

5. Under the make_decision block, click on the dku_state__get tool and view the inspection panel. This is a tool created within the block to read the ticket_data state.

6. Click on the dku_state_set tool and verify that the agent has created a state key of urgency_classification and assigned it a value.

You’ve now used all three elements of the Run Inspector — the timeline, diagram notations, and inspection panel — to gain insight into an agent’s reasoning, memory, and steps.