Generative AI on Google Cloud

113 / 120·notebook

Chapter 113

Evaluate Managed Agents

NotebookPython 323 cells

In [ ]python · cell 1

python

# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Evaluate Managed Agents

Authors
Kelsi Lakey
Jason Dai

Overview

This notebook demonstrates how to evaluate agents built with the Managed Agents API using the Gen AI Evaluation Service.

You will learn how to:

Generate conversation scenarios from your agent's configuration
Run inference to execute your agent against the generated scenarios
Evaluate the resulting conversation traces with prebuilt metrics
Evaluate existing interactions recorded with the Interactions API

Get started

Install Google Gen AI SDK and other required packages

In [ ]python · cell 7

python

%pip install --upgrade --quiet google-cloud-aiplatform[evaluation]

Authenticate your notebook environment

If you are running this notebook in Google Colab, run the cell below to authenticate your account.

In [ ]python · cell 9

python

import sys

if "google.colab" in sys.modules:
    from google.colab import auth

    auth.authenticate_user()

Set Google Cloud project information

To get started using Agent Platform, you must have an existing Google Cloud project and enable the Agent Platform API.

Learn more about setting up a project and a development environment.

In [ ]python · cell 11

python

import os

import agentplatform
from agentplatform import types as ap_types

# fmt: off
PROJECT_ID = "[your-project-id]"  # @param {type: "string", placeholder: "[your-project-id]", isTemplate: true}
# fmt: on
LOCATION = "global"  # @param {type: "string"}

if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    PROJECT_ID = str(os.environ.get("GOOGLE_CLOUD_PROJECT"))

client = agentplatform.Client(project=PROJECT_ID, location=LOCATION)

Set agent resource

Set the agent resource name for the Managed Agent you want to evaluate. To create a Managed Agent, see Create and manage agents.

In [ ]python · cell 13

python

AGENT_ID = "your-agent-id"  # @param {type: "string"}

AGENT_RESOURCE = f"projects/{PROJECT_ID}/locations/{LOCATION}/agents/{AGENT_ID}"

Generate conversation scenarios

Use generate_conversation_scenarios to automatically create diverse, multi-turn test scenarios from your agent's system instruction and tool definitions. This lets you start testing immediately without manually authoring test cases.

In [ ]python · cell 15

python

SCENARIO_COUNT = 5  # @param {type: "integer"}
GENERATION_INSTRUCTION = "Create agent scenarios"  # @param {type:"string"}

scenarios = client.evals.generate_conversation_scenarios(
    agent=AGENT_RESOURCE,
    config={
        "count": SCENARIO_COUNT,
        "generation_instruction": GENERATION_INSTRUCTION,
    },
)

scenarios.show()

Run inference

Use run_inference to execute your agent against the generated scenarios. Each scenario produces a conversation trace that captures the agent's full behavior, including tool calls, intermediate steps, and the final response.

In [ ]python · cell 17

python

MAX_TURNS = 3  # @param {type: "integer"}

inference_results = client.evals.run_inference(
    agent=AGENT_RESOURCE,
    src=scenarios,
    config={"user_simulator_config": {"max_turn": MAX_TURNS}},
)

inference_results.show()

Evaluate

Use evaluate to score the conversation traces with prebuilt metrics.

In [ ]python · cell 19

python

eval_result = client.evals.evaluate(
    dataset=inference_results,
    metrics=[
        ap_types.RubricMetric.MULTI_TURN_TASK_SUCCESS,
    ],
    agent=AGENT_RESOURCE,
)

eval_result.show()

Evaluate existing interactions

You can also evaluate interactions that have already been recorded with your agent. This is useful for assessing the quality of production conversations. You can create interactions by following Interact with agents.

In [ ]python · cell 21

python

INTERACTION_ID = "your-interaction-id"  # @param {type: "string"}

interactions_dataset = ap_types.EvaluationDataset(
    eval_cases=[
        ap_types.EvalCase(
            interactions_data_source=ap_types.InteractionsDataSource(
                interaction=(
                    f"projects/{PROJECT_ID}/locations/{LOCATION}"
                    f"/interactions/{INTERACTION_ID}"
                ),
                gemini_agent_config=ap_types.GeminiAgentConfig(
                    gemini_agent=AGENT_RESOURCE,
                ),
            ),
        ),
    ]
)

eval_result = client.evals.evaluate(
    dataset=interactions_dataset,
    metrics=[
        ap_types.RubricMetric.MULTI_TURN_TASK_SUCCESS,
    ],
    agent=AGENT_RESOURCE,
)

eval_result.show()

Cleaning up

To clean up all Google Cloud resources used in this project, you can delete the Google Cloud project you used for the tutorial.

Otherwise, you can delete the individual agent resource:

In [ ]python · cell 23

python

# Delete the agent
# client.agents.delete(name=AGENT_RESOURCE)