Using Gen AI Evaluation SDK for Google Observability Gen AI multi-modal datasets

NotebookPython 316 cells

In [ ]python · cell 1

python

# Copyright 2025 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Author
Matthew Yun

Overview

This notebook demonstrates how users can use the Vertex AI SDK Gen AI evaluation service for evaluating your Gen AI multimodal content stored in Google Observability.

The Vertex AI SDK allows users to run evaluations on their Gen AI models prompts and responses. This now includes the ability for users to run evaluations against the Gen AI data that has been stored in Google Cloud Storage (GCS) within Observability following OpenTelemetry semantic conventions.

Prompt, response, and system instruction data in Google Observability are stored in separate GCS references. This notebook gives an example for how users can read in their data from GCS and run an evaluation using the Vertex AI SDK.

Get started

Install Vertex AI SDK for Gen AI Evaluation Service

In [ ]python · cell 7

python

%pip install --upgrade "google-cloud-aiplatform[evaluation]>=1.122.0" --force-reinstall --quiet --no-warn-conflicts

Authenticate your notebook environment (Colab only)

If you're running this notebook on Google Colab, run the cell below to authenticate your environment.

In [ ]python · cell 9

python

import sys

if "google.colab" in sys.modules:
    from google.colab import auth

    auth.authenticate_user()

Set Google Cloud project information

To get started using Vertex AI, you must have an existing Google Cloud project and enable the Vertex AI API.

Learn more about setting up a project and a development environment.

In [ ]python · cell 11

python

# Use the environment variable if the user doesn't provide Project ID.
import os

# fmt: off
PROJECT_ID = ""  # @param {type: "string", placeholder: "[your-project-id]", isTemplate: true}
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    PROJECT_ID = str(os.environ.get("GOOGLE_CLOUD_PROJECT"))
LOCATION = "us-central1"  # @param {type: "string", placeholder: "us-central1", isTemplate: true}
# fmt: on
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", LOCATION)

from vertexai import Client, types

client = Client(project=PROJECT_ID, location=LOCATION)

Evaluation Dataset

Load in Google Observability Gen AI dataset

We will need to read in the data stored in Google Cloud Storage and prepare for evaluation.

In [ ]python · cell 14

python

# fmt: off
INPUT_SOURCE = ""  # @param {type: "string", placeholder: "[your-input-source]", isTemplate: true}
OUTPUT_SOURCE = ""  # @param {type: "string", placeholder: "[your-output-source]", isTemplate: true}
SYSTEM_INSTRUCTION_SOURCE = ""  # @param {type: "string", placeholder: "[your-system-instruction-source]", isTemplate: true}
# fmt: on

eval_case = types.ObservabilityEvalCase(
    input_src=INPUT_SOURCE,
    output_src=OUTPUT_SOURCE,
    system_instruction_src=SYSTEM_INSTRUCTION_SOURCE,
    api_client=client,
)
eval_dataset = types.EvaluationDataset.load_from_observability_eval_cases([eval_case])

eval_dataset.show()

Run Evaluation

Evaluate the responses using the GENERAL_QUALITY adaptive rubric-based metric by default.

In [ ]python · cell 16

python

eval_result = client.evals.evaluate(dataset=eval_dataset)

eval_result.show()