Generative AI on Google Cloud

106 / 120·notebook

Chapter 106

Getting Started with Vertex AI Python SDK for Gen AI Evaluation Service

NotebookPython 329 cells

In [ ]python · cell 1

python

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Getting Started with Vertex AI Python SDK for Gen AI Evaluation Service

Share to:


Author(s)	Jason Dai

Overview

YouTube Video: How to evaluate AI applications

In this tutorial, you will learn how to use the Vertex AI Python SDK for Gen AI Evaluation Service.

Costs

This tutorial uses billable components of Google Cloud:

Vertex AI

Learn about Vertex AI pricing and use the Pricing Calculator to generate a cost estimate based on your projected usage.

Getting Started

Install Vertex AI Python SDK for Gen AI Evaluation Service

In [ ]python · cell 8

python

%pip install --upgrade --user --quiet google-cloud-aiplatform[evaluation]

Restart runtime

To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.

The restart might take a minute or longer. After it's restarted, continue to the next step.

In [ ]python · cell 10

python

import IPython

app = IPython.Application.instance()
app.kernel.do_shutdown(True)

Authenticate your notebook environment (Colab only)

In [2]python · cell 13

python

import sys

if "google.colab" in sys.modules:
    from google.colab import auth

    auth.authenticate_user()

Set Google Cloud project information and initialize Vertex AI SDK

In [10]python · cell 15

python

PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
LOCATION = "us-central1"  # @param {type:"string"}
EXPERIMENT_NAME = "my-eval-task-experiment"  # @param {type:"string"}

if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    raise ValueError("Please set your PROJECT_ID")


import vertexai

vertexai.init(project=PROJECT_ID, location=LOCATION)

Import libraries

In [11]python · cell 17

python

import pandas as pd
from vertexai.evaluation import EvalTask, PointwiseMetric, PointwiseMetricPromptTemplate
from vertexai.preview.evaluation import notebook_utils

Set up eval metrics based on your criteria

As a developer, you would like to evaluate the text quality generated from an LLM based on two criteria: Fluency and Entertaining. You define a metric called text_quality using those two criteria.

In [20]python · cell 20

python

# Your own definition of text_quality.
text_quality = PointwiseMetric(
    metric="text_quality",
    metric_prompt_template=PointwiseMetricPromptTemplate(
        criteria={
            "fluency": "Sentences flow smoothly and are easy to read, avoiding awkward phrasing or run-on sentences. Ideas and sentences connect logically, using transitions effectively where needed.",
            "entertaining": "Short, amusing text that incorporates emojis, exclamations and questions to convey quick and spontaneous communication and diversion.",
        },
        rating_rubric={
            "1": "The response performs well on both criteria.",
            "0": "The response is somewhat aligned with both criteria",
            "-1": "The response falls short on both criteria",
        },
    ),
)

In [ ]python · cell 21

python

print(text_quality.metric_prompt_template)

Prepare your dataset

Evaluate stored generative AI model responses in an evaluation dataset.

In [22]python · cell 23

python

responses = [
    # An example of good text_quality
    "Life is a rollercoaster, full of ups and downs, but it's the thrill that keeps us coming back for more!",
    # An example of medium text_quality
    "The weather is nice today, not too hot, not too cold.",
    # An example of poor text_quality
    "The weather is, you know, whatever.",
]

eval_dataset = pd.DataFrame(
    {
        "response": responses,
    }
)

Run evaluation

With the evaluation dataset and metrics defined, you can run evaluation for an EvalTask on different models and applications, prompt templates, and many other use cases.

In [ ]python · cell 25

python

eval_result = EvalTask(
    dataset=eval_dataset, metrics=[text_quality], experiment=EXPERIMENT_NAME
).evaluate()

You can view the summary metrics and row-based metrics for each response in the EvalResult.

In [ ]python · cell 27

python

notebook_utils.display_eval_result(eval_result)

Clean up

Delete ExperimentRun created by the evaluation.

In [ ]python · cell 29

python

from google.cloud import aiplatform

aiplatform.ExperimentRun(
    run_name=eval_result.metadata["experiment_run"],
    experiment=eval_result.metadata["experiment"],
).delete()