Chapter 100
Evaluate a Translation Model
NotebookPython 329 cells
In [ ]python · cell 1
python
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.Evaluate a Translation Model
Share to:
| Author(s) | Caleb Mbakwe |
Overview
In this tutorial, you will learn how to use the Vertex AI Python SDK for Gen AI Evaluation Service to measure the translation quality of your LLM responses using BLEU, MetricX and COMET.
Getting Started
Install Vertex AI Python SDK for Gen AI Evaluation Service
In [ ]python · cell 8
python
%pip install --upgrade --user --quiet google-cloud-aiplatform[evaluation]Restart runtime
To use the newly installed packages in this Jupyter runtime, you might need to restart the runtime. You can do this by running the cell below, which restarts the current kernel.
The restart might take a minute or longer. After it's restarted, continue to the next step.
In [ ]python · cell 10
python
# import IPython
# app = IPython.Application.instance()
# app.kernel.do_shutdown(True)Authenticate your notebook environment (Colab only)
In [ ]python · cell 13
python
import sys
if "google.colab" in sys.modules:
from google.colab import auth
auth.authenticate_user()Set Google Cloud project information and initialize Vertex AI SDK
In [ ]python · cell 15
python
PROJECT_ID = "[your-project-id]" # @param {type:"string"}
LOCATION = "us-central1" # @param {type:"string"}
EXPERIMENT_NAME = "my-eval-task-experiment" # @param {type:"string"}
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
raise ValueError("Please set your PROJECT_ID")
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)Import libraries
In [ ]python · cell 17
python
# General
import pandas as pd
# Main
from vertexai import evaluation
from vertexai.evaluation.metrics import pointwise_metricIn [ ]python · cell 18
python
# @title Helper functions
from IPython.display import Markdown, display
def display_eval_result(eval_result, metrics=None, model_name=None, rows=0):
if model_name is not None:
display(Markdown("## Eval Result for %s" % model_name))
"""Display the evaluation results."""
summary_metrics, metrics_table = (
eval_result.summary_metrics,
eval_result.metrics_table,
)
metrics_df = pd.DataFrame.from_dict(summary_metrics, orient="index").T
if metrics:
metrics_df = metrics_df.filter(
[
metric
for metric in metrics_df.columns
if any(selected_metric in metric for selected_metric in metrics)
]
)
metrics_table = metrics_table.filter(
[
metric
for metric in metrics_table.columns
if any(selected_metric in metric for selected_metric in metrics)
]
)
# Display the summary metrics
display(Markdown("### Summary Metrics"))
display(metrics_df)
if rows > 0:
# Display samples from the metrics table
display(Markdown("### Row-based Metrics"))
display(metrics_table.head(rows))Set up eval metrics for your data.
In [ ]python · cell 21
python
metrics = [
"bleu",
# See https://github.com/googleapis/python-aiplatform/blob/4e332de345ef3cc4d5f99f11d6499a3334e3345f/vertexai/evaluation/metrics/pointwise_metric.py#L82 for options.
pointwise_metric.Comet(version="COMET_22_SRC_REF"), # Reference based COMET
# See https://github.com/googleapis/python-aiplatform/blob/4e332de345ef3cc4d5f99f11d6499a3334e3345f/vertexai/evaluation/metrics/pointwise_metric.py#L115 for options.
pointwise_metric.MetricX(version="METRICX_24_SRC"), # Reference free MetricX.
]Prepare your dataset
Evaluate stored generative AI model responses in an evaluation dataset.
In [ ]python · cell 23
python
sources = [
"Dem Feuer konnte Einhalt geboten werden",
"Schulen und Kindergärten wurden eröffnet.",
]
responses = [
"The fire could be stopped",
"Schools and kindergartens were open",
]
references = [
"They were able to control the fire.",
"Schools and kindergartens opened",
]
eval_dataset = pd.DataFrame(
{
"source": sources,
"response": responses,
"reference": references,
}
)Run evaluation
With the evaluation dataset and metrics defined, you can run evaluation for an EvalTask on different models and applications, and many other use cases.
In [ ]python · cell 25
python
eval_task = evaluation.EvalTask(
dataset=eval_dataset, metrics=metrics, experiment=EXPERIMENT_NAME
)
eval_result = eval_task.evaluate()You can view the summary metrics and row-based metrics for each response in the EvalResult.
In [ ]python · cell 27
python
display_eval_result(eval_result, rows=2)Clean up
Delete ExperimentRun created by the evaluation.
In [ ]python · cell 29
python
from google.cloud import aiplatform
aiplatform.ExperimentRun(
run_name=eval_result.metadata["experiment_run"],
experiment=eval_result.metadata["experiment"],
).delete()