Chapter 99
Overview
NotebookPython 367 cells
In [ ]python · cell 1
python
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.Evaluate Generated Answers from Retrieval-Augmented Generation (RAG) for Question Answering with Gen AI Evaluation Service SDK
Share to:
| Author(s) | Jason Dai, Kelsi Lakey |
NOTE: This notebook has been tested in the following environment:
- Python version = 3.9
Overview
In this tutorial, you will learn how to use the use the Vertex AI Python SDK for Gen AI Evaluation Service to evaluate Retrieval-Augmented Generation (RAG) generated answers for Question Answering (QA) task.
RAG is a technique to improve groundness, relevancy and factuality of large language models (LLMs) by finding relevant information from the model's knowledge base. RAG is done by converting a query into a vector representation (embeddings), and then finding the most similar vectors in the knowledge base. The most similar vectors are then used to help generate the response.
This tutorial demonstrates how to use Gen AI Evaluation for a Bring-Your-Own-Response scenario:
-
The context was retrieved and the answers were generated based on the retrieved context using RAG.
-
Evaluate the quality of the RAG generated answers for QA task programmatically using the SDK.
The examples used in this notebook is from Stanford Question Answering Dataset SQuAD 2.0.
See also:
- Learn more about Vertex Gen AI Evaluation Service SDK.
Getting Started
Install Vertex AI SDK for Gen AI Evaluation
In [ ]python · cell 9
python
%pip install --upgrade --user --quiet google-cloud-aiplatform[evaluation]Restart runtime
To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.
The restart might take a minute or longer. After it's restarted, continue to the next step.
In [ ]python · cell 11
python
# import IPython
# app = IPython.Application.instance()
# app.kernel.do_shutdown(True)Authenticate your notebook environment (Colab only)
In [ ]python · cell 14
python
import sys
if "google.colab" in sys.modules:
from google.colab import auth
auth.authenticate_user()Increase quota (optional)
Increasing the quota may lead to better performance and user experience. Read more about this at online evaluation quotas.
Set Google Cloud project information and initialize Vertex AI SDK
In [ ]python · cell 18
python
PROJECT_ID = "[your-project-id]" # @param {type:"string"}
LOCATION = "us-central1" # @param {type:"string"}
EXPERIMENT = "rag-eval-01" # @param {type:"string"}
import vertexai
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
raise ValueError("Please set your PROJECT_ID")
vertexai.init(project=PROJECT_ID, location=LOCATION)Import libraries
In [ ]python · cell 20
python
import pandas as pd
from vertexai.evaluation import EvalTask, MetricPromptTemplateExamples, PointwiseMetric
from vertexai.preview.evaluation import notebook_utilsBring-Your-Own-Response Evaluation for RAG: Reference-Free (without Golden Answer)
Perform bring-your-own-response evaluation by assessing the generated answer's quality based on the context retrieved. It does not compare with golden answer.
Prepare your dataset
To evaluate the RAG generated answers, the evaluation dataset is required to contain the following fields:
- Prompt: The user supplied prompt consisting of the User Question and the RAG Retrieved Context
- Response: The RAG Generated Answer
Your dataset must include a minimum of one evaluation example. We recommend around 100 examples to ensure high-quality aggregated metrics and statistically significant results.
In [ ]python · cell 24
python
questions = [
"Which part of the brain does short-term memory seem to rely on?",
"What provided the Roman senate with exuberance?",
"What area did the Hasan-jalalians command?",
]
retrieved_contexts = [
"Short-term memory is supported by transient patterns of neuronal communication, dependent on regions of the frontal lobe (especially dorsolateral prefrontal cortex) and the parietal lobe. Long-term memory, on the other hand, is maintained by more stable and permanent changes in neural connections widely spread throughout the brain. The hippocampus is essential (for learning new information) to the consolidation of information from short-term to long-term memory, although it does not seem to store information itself. Without the hippocampus, new memories are unable to be stored into long-term memory, as learned from patient Henry Molaison after removal of both his hippocampi, and there will be a very short attention span. Furthermore, it may be involved in changing neural connections for a period of three months or more after the initial learning.",
"In 62 BC, Pompey returned victorious from Asia. The Senate, elated by its successes against Catiline, refused to ratify the arrangements that Pompey had made. Pompey, in effect, became powerless. Thus, when Julius Caesar returned from a governorship in Spain in 61 BC, he found it easy to make an arrangement with Pompey. Caesar and Pompey, along with Crassus, established a private agreement, now known as the First Triumvirate. Under the agreement, Pompey's arrangements would be ratified. Caesar would be elected consul in 59 BC, and would then serve as governor of Gaul for five years. Crassus was promised a future consulship.",
"The Seljuk Empire soon started to collapse. In the early 12th century, Armenian princes of the Zakarid noble family drove out the Seljuk Turks and established a semi-independent Armenian principality in Northern and Eastern Armenia, known as Zakarid Armenia, which lasted under the patronage of the Georgian Kingdom. The noble family of Orbelians shared control with the Zakarids in various parts of the country, especially in Syunik and Vayots Dzor, while the Armenian family of Hasan-Jalalians controlled provinces of Artsakh and Utik as the Kingdom of Artsakh.",
]
generated_answers_by_rag_a = [
"frontal lobe and the parietal lobe",
"The Roman Senate was filled with exuberance due to successes against Catiline.",
"The Hasan-Jalalians commanded the area of Syunik and Vayots Dzor.",
]
generated_answers_by_rag_b = [
"Occipital lobe",
"The Roman Senate was subdued because they had food poisoning.",
"The Galactic Empire commanded the state of Utah.",
]
eval_dataset_rag_a = pd.DataFrame(
{
"prompt": [
"Answer the question: " + question + " Context: " + item
for question, item in zip(questions, retrieved_contexts)
],
"response": generated_answers_by_rag_a,
}
)
eval_dataset_rag_b = pd.DataFrame(
{
"prompt": [
"Answer the question: " + question + " Context: " + item
for question, item in zip(questions, retrieved_contexts)
],
"response": generated_answers_by_rag_b,
}
)
eval_dataset_rag_aSelect and create metrics
You can run evaluation for just one metric, or a combination of metrics. For this example, we select a few RAG-related predefined metrics, and create a few of our own custom metrics.
Explore predefined metrics
In [ ]python · cell 27
python
# See all the available metric examples
MetricPromptTemplateExamples.list_example_metric_names()In [ ]python · cell 28
python
# See the prompt example for one of the pointwise metrics
print(MetricPromptTemplateExamples.get_prompt_template("question_answering_quality"))Create custom metrics
In [ ]python · cell 30
python
relevance_prompt_template = """
You are a professional writing evaluator. Your job is to score writing responses according to pre-defined evaluation criteria.
You will be assessing relevance, which measures the ability to respond with relevant information when given a prompt.
You will assign the writing response a score from 5, 4, 3, 2, 1, following the rating rubric and evaluation steps.
## Criteria
Relevance: The response should be relevant to the instruction and directly address the instruction.
## Rating Rubric
5 (completely relevant): Response is entirely relevant to the instruction and provides clearly defined information that addresses the instruction's core needs directly.
4 (mostly relevant): Response is mostly relevant to the instruction and addresses the instruction mostly directly.
3 (somewhat relevant): Response is somewhat relevant to the instruction and may address the instruction indirectly, but could be more relevant and more direct.
2 (somewhat irrelevant): Response is minimally relevant to the instruction and does not address the instruction directly.
1 (irrelevant): Response is completely irrelevant to the instruction.
## Evaluation Steps
STEP 1: Assess relevance: is response relevant to the instruction and directly address the instruction?
STEP 2: Score based on the criteria and rubrics.
Give step by step explanations for your scoring, and only choose scores from 5, 4, 3, 2, 1.
# User Inputs and AI-generated Response
## User Inputs
### Prompt
{prompt}
## AI-generated Response
{response}
"""In [ ]python · cell 31
python
helpfulness_prompt_template = """
You are a professional writing evaluator. Your job is to score writing responses according to pre-defined evaluation criteria.
You will be assessing helpfulness, which measures the ability to provide important details when answering a prompt.
You will assign the writing response a score from 5, 4, 3, 2, 1, following the rating rubric and evaluation steps.
## Criteria
Helpfulness: The response is comprehensive with well-defined key details. The user would feel very satisfied with the content in a good response.
## Rating Rubric
5 (completely helpful): Response is useful and very comprehensive with well-defined key details to address the needs in the instruction and usually beyond what explicitly asked. The user would feel very satisfied with the content in the response.
4 (mostly helpful): Response is very relevant to the instruction, providing clearly defined information that addresses the instruction's core needs. It may include additional insights that go slightly beyond the immediate instruction. The user would feel quite satisfied with the content in the response.
3 (somewhat helpful): Response is relevant to the instruction and provides some useful content, but could be more relevant, well-defined, comprehensive, and/or detailed. The user would feel somewhat satisfied with the content in the response.
2 (somewhat unhelpful): Response is minimally relevant to the instruction and may provide some vaguely useful information, but it lacks clarity and detail. It might contain minor inaccuracies. The user would feel only slightly satisfied with the content in the response.
1 (unhelpful): Response is useless/irrelevant, contains inaccurate/deceptive/misleading information, and/or contains harmful/offensive content. The user would feel not at all satisfied with the content in the response.
## Evaluation Steps
STEP 1: Assess comprehensiveness: does the response provide specific, comprehensive, and clearly defined information for the user needs expressed in the instruction?
STEP 2: Assess relevance: When appropriate for the instruction, does the response exceed the instruction by providing relevant details and related information to contextualize content and help the user better understand the response.
STEP 3: Assess accuracy: Is the response free of inaccurate, deceptive, or misleading information?
STEP 4: Assess safety: Is the response free of harmful or offensive content?
Give step by step explanations for your scoring, and only choose scores from 5, 4, 3, 2, 1.
# User Inputs and AI-generated Response
## User Inputs
### Prompt
{prompt}
## AI-generated Response
{response}
"""In [ ]python · cell 32
python
relevance = PointwiseMetric(
metric="relevance",
metric_prompt_template=relevance_prompt_template,
)
helpfulness = PointwiseMetric(
metric="helpfulness",
metric_prompt_template=helpfulness_prompt_template,
)Run evaluation with your dataset
In [ ]python · cell 34
python
rag_eval_task_rag_a = EvalTask(
dataset=eval_dataset_rag_a,
metrics=[
"question_answering_quality",
relevance,
helpfulness,
"groundedness",
"safety",
"instruction_following",
],
experiment=EXPERIMENT,
)
rag_eval_task_rag_b = EvalTask(
dataset=eval_dataset_rag_b,
metrics=[
"question_answering_quality",
relevance,
helpfulness,
"groundedness",
"safety",
"instruction_following",
],
experiment=EXPERIMENT,
)In [ ]python · cell 35
python
result_rag_a = rag_eval_task_rag_a.evaluate()
result_rag_b = rag_eval_task_rag_b.evaluate()Display evaluation results
View summary results
If you want to have an overall view of all the metrics from individual model's evaluation result in one table, you can use the display_eval_result() helper function.
In [ ]python · cell 38
python
notebook_utils.display_eval_result(
title="Model A Eval Result", eval_result=result_rag_a
)In [ ]python · cell 39
python
notebook_utils.display_eval_result(
title="Model B Eval Result",
eval_result=result_rag_b,
)Visualize evaluation results
In [ ]python · cell 41
python
eval_results = []
eval_results.append(("Model A", result_rag_a))
eval_results.append(("Model B", result_rag_b))In [ ]python · cell 42
python
notebook_utils.display_radar_plot(
eval_results,
metrics=[
"question_answering_quality",
"safety",
"groundedness",
"instruction_following",
"relevance",
"helpfulness",
],
)In [ ]python · cell 43
python
notebook_utils.display_bar_plot(
eval_results,
metrics=[
"question_answering_quality",
"safety",
"groundedness",
"instruction_following",
"relevance",
"helpfulness",
],
)View detailed explanation for an individual instance
If you need to delve into the individual result's detailed explanations on why a score is assigned and how confident the model is for each model-based metric, you can use the display_explanations() helper function. For example, you can set n=2 to display explanation of the 2nd instance result as follows:
In [ ]python · cell 45
python
notebook_utils.display_explanations(result_rag_a, num=2)You can also focus on one or a few metrics as follows.
In [ ]python · cell 47
python
notebook_utils.display_explanations(result_rag_b, metrics=["groundedness"])Bring-Your-Own-Response Evaluation for RAG: Referenced (with Golden Answer)
Perform bring-your-own-response evaluation by assessing the generated answer's quality based on the context retrieved and the golden answer provided in the reference.
Prepare your dataset
To evaluate the RAG generated answers, the evaluation dataset is required to contain the following fields:
- Prompt: The user supplied prompt consisting of the User Question and the RAG Retrieved Context
- Response: The RAG Generated Answer
- Reference: The Golden Answer groundtruth to compare model response to
In [ ]python · cell 51
python
questions = [
"Which part of the brain does short-term memory seem to rely on?",
"What provided the Roman senate with exuberance?",
"What area did the Hasan-jalalians command?",
]
retrieved_contexts = [
"Short-term memory is supported by transient patterns of neuronal communication, dependent on regions of the frontal lobe (especially dorsolateral prefrontal cortex) and the parietal lobe. Long-term memory, on the other hand, is maintained by more stable and permanent changes in neural connections widely spread throughout the brain. The hippocampus is essential (for learning new information) to the consolidation of information from short-term to long-term memory, although it does not seem to store information itself. Without the hippocampus, new memories are unable to be stored into long-term memory, as learned from patient Henry Molaison after removal of both his hippocampi, and there will be a very short attention span. Furthermore, it may be involved in changing neural connections for a period of three months or more after the initial learning.",
"In 62 BC, Pompey returned victorious from Asia. The Senate, elated by its successes against Catiline, refused to ratify the arrangements that Pompey had made. Pompey, in effect, became powerless. Thus, when Julius Caesar returned from a governorship in Spain in 61 BC, he found it easy to make an arrangement with Pompey. Caesar and Pompey, along with Crassus, established a private agreement, now known as the First Triumvirate. Under the agreement, Pompey's arrangements would be ratified. Caesar would be elected consul in 59 BC, and would then serve as governor of Gaul for five years. Crassus was promised a future consulship.",
"The Seljuk Empire soon started to collapse. In the early 12th century, Armenian princes of the Zakarid noble family drove out the Seljuk Turks and established a semi-independent Armenian principality in Northern and Eastern Armenia, known as Zakarid Armenia, which lasted under the patronage of the Georgian Kingdom. The noble family of Orbelians shared control with the Zakarids in various parts of the country, especially in Syunik and Vayots Dzor, while the Armenian family of Hasan-Jalalians controlled provinces of Artsakh and Utik as the Kingdom of Artsakh.",
]
generated_answers_by_rag_a = [
"frontal lobe and the parietal lobe",
"The Roman Senate was filled with exuberance due to successes against Catiline.",
"The Hasan-Jalalians commanded the area of Syunik and Vayots Dzor.",
]
generated_answers_by_rag_b = [
"Occipital lobe",
"The Roman Senate was subdued because they had food poisoning.",
"The Galactic Empire commanded the state of Utah.",
]
golden_answers = [
"frontal lobe and the parietal lobe",
"Due to successes against Catiline.",
"The Hasan-Jalalians commanded the area of Artsakh and Utik.",
]
referenced_eval_dataset_rag_a = pd.DataFrame(
{
"prompt": [
"Answer the question: " + question + " Context: " + item
for question, item in zip(questions, retrieved_contexts)
],
"response": generated_answers_by_rag_a,
"reference": golden_answers,
}
)
referenced_eval_dataset_rag_b = pd.DataFrame(
{
"prompt": [
"Answer the question: " + question + " Context: " + item
for question, item in zip(questions, retrieved_contexts)
],
"response": generated_answers_by_rag_b,
"reference": golden_answers,
}
)Create custom metrics
Create a custom metric to compare model response to the golden answer.
In [ ]python · cell 53
python
question_answering_correctness_prompt_template = """
You are a professional writing evaluator. Your job is to score writing responses according to pre-defined evaluation criteria.
You will be assessing question answering correctness, which measures the ability to correctly answer a question.
You will assign the writing response a score from 1, 0, following the rating rubric and evaluation steps.
### Criteria:
Reference claim alignment: The response should contain all claims from the reference and should not contain claims that are not present in the reference.
### Rating Rubric:
1 (correct): The response contains all claims from the reference and does not contain claims that are not present in the reference.
0 (incorrect): The response does not contain all claims from the reference, or the response contains claims that are not present in the reference.
### Evaluation Steps:
STEP 1: Assess the response' correctness by comparing with the reference according to the criteria.
STEP 2: Score based on the rubrics.
Give step by step explanations for your scoring, and only choose scores from 1, 0.
# User Inputs and AI-generated Response
## User Inputs
### Prompt
{prompt}
## Reference
{reference}
## AI-generated Response
{response}
"""In [ ]python · cell 54
python
question_answering_correctness = PointwiseMetric(
metric="question_answering_correctness",
metric_prompt_template=question_answering_correctness_prompt_template,
)Run evaluation with your dataset
In [ ]python · cell 56
python
referenced_answer_eval_task_rag_a = EvalTask(
dataset=referenced_eval_dataset_rag_a,
metrics=[
question_answering_correctness,
"rouge",
"bleu",
"exact_match",
],
experiment=EXPERIMENT,
)
referenced_answer_eval_task_rag_b = EvalTask(
dataset=referenced_eval_dataset_rag_b,
metrics=[
question_answering_correctness,
"rouge",
"bleu",
"exact_match",
],
experiment=EXPERIMENT,
)In [ ]python · cell 57
python
referenced_result_rag_a = referenced_answer_eval_task_rag_a.evaluate()
referenced_result_rag_b = referenced_answer_eval_task_rag_b.evaluate()Display evaluation results
View summary results
If you want to have an overall view of all the metrics evaluation result in one table, you can use the display_eval_result() helper function.
In [ ]python · cell 60
python
notebook_utils.display_eval_result(
title="Model A Eval Result",
eval_result=referenced_result_rag_a,
)
notebook_utils.display_eval_result(
title="Model B Eval Result",
eval_result=referenced_result_rag_b,
)Visualize evaluation results
In [ ]python · cell 62
python
referenced_eval_results = []
referenced_eval_results.append(
"Model A",
referenced_result_rag_a,
)
referenced_eval_results.append(
"Model B",
referenced_result_rag_b,
)In [ ]python · cell 63
python
notebook_utils.display_radar_plot(
referenced_eval_results,
metrics=[
"question_answering_correctness",
"rouge",
"bleu",
"exact_match",
],
)In [ ]python · cell 64
python
notebook_utils.display_bar_plot(
referenced_eval_results,
metrics=[
"question_answering_correctness",
"rouge",
"bleu",
"exact_match",
],
)View detailed explanation for an individual instance
If you need to delve into the individual result's detailed explanations on why a score is assigned and how confident the model is for each model-based metric, you can use the display_explanations() helper function. For example, you can set n=2 to display explanation of the 2nd instance result as follows:
In [ ]python · cell 66
python
notebook_utils.display_explanations(referenced_result_rag_a, num=2)In [ ]python · cell 67
python
notebook_utils.display_explanations(
referenced_result_rag_a, metrics=["question_answering_correctness"]
)