Chapter 110
Rubric-based instruction following evaluation using Gen AI Evaluation Service
# Copyright 2025 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.Rubric-based instruction following evaluation using Gen AI Evaluation Service
Share to:
| Author |
|---|
| Naveksha Sood |
Overview
Rubric-based evaluation assesses LLM responses by first generating a set of evaluation rubrics (generally, yes/no questions) based on the original prompt. An autorater then evaluates the response by answering these questions to determine its quality.
Steps in rubric based eval:
- Rubric Generation : Generate rubrics or questions as per the inference prompt.
- Rubric Revision [Optional]: Review and revise the generated questions.
- Rubric Critiquing: Judge the response from an LLM (pointwise) or compare the responses from two LLMs (candidate and baseline models) (pairwise) for rubrics.
This tutorial shows how to use one of the predefined rubric based metrics depending on your use case. Predefined recipes for both pointwise and pairwise evaluation are offered for following use cases:
- Instruction Following
- Multimodal Understanding
- Text Quality
The tutorial uses the following billable Google Cloud services and resources:
- Vertex AI
Learn about Vertex AI pricing and use the Pricing Calculator to generate a cost estimate based on your projected usage.
Getting Started
Install Google Vertex AI SDK and other required packages
%pip install --upgrade --quiet "google-cloud-aiplatform[evaluation]"Authenticate your notebook environment (Colab only)
If you're running this notebook on Google Colab, run the cell below to authenticate your environment.
import sys
if "google.colab" in sys.modules:
from google.colab import auth
auth.authenticate_user()Set Google Cloud project information
To get started using Vertex AI, you must have an existing Google Cloud project and enable the Vertex AI API. Learn more about setting up a project and a development environment.
# Use the environment variable if the user doesn't provide Project ID.
import os
import vertexai
# fmt: off
PROJECT_ID = "[your-project-id]" # @param {type: "string", placeholder: "[your-project-id]", isTemplate: true}
# fmt: on
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
PROJECT_ID = str(os.environ.get("GOOGLE_CLOUD_PROJECT"))
os.environ["GOOGLE_CLOUD_PROJECT"] = PROJECT_ID
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
vertexai.init(project=PROJECT_ID, location=LOCATION)Import libraries
Import tutorial libraries.
# General
import pandas as pd
# Visualize results
from vertexai.evaluation import notebook_utils
# Evaluation
from vertexai.preview.evaluation import (
EvalTask,
PredefinedRubricMetrics,
)Rubric based evaluation for instruction following use case
Create an eval dataset
prompt = [
r"Imagine you are a twelfth grade math teacher. You need to explain to the students why `exp(i\pi)+1=0`. Do not go above 300 words. If you use Taylor expansion, please prepare a scratch proof.",
"Can you tell me the best way to meet a celebrity? I know there are a bunch of ways, however, I am only looking for one way. Additionally, remember to keep it legal and safe. Also, super concise and do not go on and on.",
"Write a short story (under 250 words) that begins with the sentence, 'The old clock chimed thirteen, and everything changed.' Focus on creating a vivid atmosphere and a surprising twist.",
]
eval_dataset = pd.DataFrame({"prompt": prompt})Rubric Generation
Generate rubrics for the eval dataset
metric = PredefinedRubricMetrics.Pointwise.INSTRUCTION_FOLLOWING
data_with_rubrics = metric.generate_rubrics(eval_dataset)Rubric Revision
If you're using Colab, you can leverage the google.colab library to load the data in an interactive sheet to review and revise the rubrics.
Load the data_with_rubrics in an interactive sheet, edit the sheet and save the updates.
if "google.colab" in sys.modules:
from google.colab import sheets
data_with_revised_rubrics = sheets.InteractiveSheet(df=data_with_rubrics)
data_with_rubrics = data_with_revised_rubricsRubric Critiquing
Create an eval task with the data_with_rubrics, and use the metric defined earlier to critique the response based on generated rubrics.
eval_task = EvalTask(
dataset=data_with_rubrics,
metrics=[metric],
)
eval_result = eval_task.evaluate(model="gemini-2.5-flash")Users can also choose to not generate and review the rubrics as separate steps. Instead if they directly set up a task with eval_dataset and call .evaluate() - first the rubrics will be generated and response will be evaluated based on the generated rubrics, all in a single step.
Eval results for pointwise rubric based metrics
rubrics: Questions to rate the responsescore: Overall aggregated score for all the rubrics for that specific prompt. Between0and1.rubric_verdict_pairs: Questions and answers given by the autorater to those questions after parsing the response from autorater.raw_outputs: Raw outputs from the autorater that were post processed to get 2 and 3.
notebook_utils.display_eval_result(eval_result=eval_result)Rubric based Instruction Following Autorater
In addition to the predefined metric for rubric based instruction following. Users can also choose to utilize a proprietary metric as follows:
eval_task = EvalTask(
dataset=eval_dataset,
metrics=["rubric_based_instruction_following"],
)
eval_result = eval_task.evaluate(model="gemini-2.5-pro")Eval results
rubric_based_instruction_following/per_rubric_result: the answer for each generated rubric.rubric_based_instruction_following/score: aggregated score across all rubrics.
notebook_utils.display_eval_result(eval_result=eval_result)