Chapter 44
open rag eval example
Evaluating RAG with Open-RAG-Eval
Overview
Open-RAG-Eval is an open-source framework that lets teams evaluate RAG systems without needing predefined (golden) answers, making it faster and easier to compare solutions or configurations. With automated, research-backed metrics like UMBRELA and Hallucination, it brings transparency and rigor to RAG performance testing at any scale.
This notebook provides a simple example of "how to use" Open-RAG-Eval. You can read more about the overall vision and metrics in this blog post
Specifically, we run the evaluation framework on the fiqa dataset from the BEIR benchmark, which contains question and answer paris for the financial domain.
In a typical RAG evaluation flow - one would need to create a RAG pipeline, run the queries againat that RAG pipeline adn collect the chunks and responses. Since this type of flow doesn't fit well in a notebook, we already pre-indexed the dataset in Vectara and ran a subset of queries (from the same fiqa dataset) against the indexed data, and stored the outputs in the fiqa_output.csv file, which contains the queries, retrieved results and LLM generated answers to those queries.
If you have a Vectara account, you can set Open-RAG-Eval with the Vectara connector and a set of queries to automatically create this file by runnning each query through your Vectara RAG.
RAG Evaluation outputs
To get started, let's look at the fiqa_output.csv file:
import pandas as pd
import os
from dotenv import load_dotenv
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from open_rag_eval.evaluators import TRECEvaluator
from open_rag_eval.models.llm_judges import OpenAIModel
from open_rag_eval.connectors import CSVConnector
from open_rag_eval.data_classes import eval_scores
import pprint
pp = pprint.PrettyPrinter(indent=4, width=80)
df = pd.read_csv('https://raw.githubusercontent.com/vectara/example-notebooks/main/data/fiqa_output.csv').fillna('')
df = df.head(50) # Limiting to first 50 rows for faster run time
df.head(10)Output
query_id \
0 8311fbce-c34a-43da-9bfe-24b72c2d3a0c
1 8311fbce-c34a-43da-9bfe-24b72c2d3a0c
2 8311fbce-c34a-43da-9bfe-24b72c2d3a0c
3 8311fbce-c34a-43da-9bfe-24b72c2d3a0c
4 8311fbce-c34a-43da-9bfe-24b72c2d3a0c
5 45332c9d-14de-4107-b136-b71f6673b8e2
6 45332c9d-14de-4107-b136-b71f6673b8e2
7 45332c9d-14de-4107-b136-b71f6673b8e2
8 45332c9d-14de-4107-b136-b71f6673b8e2
9 45332c9d-14de-4107-b136-b71f6673b8e2
query passage_id \
0 accredited investor definition [1]
1 accredited investor definition [2]
2 accredited investor definition [3]
3 accredited investor definition [4]
4 accredited investor definition [5]
5 what is the purpose of a job training [1]
6 what is the purpose of a job training [2]
7 what is the purpose of a job training [3]
8 what is the purpose of a job training [4]
9 what is the purpose of a job training [5]
passage \
0 Here are the SEC requirements: The federal se...
1 "The (U.S) ""accredited investor"" laws apply...
2 Accredited investors are required to have 1 m...
3 Does me holding stock in the company make me ...
4 SecondMarket attempts to add liquidity to pri...
5 I'm not saying I don't like the idea of on-th...
6 A lot of jobs, if not most, just don't requir...
7 > but we as a society do not even offer en...
8 We also have for-profit, career-oriented scho...
9 What about electrician, plumber, welder, cons...
generated_answer
0 Based on the provided sources, an accredited i...
1
2
3
4
5 Based on the provided sources, the purpose of ...
6
7
8
9 | query_id | query | passage_id | passage | generated_answer | |
|---|---|---|---|---|---|
| 0 | 8311fbce-c34a-43da-9bfe-24b72c2d3a0c | accredited investor definition | [1] | Here are the SEC requirements: The federal se... | Based on the provided sources, an accredited i... |
| 1 | 8311fbce-c34a-43da-9bfe-24b72c2d3a0c | accredited investor definition | [2] | "The (U.S) ""accredited investor"" laws apply... | |
| 2 | 8311fbce-c34a-43da-9bfe-24b72c2d3a0c | accredited investor definition | [3] | Accredited investors are required to have 1 m... | |
| 3 | 8311fbce-c34a-43da-9bfe-24b72c2d3a0c | accredited investor definition | [4] | Does me holding stock in the company make me ... | |
| 4 | 8311fbce-c34a-43da-9bfe-24b72c2d3a0c | accredited investor definition | [5] | SecondMarket attempts to add liquidity to pri... | |
| 5 | 45332c9d-14de-4107-b136-b71f6673b8e2 | what is the purpose of a job training | [1] | I'm not saying I don't like the idea of on-th... | Based on the provided sources, the purpose of ... |
| 6 | 45332c9d-14de-4107-b136-b71f6673b8e2 | what is the purpose of a job training | [2] | A lot of jobs, if not most, just don't requir... | |
| 7 | 45332c9d-14de-4107-b136-b71f6673b8e2 | what is the purpose of a job training | [3] | > but we as a society do not even offer en... | |
| 8 | 45332c9d-14de-4107-b136-b71f6673b8e2 | what is the purpose of a job training | [4] | We also have for-profit, career-oriented scho... | |
| 9 | 45332c9d-14de-4107-b136-b71f6673b8e2 | what is the purpose of a job training | [5] | What about electrician, plumber, welder, cons... |
fiqa_output.csv comes in the CSV format that Open-RAG-Eval expects.
- Each query has a unique
query_id, with multiple rows corresponding to each query. This is because we use one row per retrieved chunk, and in this case we retrieved 5 passages per query, and you can see that for each passage we have a separate passage_id and passage text. - The
generated_answercolumn includes the answer generated by the LLM to the query grounded on the passages.
Run an Open-RAG-Eval Evaluation
To run a full evaluation, we need to
- Specify the evaluator: for this example we're using the TREC RAG Evaluator. This Evaluator needs an LLM as the judge model which will be scoring responses, in this case we will use GPT-4o-mini.
- Creatae a
CSVConnectorto read the data and simply run the evaluator - Run the evaluation
# Make sure your OPENAI_API_KEY is in a .env file in the same directory as this notebook.
load_dotenv()
judge_model = OpenAIModel(model_name="gpt-4o-mini", api_key=os.getenv("OPENAI_API_KEY"))
evaluator = TRECEvaluator(model=judge_model)
rag_results = CSVConnector('https://raw.githubusercontent.com/vectara/example-notebooks/main/data/fiqa_output.csv').fetch_data()
scored_results = evaluator.evaluate_batch(rag_results)Output
You are using a model of type HHEMv2Config to instantiate a model of type HHEMv2. This is not supported for all configurations of models and can yield errors. Evaluating using TRECRAG evaluator.: 0%| | 0/10 [00:00<?, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1615 > 512). Running this sequence through the model will result in indexing errors Evaluating using TRECRAG evaluator.: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:48<00:00, 4.84s/it]
Note: you can ignore the 'Token indices sequence length is longer than the specified maximum sequence length for this model (1782 > 512). It's not important.
Review Evaluation Results
The evaluator uses multiple metrics.
Internally, each metric class scores either the retrieval results or the generated response and adds its own scores to the scored_results object under a unique key, which contain both the original query, retrieved passages and the scored metrics along with any intermediate metric output for easy debugging and in-depth analysis.
Let's look at the metrics in more depth, starting with the UMBRELA metric:
UMBRELA Metric
The UMBRELA metric assigns a value to each retrieved passage as follows:
- NO_RELEVANCE = "0"
- RELATED = "1"
- PARTIAL_ANSWER = "2"
- EXACT_ANSWER = "3"
Let's pick a query to show how the UMBRELA metrics look:
for scored_result in scored_results:
if scored_result.rag_result.retrieval_result.query == "accredited investor definition":
break
scores = scored_results[0].scores
# Our query is:
pp.pprint(f"Query: {scored_result.rag_result.retrieval_result.query}")
pp.pprint("------")
num_results_to_show = 5
for key, passage in scored_result.rag_result.retrieval_result.retrieved_passages.items():
pp.pprint(f"Passage: {passage}")
pp.pprint(f"Score: {scores.retrieval_score.scores['umbrela_scores'][key]}")
pp.pprint("------")
num_results_to_show -= 1
if num_results_to_show == 0:
breakOutput
'Query: accredited investor definition'
'------'
('Passage: Here are the SEC requirements: The federal securities laws define '
'the term accredited investor in Rule 501 of Regulation D as: a bank, '
'insurance company, registered investment company, business development '
'company, or small business investment company; an employee benefit plan, '
'within the meaning of the Employee Retirement Income Security Act, if a '
'bank, insurance company, or registered investment adviser makes the '
'investment decisions, or if the plan has total assets in excess of $5 '
'million; a charitable organization, corporation, or partnership with assets '
'exceeding $5 million; a director, executive officer, or general partner of '
'the company selling the securities; a business in which all the equity '
'owners are accredited investors; a natural person who has individual net '
'worth, or joint net worth with the person’s spouse, that exceeds $1 million '
'at the time of the purchase, excluding the value of the primary residence '
'of such person; a natural person with income exceeding $200,000 in each of '
'the two most recent years or joint income with a spouse exceeding $300,000 '
'for those years and a reasonable expectation of the same income level in '
'the current year; or a trust with assets in excess of $5 million, not formed '
'to acquire the securities offered, whose purchases a sophisticated person '
'makes. No citizenship/residency requirements.')
'Score: 2'
'------'
('Passage: "The (U.S) ""accredited investor"" laws apply to investments in '
'the U.S. Foreign countries may or may not have their own laws regarding '
'investment in startups, and if so, the foreign laws apply. One way around '
'the net worth minimum is to be a member of the management team. ""Active"" '
"(management) investors don't need to be accredited because they can see "
"what's going on on a day to day basis. The accredited investor laws apply to "
'the target companies, not to the investors. Basically, a start-up company '
'can\'t take ""other people\'s money"" from a non-accredited investor. But '
'you can invest ""your own"" money in it if you are a manager."')
'Score: 1'
'------'
('Passage: Accredited investors are required to have 1 million in assets (not '
'including primary residence) or $200,000/yr income for the last 3 years. '
'These kinds of regulations come from the SEC, not the company involved, '
"which means the SEC thinks it's a risky investment. If I recall correctly, "
'[someone I know] had to submit evidence of being an accredited investor to '
'trade options on [his] IRA. It may be that this is related to the '
'classification of the options.')
'Score: 1'
'------'
('Passage: Does me holding stock in the company make me an accredited '
'investor with this company in particular? No. But maybe the site will let '
'you trade it your shares to another accredited investor. Just ask, if the '
'site operators have a securities lawyer they should be able to accomodate')
'Score: 1'
'------'
('Passage: SecondMarket attempts to add liquidity to privately held '
'companies. You may be able to find a buyer there, but this is still '
'incredibly illiquid due to accredited investor regulations constricting '
'businesses from catering to the 99%. As around 1% of the United States '
'population qualifies as an accredited investor.')
'Score: 0'
'------'
AutoNuggetizer Metric
Another interesting metric to look at is the autoNuggetizer metric
We can examine which nuggets of information the metric thought a good answer should have, based on the passages (the so-called 'auto nuggets'). Each nugget is deemed 'vital' or 'okay' to be included in the answer, and then the generated answer is compared to each nugget to see if it "contains", "partially contains" or "does not contain" the nugget (called "support", "partial_support" and "not_support")
autonuggetizer_scores = scores.generation_score.scores['autonugget_scores']
# Let's look at the generated answer first (this needs a bit of construction as we decompose and store it as text + citations).
recreated_answer = []
for part in scored_result.rag_result.generation_result.generated_answer:
recreated_answer.append(part.text)
for citation in part.citations:
recreated_answer.append(citation)
generated_answer =" ".join(recreated_answer)
pp.pprint(f"Generated Answer: {generated_answer}")
for nugget, importance, support in zip(autonuggetizer_scores['nuggets'], autonuggetizer_scores['labels'], autonuggetizer_scores['assignments']):
pp.pprint(f"Nugget: {nugget}, Importance: {importance}, Support: {support}")Output
('Generated Answer: Based on the provided sources, an accredited investor is '
'defined as:\n'
'\n'
'* A bank, insurance company, registered investment company, business '
'development company, or small business investment company [1] * An employee '
'benefit plan with assets exceeding $5 million [1] * A charitable '
'organization, corporation, or partnership with assets exceeding $5 million '
'[1] * A director, executive officer, or general partner of the company '
'selling the securities [1] * A business in which all the equity owners are '
'accredited investors [1] * A natural person who has individual net worth, or '
"joint net worth with the person's spouse, that exceeds $1 million at the "
'time of the purchase, excluding the value of the primary residence of such '
'person [1] * A natural person with income exceeding $200,000 in each of the '
'two most recent years or joint income with a spouse exceeding $300,000 for '
'those years and a reasonable expectation of the same income level in the '
'current year [1] * A trust with assets in excess of $5 million, not formed '
'to acquire the securities offered, whose purchases a sophisticated person '
'makes [1] Additionally, accredited investors are required to have:\n'
'\n'
'* $1 million in assets (not including primary residence) or $200,000/yr '
"income for the last 3 years [3] It's worth noting that being an accredited "
'investor does not necessarily mean that one is a member of the management '
'team [2] , and holding stock in a company does not automatically make one an '
'accredited investor with that company [4] .\n'
'\n'
'Sources: [1] [2] [3] [4]')
('Nugget: Canadian importers can pay in CNY, Importance: '
'NuggetImportanceValues.VITAL, Support: NuggetAssignmentValues.SUPPORT')
('Nugget: Transaction costs reduced using CNY, Importance: '
'NuggetImportanceValues.VITAL, Support: '
'NuggetAssignmentValues.PARTIAL_SUPPORT')
('Nugget: China-Canada trade exceeds $70 billion annually, Importance: '
'NuggetImportanceValues.VITAL, Support: NuggetAssignmentValues.SUPPORT')
('Nugget: Currency risk affects large transactions, Importance: '
'NuggetImportanceValues.VITAL, Support: NuggetAssignmentValues.NOT_SUPPORT')
('Nugget: CNY avoids USD exchange risk, Importance: '
'NuggetImportanceValues.OKAY, Support: NuggetAssignmentValues.SUPPORT')
('Nugget: Canadian exporters can accept payments in RMB, Importance: '
'NuggetImportanceValues.OKAY, Support: NuggetAssignmentValues.NOT_SUPPORT')
('Nugget: CNY appreciates against USD, Importance: '
'NuggetImportanceValues.OKAY, Support: NuggetAssignmentValues.PARTIAL_SUPPORT')
('Nugget: Forex trading exceeds real trade amounts, Importance: '
'NuggetImportanceValues.OKAY, Support: NuggetAssignmentValues.NOT_SUPPORT')
('Nugget: Imports from cheaper countries impact pricing, Importance: '
'NuggetImportanceValues.OKAY, Support: NuggetAssignmentValues.PARTIAL_SUPPORT')
('Nugget: Japanese firms used to price in USD, Importance: '
'NuggetImportanceValues.OKAY, Support: NuggetAssignmentValues.NOT_SUPPORT')
('Nugget: Investors prefer TSX listings in CAD, Importance: '
'NuggetImportanceValues.OKAY, Support: NuggetAssignmentValues.NOT_SUPPORT')
('Nugget: Increased liquidity for Canadian companies with dual listings, '
'Importance: NuggetImportanceValues.OKAY, Support: '
'NuggetAssignmentValues.NOT_SUPPORT')
Hallucination Metric
This metric indicates whether the generated answer that you see in the previous section is hallucinated or not based on the retrieved passages that you can see in the UMBRELA section.
The score ranges from [0, 1] with a higher score indicating a higher consistency, we usually threshold the score at 0.5 (so scores >= 0.5 is considered 'factually consistent' and scores < 0.5 are considered hallucinated).
print("Hallucination Score:", scores.generation_score.scores['hallucination_scores'])Output
Hallucination Score: 0.8428813219070435
Citation Metric
The citation metric determines if the citations produced by the generative model are accurate or not. If for example the LLM says a sentence came from retrieved passage [1], does that passage actually support the sentence or not. You can view a per citation view of this or an aggregate F1 score.
print("Citation Scores:")
scores.generation_score.scores['citation_scores']Output
Citation Scores:
{'citation_score_[1]': 1,
'citation_score_[3]': 0,
'weighted_precision': 0.5,
'weighted_recall': 0.2,
'f1': 0.28571428571428575}Plotting results
Now that we have the scored_results, let's save them into a file, and then use the plot_metrics function to create a summary plot of the evaluation results.
If you have multiple results files you can pass them to the plot_metrics function as a list and all the results will be plotted on the same graph object. Here we only have one.
# Save the evaluation results to a CSV file
eval_scores.to_csv(scored_results, file_path="evaluation_results.csv")
# Plot the metrics
TRECEvaluator.plot_metrics(csv_files=["evaluation_results.csv"], output_file="metrics_summary.png")Output
Graph saved to metrics_summary.png
plt.figure(figsize=(10, 8))
img = mpimg.imread('metrics_summary.png')
plt.imshow(img)
plt.axis('off')
plt.show()Output
<Figure size 1000x800 with 1 Axes>
[省略较大 image/png 输出]