Chapter 47
04 llm judge
Notebookllm-zoomcamp-2026-evals24 cells
In [2]python · cell 1
python
from pydantic import BaseModel, Field
from typing import Literal
class AnswerEvaluation(BaseModel):
reasoning: str = Field(
description="Reasoning about the quality of the answer."
)
score: Literal["good", "bad"] = Field(
description="'good' if the answer is correct and complete, 'bad' otherwise."
)In [3]python · cell 2
python
aqa_judge_instructions = """
You are an expert evaluator. You will be given:
1. A question from a student
2. The original answer from the FAQ (ground truth)
3. An answer generated by an AI assistant
Your task is to decide if the AI answer is semantically equivalent to
the original answer.
Rules:
- The AI answer does NOT need to be word-for-word identical
- It should convey the same key information
- Extra detail is fine as long as the core answer is correct
- Mark 'bad' only if the AI answer is wrong or misses the key point
Be fair and focus on correctness, not style.
""".strip()In [4]python · cell 3
python
aqa_judge_prompt = """
Question:
{question}
Original Answer (ground truth):
{answer_orig}
AI Answer:
{answer_llm}
""".strip()In [5]python · cell 4
python
from dotenv import load_dotenv
from openai import OpenAI
from evaluation_utils import calc_price, calc_total_price, llm_structured_retry, map_progress
load_dotenv()
openai_client = OpenAI()In [ ]python · cell 5
python
from dotenv import load_dotenv
from openai import OpenAI
from evaluation_utils import calc_price, calc_total_price, llm_structured_retry, map_progress
load_dotenv()
openai_client = OpenAI()In [7]python · cell 6
python
import pandas as pd
df_answers = pd.read_csv("data/rag-answers-new.csv")
answers = df_answers.to_dict(orient="records")In [8]python · cell 7
python
rec = answers[0]
recOutput
{'question': 'Is it okay to join the course late if I just found it now?',
'answer_llm': 'Yes, you can still join the course late. If you want a certificate, though, you need to submit your project while submissions are still being accepted.',
'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.',
'document': '74eb249bbf'}In [9]python · cell 8
python
prompt = aqa_judge_prompt.format(
question=rec["question"],
answer_orig=rec["answer_orig"],
answer_llm=rec["answer_llm"]
)
print(prompt)Output
Question: Is it okay to join the course late if I just found it now? Original Answer (ground truth): Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions. AI Answer: Yes, you can still join the course late. If you want a certificate, though, you need to submit your project while submissions are still being accepted.
In [10]python · cell 9
python
eval_result, usage = llm_structured_retry(
openai_client,
aqa_judge_instructions,
prompt,
AnswerEvaluation,
)
eval_resultOutput
AnswerEvaluation(reasoning='The AI answer preserves the key meaning of the ground truth: late joining is allowed, and certificate eligibility requires submitting the project before submissions close. It is semantically equivalent.', score='good')
In [11]python · cell 10
python
calc_price(usage)Output
{'input_cost': 0.00022275000000000002,
'output_cost': 0.00022500000000000002,
'total_cost': 0.00044775}In [12]python · cell 11
python
def evaluate_aqa(question, answer_orig, answer_llm, model="gpt-5.4-mini"):
prompt = aqa_judge_prompt.format(
question=question,
answer_orig=answer_orig,
answer_llm=answer_llm
)
result, usage = llm_structured_retry(
openai_client,
aqa_judge_instructions,
prompt,
AnswerEvaluation,
model=model,
)
return result, usageIn [13]python · cell 12
python
eval_result, usage = evaluate_aqa(
question=rec["question"],
answer_orig=rec["answer_orig"],
answer_llm=rec["answer_llm"]
)
eval_resultOutput
AnswerEvaluation(reasoning='The AI answer preserves the original meaning: late joining is allowed, and certificate eligibility depends on submitting the project before submissions close. This is semantically equivalent to the ground truth.', score='good')
In [14]python · cell 13
python
def judge_record(rec):
eval_result, usage = evaluate_aqa(
question=rec["question"],
answer_orig=rec["answer_orig"],
answer_llm=rec["answer_llm"]
)
result = {
"question": rec["question"],
"document": rec["document"],
"score": eval_result.score,
"reasoning": eval_result.reasoning,
}
return result, usageIn [15]python · cell 14
python
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=6) as pool:
results = map_progress(pool, answers, judge_record)Output
0%| | 0/395 [00:00<?, ?it/s]
In [16]python · cell 15
python
results[10]Output
({'question': 'How do I join the Office Hours or live workshop if I don’t have the Zoom link?',
'document': '489dd1c9d9',
'score': 'good',
'reasoning': "The AI answer matches the ground truth: it states students don't need the Zoom link, that participation is via YouTube Live, the URL is posted in the Telegram/Slack announcements channel, questions go through Slido, and the YouTube channel is available. It omits the caution about not posting questions in chat, but that is ancillary and doesn't change the core answer."},
ResponseUsage(input_tokens=433, input_tokens_details=InputTokensDetails(cached_tokens=0), output_tokens=90, output_tokens_details=OutputTokensDetails(reasoning_tokens=0), total_tokens=523))In [17]python · cell 16
python
evaluations = []
usages = []
for evaluation, usage in results:
evaluations.append(evaluation)
usages.append(usage)In [19]python · cell 17
python
calc_total_price(usages)Output
0.251331
In [18]python · cell 18
python
df_eval = pd.DataFrame(evaluations)In [21]python · cell 19
python
df_eval.head()Output
question document score \
0 Is it okay to join the course late if I just f... 74eb249bbf good
1 Can I still take this course even if I missed ... 74eb249bbf good
2 If I join after the course has already started... 74eb249bbf good
3 Do I need to submit my project before submissi... 74eb249bbf good
4 I’m a bit late to the course—what do I need to... 74eb249bbf good
reasoning
0 The AI answer preserves the ground truth meani...
1 The AI answer preserves the core meaning: you ...
2 The AI answer preserves the key point: joining...
3 The AI answer preserves the key point of the g...
4 The AI answer includes the core requirement fr...
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| question | document | score | reasoning | |
|---|---|---|---|---|
| 0 | Is it okay to join the course late if I just f... | 74eb249bbf | good | The AI answer preserves the ground truth meani... |
| 1 | Can I still take this course even if I missed ... | 74eb249bbf | good | The AI answer preserves the core meaning: you ... |
| 2 | If I join after the course has already started... | 74eb249bbf | good | The AI answer preserves the key point: joining... |
| 3 | Do I need to submit my project before submissi... | 74eb249bbf | good | The AI answer preserves the key point of the g... |
| 4 | I’m a bit late to the course—what do I need to... | 74eb249bbf | good | The AI answer includes the core requirement fr... |
In [22]python · cell 20
python
df_eval.score.value_counts()Output
score good 379 bad 16 Name: count, dtype: int64
In [23]python · cell 21
python
df_eval.score.value_counts(normalize=True)Output
score good 0.959494 bad 0.040506 Name: proportion, dtype: float64
In [24]python · cell 22
python
df_eval[df_eval["score"] == "bad"].head()Output
question document score \
15 How do the free GPU hours work on these cloud ... c6c2888275 bad
29 Is peer-review of the capstone project require... 69d122f12e bad
38 How will I know when a module is actually read... 96286b4be4 bad
73 Which model should I use in chat.completions.c... 152af39a53 bad
106 Do I need an OpenAI API key just to check how ... fe8fed31e6 bad
reasoning
15 The AI answer does not convey the ground truth...
29 The AI answer is not semantically equivalent t...
38 The AI answer does not convey the ground truth...
73 The ground truth says the issue is insufficien...
106 The AI answer does not address the question or...
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| question | document | score | reasoning | |
|---|---|---|---|---|
| 15 | How do the free GPU hours work on these cloud ... | c6c2888275 | bad | The AI answer does not convey the ground truth... |
| 29 | Is peer-review of the capstone project require... | 69d122f12e | bad | The AI answer is not semantically equivalent t... |
| 38 | How will I know when a module is actually read... | 96286b4be4 | bad | The AI answer does not convey the ground truth... |
| 73 | Which model should I use in chat.completions.c... | 152af39a53 | bad | The ground truth says the issue is insufficien... |
| 106 | Do I need an OpenAI API key just to check how ... | fe8fed31e6 | bad | The AI answer does not address the question or... |
In [25]python · cell 23
python
df_eval.to_csv("data/rag-evaluations-new.csv", index=False)In [ ]python · cell 24
python