Chapter 61

03 rag evals

Notebookllm-zoomcamp-2026-evals27 cells

In [1]python · cell 1

python

import pandas as pd

df_ground_truth = pd.read_csv("data/ground_truth-new.csv")
ground_truth = df_ground_truth.to_dict(orient="records")

In [2]python · cell 2

python

ground_truth[10]

Output

{'question': 'How do I join the Office Hours or live workshop if I don’t have the Zoom link?',
 'document': '489dd1c9d9'}

In [3]python · cell 3

python

from ingest import load_faq_data, build_index

documents = load_faq_data()

documents_llm = []

for doc in documents:
    if doc["course"] == "llm-zoomcamp":
        documents_llm.append(doc)

documents = documents_llm
index = build_index(documents)

In [4]python · cell 4

python

doc_idx = {}

for doc in documents:
    doc_idx[doc["id"]] = doc

In [5]python · cell 5

python

q = ground_truth[10]
q

Output

{'question': 'How do I join the Office Hours or live workshop if I don’t have the Zoom link?',
 'document': '489dd1c9d9'}

In [6]python · cell 6

python

doc_idx[q['document']]

Output

{'id': '489dd1c9d9',
 'course': 'llm-zoomcamp',
 'section': 'General Course-Related Questions',
 'question': 'What is the video/zoom link to the stream for the “Office Hours” or live/workshop sessions?',
 'answer': 'The zoom link is only published to instructors/presenters/TAs.\n\nStudents participate via YouTube Live and submit questions to Slido (link is pinned in the chat when live). The video URL should be posted in the [announcements channel on Telegram & Slack](https://t.me/dezoomcamp) before it begins. You can also watch live on the DataTalksClub [YouTube Channel](https://www.youtube.com/c/DataTalksClub).\n\nDon’t post questions in chat as they may be missed if the room is very active.'}

In [7]python · cell 7

python

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
openai_client = OpenAI()

In [9]python · cell 8

python

from evaluation_utils import RAGWithUsage

assistant = RAGWithUsage(
    index=index,
    llm_client=openai_client,
    course='llm-zoomcamp',
)

In [11]python · cell 9

python

q['question']

Output

'How do I join the Office Hours or live workshop if I don’t have the Zoom link?'

In [12]python · cell 10

python

answer = assistant.rag(q['question'])

In [15]python · cell 11

python

assistant.total_cost()

Output

0.0018675

In [14]python · cell 12

python

print(answer)

Output

You don’t need the Zoom link as a student.

- **Office Hours / live workshop sessions are watched via YouTube Live**
- **Questions are submitted through Slido** (the link is pinned in chat when live)
- The **video URL is posted in the announcements channel on Telegram & Slack** before the session starts
- You can also watch on the **DataTalksClub YouTube channel**

If you want, I can also tell you where to find the announcements channel.

In [16]python · cell 13

python

doc_id = q["document"]
original_doc = doc_idx[doc_id]
answer_orig = original_doc["answer"]

answer_orig

Output

'The zoom link is only published to instructors/presenters/TAs.\n\nStudents participate via YouTube Live and submit questions to Slido (link is pinned in the chat when live). The video URL should be posted in the [announcements channel on Telegram & Slack](https://t.me/dezoomcamp) before it begins. You can also watch live on the DataTalksClub [YouTube Channel](https://www.youtube.com/c/DataTalksClub).\n\nDon’t post questions in chat as they may be missed if the room is very active.'

In [17]python · cell 14

python

rag_result = {
    "question": q['question'],
    "answer_llm": answer,
    "answer_orig": answer_orig,
    "document": doc_id,
}

rag_result

Output

{'question': 'How do I join the Office Hours or live workshop if I don’t have the Zoom link?',
 'answer_llm': 'You don’t need the Zoom link as a student.\n\n- **Office Hours / live workshop sessions are watched via YouTube Live**\n- **Questions are submitted through Slido** (the link is pinned in chat when live)\n- The **video URL is posted in the announcements channel on Telegram & Slack** before the session starts\n- You can also watch on the **DataTalksClub YouTube channel**\n\nIf you want, I can also tell you where to find the announcements channel.',
 'answer_orig': 'The zoom link is only published to instructors/presenters/TAs.\n\nStudents participate via YouTube Live and submit questions to Slido (link is pinned in the chat when live). The video URL should be posted in the [announcements channel on Telegram & Slack](https://t.me/dezoomcamp) before it begins. You can also watch live on the DataTalksClub [YouTube Channel](https://www.youtube.com/c/DataTalksClub).\n\nDon’t post questions in chat as they may be missed if the room is very active.',
 'document': '489dd1c9d9'}

In [18]python · cell 15

python

def generate_rag_answer(rec):
    question = rec["question"]
    doc_id = rec["document"]
    original_doc = doc_idx[doc_id]

    answer_llm = assistant.rag(question)
    answer_orig = original_doc["answer"]

    result = {
        "question": question,
        "answer_llm": answer_llm,
        "answer_orig": answer_orig,
        "document": doc_id,
    }

    return result

In [19]python · cell 16

python

record = generate_rag_answer(q)
record

Output

{'question': 'How do I join the Office Hours or live workshop if I don’t have the Zoom link?',
 'answer_llm': 'Students don’t use the Zoom link directly. For Office Hours or live workshops, join via:\n\n- **YouTube Live**\n- **Slido** for questions (the link is pinned in chat when live)\n- The live video URL is usually posted in the **announcements channel on Telegram & Slack** before it starts\n- You can also watch on the **DataTalksClub YouTube Channel**\n\nIf you want, I can also tell you where to find the announcements channel.',
 'answer_orig': 'The zoom link is only published to instructors/presenters/TAs.\n\nStudents participate via YouTube Live and submit questions to Slido (link is pinned in the chat when live). The video URL should be posted in the [announcements channel on Telegram & Slack](https://t.me/dezoomcamp) before it begins. You can also watch live on the DataTalksClub [YouTube Channel](https://www.youtube.com/c/DataTalksClub).\n\nDon’t post questions in chat as they may be missed if the room is very active.',
 'document': '489dd1c9d9'}

In [20]python · cell 17

python

assistant.total_cost()

Output

0.00282375

In [21]python · cell 18

python

assistant.reset_usage()

In [22]python · cell 19

python

assistant.total_cost()

Output

0.0

In [23]python · cell 20

python

from concurrent.futures import ThreadPoolExecutor
from evaluation_utils import map_progress

In [24]python · cell 21

python

with ThreadPoolExecutor(max_workers=6) as pool:
    results = map_progress(pool, ground_truth, generate_rag_answer)

Output

  0%|          | 0/395 [00:00<?, ?it/s]

In [26]python · cell 22

python

results[:10]

Output

[{'question': 'Is it okay to join the course late if I just found it now?',
'answer_llm': 'Yes, you can still join the course late. If you want a certificate, though, you need to submit your project while submissions are still being accepted.',
'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.',
'document': '74eb249bbf'},
{'question': 'Can I still take this course even if I missed the start date?',
'answer_llm': 'Yes, you can still join if you missed the start date, but if you want a certificate, you need to finish with the live cohort and submit your project while submissions are still open.',
'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.',
'document': '74eb249bbf'},
{'question': 'If I join after the course has already started, am I still eligible for a certificate?',
'answer_llm': 'Yes, as long as you join while the course is still running and submit your project before submissions close. Certificates are only available for the live cohort, not for self-paced participation.',
'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.',
'document': '74eb249bbf'},
{'question': 'Do I need to submit my project before submissions close to get the certificate?',
'answer_llm': 'Yes — to get the certificate, you need to submit your project while submissions are still open.',
'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.',
'document': '74eb249bbf'},
{'question': 'I’m a bit late to the course—what do I need to do to still earn the certificate?',
'answer_llm': 'To still earn the certificate, you need to:\n\n- **Finish the course with the live cohort** — certificates are **not** awarded for self-paced mode.\n- **Submit your capstone project while submissions are still open**.\n- **Peer-review 3 other participants’ capstone projects** after you submit yours.\n\nIf you’re late, you may still catch up if the **submission window is still open**.',
'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.',
'document': '74eb249bbf'},
{'question': 'I registered for the LLM Zoomcamp — when should I expect a confirmation email?',
'answer_llm': 'You don’t need to սպաս expect a confirmation email. You’re accepted, and you can start learning and submitting homework while the form is open. Registration is just to gauge interest.',
'answer_orig': "You don't need it. You're accepted. You can also just start learning and submitting homework (while the form is open) without registering. It is not checked against any registered list. Registration is just to gauge interest before the start date.",
'document': '977bf7786c'},
{'question': 'Do I actually need an acceptance email before I can start the course and hand in homework?',
'answer_llm': 'No. You don’t need an acceptance email — you’re already accepted.\n\nYou can start learning and submit homework as long as the homework form is open, even without registering. Registration is only used to gauge interest before the start date.',
'answer_orig': "You don't need it. You're accepted. You can also just start learning and submitting homework (while the form is open) without registering. It is not checked against any registered list. Registration is just to gauge interest before the start date.",
'document': '977bf7786c'},
{'question': 'If I filled out the registration form, does that mean I’m officially on a checked list for the course?',
'answer_llm': 'No. Registration is only to gauge interest, and it is not checked against a registered list. You can start learning and submitting homework as long as the form is open.',
'answer_orig': "You don't need it. You're accepted. You can also just start learning and submitting homework (while the form is open) without registering. It is not checked against any registered list. Registration is just to gauge interest before the start date.",
'document': '977bf7786c'},
{'question': 'Is registering for the LLM Zoomcamp required, or is it just for estimating interest before the course starts?',
'answer_llm': 'Registering is **not required**. It’s just used to **gauge interest before the course starts**.\n\nYou can also start learning and submitting homework while the form is open without registering.',
'answer_orig': "You don't need it. You're accepted. You can also just start learning and submitting homework (while the form is open) without registering. It is not checked against any registered list. Registration is just to gauge interest before the start date.",
'document': '977bf7786c'},
{'question': 'Can I begin learning and submit homework even if I never got any email after signing up?',
'answer_llm': 'Yes — you can start learning and submit homework even if you never received a confirmation email. The course says you’re accepted, and registration isn’t required for homework submission while the form is open.',
'answer_orig': "You don't need it. You're accepted. You can also just start learning and submitting homework (while the form is open) without registering. It is not checked against any registered list. Registration is just to gauge interest before the start date.",
'document': '977bf7786c'}]

In [27]python · cell 23

python

df_results = pd.DataFrame(results)

In [28]python · cell 24

python

df_results.head()

Output

                                            question  \
0  Is it okay to join the course late if I just f...   
1  Can I still take this course even if I missed ...   
2  If I join after the course has already started...   
3  Do I need to submit my project before submissi...   
4  I’m a bit late to the course—what do I need to...   

                                          answer_llm  \
0  Yes, you can still join the course late. If yo...   
1  Yes, you can still join if you missed the star...   
2  Yes, as long as you join while the course is s...   
3  Yes — to get the certificate, you need to subm...   
4  To still earn the certificate, you need to:\n\...   

                                         answer_orig    document  
0  Yes, but if you want to receive a certificate,...  74eb249bbf  
1  Yes, but if you want to receive a certificate,...  74eb249bbf  
2  Yes, but if you want to receive a certificate,...  74eb249bbf  
3  Yes, but if you want to receive a certificate,...  74eb249bbf  
4  Yes, but if you want to receive a certificate,...  74eb249bbf

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	question	answer_llm	answer_orig	document
0	Is it okay to join the course late if I just f...	Yes, you can still join the course late. If yo...	Yes, but if you want to receive a certificate,...	74eb249bbf
1	Can I still take this course even if I missed ...	Yes, you can still join if you missed the star...	Yes, but if you want to receive a certificate,...	74eb249bbf
2	If I join after the course has already started...	Yes, as long as you join while the course is s...	Yes, but if you want to receive a certificate,...	74eb249bbf
3	Do I need to submit my project before submissi...	Yes — to get the certificate, you need to subm...	Yes, but if you want to receive a certificate,...	74eb249bbf
4	I’m a bit late to the course—what do I need to...	To still earn the certificate, you need to:\n\...	Yes, but if you want to receive a certificate,...	74eb249bbf

In [29]python · cell 25

python

assistant.total_cost()

Output

0.34332825

In [ ]python · cell 26

python

df_results.to_csv("data/rag-answers-new.csv", index=False)

In [ ]python · cell 27

python

from ingest import load_faq_data, build_index documents = load_faq_data() documents_llm = [] for doc in documents: if doc["course"] == "llm-zoomcamp": documents_llm.append(doc) documents = documents_llm index = build_index(documents)

{'id': '489dd1c9d9', 'course': 'llm-zoomcamp', 'section': 'General Course-Related Questions', 'question': 'What is the video/zoom link to the stream for the “Office Hours” or live/workshop sessions?', 'answer': 'The zoom link is only published to instructors/presenters/TAs.\n\nStudents participate via YouTube Live and submit questions to Slido (link is pinned in the chat when live). The video URL should be posted in the [announcements channel on Telegram & Slack](https://t.me/dezoomcamp) before it begins. You can also watch live on the DataTalksClub [YouTube Channel](https://www.youtube.com/c/DataTalksClub).\n\nDon’t post questions in chat as they may be missed if the room is very active.'}

You don’t need the Zoom link as a student. - **Office Hours / live workshop sessions are watched via YouTube Live** - **Questions are submitted through Slido** (the link is pinned in chat when live) - The **video URL is posted in the announcements channel on Telegram & Slack** before the session starts - You can also watch on the **DataTalksClub YouTube channel** If you want, I can also tell you where to find the announcements channel.

'The zoom link is only published to instructors/presenters/TAs.\n\nStudents participate via YouTube Live and submit questions to Slido (link is pinned in the chat when live). The video URL should be posted in the [announcements channel on Telegram & Slack](https://t.me/dezoomcamp) before it begins. You can also watch live on the DataTalksClub [YouTube Channel](https://www.youtube.com/c/DataTalksClub).\n\nDon’t post questions in chat as they may be missed if the room is very active.'

{'question': 'How do I join the Office Hours or live workshop if I don’t have the Zoom link?', 'answer_llm': 'You don’t need the Zoom link as a student.\n\n- **Office Hours / live workshop sessions are watched via YouTube Live**\n- **Questions are submitted through Slido** (the link is pinned in chat when live)\n- The **video URL is posted in the announcements channel on Telegram & Slack** before the session starts\n- You can also watch on the **DataTalksClub YouTube channel**\n\nIf you want, I can also tell you where to find the announcements channel.', 'answer_orig': 'The zoom link is only published to instructors/presenters/TAs.\n\nStudents participate via YouTube Live and submit questions to Slido (link is pinned in the chat when live). The video URL should be posted in the [announcements channel on Telegram & Slack](https://t.me/dezoomcamp) before it begins. You can also watch live on the DataTalksClub [YouTube Channel](https://www.youtube.com/c/DataTalksClub).\n\nDon’t post questions in chat as they may be missed if the room is very active.', 'document': '489dd1c9d9'}

def generate_rag_answer(rec): question = rec["question"] doc_id = rec["document"] original_doc = doc_idx[doc_id] answer_llm = assistant.rag(question) answer_orig = original_doc["answer"] result = { "question": question, "answer_llm": answer_llm, "answer_orig": answer_orig, "document": doc_id, } return result

{'question': 'How do I join the Office Hours or live workshop if I don’t have the Zoom link?', 'answer_llm': 'Students don’t use the Zoom link directly. For Office Hours or live workshops, join via:\n\n- **YouTube Live**\n- **Slido** for questions (the link is pinned in chat when live)\n- The live video URL is usually posted in the **announcements channel on Telegram & Slack** before it starts\n- You can also watch on the **DataTalksClub YouTube Channel**\n\nIf you want, I can also tell you where to find the announcements channel.', 'answer_orig': 'The zoom link is only published to instructors/presenters/TAs.\n\nStudents participate via YouTube Live and submit questions to Slido (link is pinned in the chat when live). The video URL should be posted in the [announcements channel on Telegram & Slack](https://t.me/dezoomcamp) before it begins. You can also watch live on the DataTalksClub [YouTube Channel](https://www.youtube.com/c/DataTalksClub).\n\nDon’t post questions in chat as they may be missed if the room is very active.', 'document': '489dd1c9d9'}

[{'question': 'Is it okay to join the course late if I just found it now?', 'answer_llm': 'Yes, you can still join the course late. If you want a certificate, though, you need to submit your project while submissions are still being accepted.', 'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.', 'document': '74eb249bbf'}, {'question': 'Can I still take this course even if I missed the start date?', 'answer_llm': 'Yes, you can still join if you missed the start date, but if you want a certificate, you need to finish with the live cohort and submit your project while submissions are still open.', 'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.', 'document': '74eb249bbf'}, {'question': 'If I join after the course has already started, am I still eligible for a certificate?', 'answer_llm': 'Yes, as long as you join while the course is still running and submit your project before submissions close. Certificates are only available for the live cohort, not for self-paced participation.', 'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.', 'document': '74eb249bbf'}, {'question': 'Do I need to submit my project before submissions close to get the certificate?', 'answer_llm': 'Yes — to get the certificate, you need to submit your project while submissions are still open.', 'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.', 'document': '74eb249bbf'}, {'question': 'I’m a bit late to the course—what do I need to do to still earn the certificate?', 'answer_llm': 'To still earn the certificate, you need to:\n\n- **Finish the course with the live cohort** — certificates are **not** awarded for self-paced mode.\n- **Submit your capstone project while submissions are still open**.\n- **Peer-review 3 other participants’ capstone projects** after you submit yours.\n\nIf you’re late, you may still catch up if the **submission window is still open**.', 'answer_orig': 'Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.', 'document': '74eb249bbf'}, {'question': 'I registered for the LLM Zoomcamp — when should I expect a confirmation email?', 'answer_llm': 'You don’t need to սպաս expect a confirmation email. You’re accepted, and you can start learning and submitting homework while the form is open. Registration is just to gauge interest.', 'answer_orig': "You don't need it. You're accepted. You can also just start learning and submitting homework (while the form is open) without registering. It is not checked against any registered list. Registration is just to gauge interest before the start date.", 'document': '977bf7786c'}, {'question': 'Do I actually need an acceptance email before I can start the course and hand in homework?', 'answer_llm': 'No. You don’t need an acceptance email — you’re already accepted.\n\nYou can start learning and submit homework as long as the homework form is open, even without registering. Registration is only used to gauge interest before the start date.', 'answer_orig': "You don't need it. You're accepted. You can also just start learning and submitting homework (while the form is open) without registering. It is not checked against any registered list. Registration is just to gauge interest before the start date.", 'document': '977bf7786c'}, {'question': 'If I filled out the registration form, does that mean I’m officially on a checked list for the course?', 'answer_llm': 'No. Registration is only to gauge interest, and it is not checked against a registered list. You can start learning and submitting homework as long as the form is open.', 'answer_orig': "You don't need it. You're accepted. You can also just start learning and submitting homework (while the form is open) without registering. It is not checked against any registered list. Registration is just to gauge interest before the start date.", 'document': '977bf7786c'}, {'question': 'Is registering for the LLM Zoomcamp required, or is it just for estimating interest before the course starts?', 'answer_llm': 'Registering is **not required**. It’s just used to **gauge interest before the course starts**.\n\nYou can also start learning and submitting homework while the form is open without registering.', 'answer_orig': "You don't need it. You're accepted. You can also just start learning and submitting homework (while the form is open) without registering. It is not checked against any registered list. Registration is just to gauge interest before the start date.", 'document': '977bf7786c'}, {'question': 'Can I begin learning and submit homework even if I never got any email after signing up?', 'answer_llm': 'Yes — you can start learning and submit homework even if you never received a confirmation email. The course says you’re accepted, and registration isn’t required for homework submission while the form is open.', 'answer_orig': "You don't need it. You're accepted. You can also just start learning and submitting homework (while the form is open) without registering. It is not checked against any registered list. Registration is just to gauge interest before the start date.", 'document': '977bf7786c'}]

question \ 0 Is it okay to join the course late if I just f... 1 Can I still take this course even if I missed ... 2 If I join after the course has already started... 3 Do I need to submit my project before submissi... 4 I’m a bit late to the course—what do I need to... answer_llm \ 0 Yes, you can still join the course late. If yo... 1 Yes, you can still join if you missed the star... 2 Yes, as long as you join while the course is s... 3 Yes — to get the certificate, you need to subm... 4 To still earn the certificate, you need to:\n\... answer_orig document 0 Yes, but if you want to receive a certificate,... 74eb249bbf 1 Yes, but if you want to receive a certificate,... 74eb249bbf 2 Yes, but if you want to receive a certificate,... 74eb249bbf 3 Yes, but if you want to receive a certificate,... 74eb249bbf 4 Yes, but if you want to receive a certificate,... 74eb249bbf

question

answer_llm

answer_orig

document

Is it okay to join the course late if I just f...

Yes, you can still join the course late. If yo...