Chapter 109
Load documents with IDs
NotebookPython 3 (ipykernel)87 cells
Load documents with IDs
In [17]python · cell 2
python
import requests
base_url = 'https://github.com/DataTalksClub/llm-zoomcamp/blob/main'
relative_url = '03-vector-search/eval/documents-with-ids.json'
docs_url = f'{base_url}/{relative_url}?raw=1'
docs_response = requests.get(docs_url)
documents = docs_response.json()In [54]python · cell 3
python
documents[10]Output
{'text': 'It depends on your background and previous experience with modules. It is expected to require about 5 - 15 hours per week. [source1] [source2]\nYou can also calculate it yourself using this data and then update this answer.',
'section': 'General course-related questions',
'question': 'Course - \u200b\u200bHow many hours per week am I expected to spend on this course?',
'course': 'data-engineering-zoomcamp',
'id': 'ea739c65'}Load ground truth
In [3]python · cell 5
python
import pandas as pd
base_url = 'https://github.com/DataTalksClub/llm-zoomcamp/blob/main'
relative_url = '03-vector-search/eval/ground-truth-data.csv'
ground_truth_url = f'{base_url}/{relative_url}?raw=1'
df_ground_truth = pd.read_csv(ground_truth_url)
df_ground_truth = df_ground_truth[df_ground_truth.course == 'machine-learning-zoomcamp']
ground_truth = df_ground_truth.to_dict(orient='records')In [24]python · cell 6
python
ground_truth[10]Output
{'question': 'Are sessions recorded if I miss one?',
'course': 'machine-learning-zoomcamp',
'document': '5170565b'}In [60]python · cell 7
python
doc_idx = {d['id']: d for d in documents}
doc_idx['5170565b']['text']Output
'Everything is recorded, so you won’t miss anything. You will be able to ask your questions for office hours in advance and we will cover them during the live stream. Also, you can always ask questions in Slack.'
Index data
In [4]python · cell 9
python
from sentence_transformers import SentenceTransformer
model_name = 'multi-qa-MiniLM-L6-cos-v1'
model = SentenceTransformer(model_name)In [6]python · cell 10
python
from elasticsearch import Elasticsearch
es_client = Elasticsearch('http://localhost:9200')
index_settings = {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"properties": {
"text": {"type": "text"},
"section": {"type": "text"},
"question": {"type": "text"},
"course": {"type": "keyword"},
"id": {"type": "keyword"},
"question_text_vector": {
"type": "dense_vector",
"dims": 384,
"index": True,
"similarity": "cosine"
},
}
}
}
index_name = "course-questions"
es_client.indices.delete(index=index_name, ignore_unavailable=True)
es_client.indices.create(index=index_name, body=index_settings)Output
ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'course-questions'})In [10]python · cell 11
python
from tqdm.auto import tqdm
for doc in tqdm(documents):
question = doc['question']
text = doc['text']
doc['question_text_vector'] = model.encode(question + ' ' + text)
es_client.index(index=index_name, document=doc)Output
0%| | 0/948 [00:00<?, ?it/s]
Retrieval
In [26]python · cell 13
python
def elastic_search_knn(field, vector, course):
knn = {
"field": field,
"query_vector": vector,
"k": 5,
"num_candidates": 10000,
"filter": {
"term": {
"course": course
}
}
}
search_query = {
"knn": knn,
"_source": ["text", "section", "question", "course", "id"]
}
es_results = es_client.search(
index=index_name,
body=search_query
)
result_docs = []
for hit in es_results['hits']['hits']:
result_docs.append(hit['_source'])
return result_docs
def question_text_vector_knn(q):
question = q['question']
course = q['course']
v_q = model.encode(question)
return elastic_search_knn('question_text_vector', v_q, course)In [37]python · cell 14
python
question_text_vector_knn(dict(
question='Are sessions recorded if I miss one?',
course='machine-learning-zoomcamp'
))Output
[{'question': 'What if I miss a session?',
'course': 'machine-learning-zoomcamp',
'section': 'General course-related questions',
'text': 'Everything is recorded, so you won’t miss anything. You will be able to ask your questions for office hours in advance and we will cover them during the live stream. Also, you can always ask questions in Slack.',
'id': '5170565b'},
{'question': 'Is it going to be live? When?',
'course': 'machine-learning-zoomcamp',
'section': 'General course-related questions',
'text': 'The course videos are pre-recorded, you can start watching the course right now.\nWe will also occasionally have office hours - live sessions where we will answer your questions. The office hours sessions are recorded too.\nYou can see the office hours as well as the pre-recorded course videos in the course playlist on YouTube.',
'id': '39fda9f0'},
{'question': 'The same accuracy on epochs',
'course': 'machine-learning-zoomcamp',
'section': '8. Neural Networks and Deep Learning',
'text': "Problem description\nThe accuracy and the loss are both still the same or nearly the same while training.\nSolution description\nIn the homework, you should set class_mode='binary' while reading the data.\nAlso, problem occurs when you choose the wrong optimizer, batch size, or learning rate\nAdded by Ekaterina Kutovaia",
'id': '7d11d5ce'},
{'question': 'Useful Resource for Missing Data Treatment\nhttps://www.kaggle.com/code/parulpandey/a-guide-to-handling-missing-values-in-python/notebook',
'course': 'machine-learning-zoomcamp',
'section': '2. Machine Learning for Regression',
'text': '(Hrithik Kumar Advani)',
'id': '81b8e8d0'},
{'question': 'Will I get a certificate if I missed the midterm project?',
'course': 'machine-learning-zoomcamp',
'section': 'General course-related questions',
'text': "Yes, it's possible. See the previous answer.",
'id': '1d644223'}]The RAG flow
In [47]python · cell 16
python
def build_prompt(query, search_results):
prompt_template = """
You're a course teaching assistant. Answer the QUESTION based on the CONTEXT from the FAQ database.
Use only the facts from the CONTEXT when answering the QUESTION.
QUESTION: {question}
CONTEXT:
{context}
""".strip()
context = ""
for doc in search_results:
context = context + f"section: {doc['section']}\nquestion: {doc['question']}\nanswer: {doc['text']}\n\n"
prompt = prompt_template.format(question=query, context=context).strip()
return promptIn [115]python · cell 17
python
from openai import OpenAI
client = OpenAI()
def llm(prompt, model='gpt-4o'):
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.contentIn [116]python · cell 18
python
# previously: rag(query: str) -> str
def rag(query: dict, model='gpt-4o') -> str:
search_results = question_text_vector_knn(query)
prompt = build_prompt(query['question'], search_results)
answer = llm(prompt, model=model)
return answerIn [55]python · cell 19
python
ground_truth[10]Output
{'question': 'Are sessions recorded if I miss one?',
'course': 'machine-learning-zoomcamp',
'document': '5170565b'}In [51]python · cell 20
python
rag(ground_truth[10])Output
'Yes, sessions are recorded if you miss one. Everything is recorded, allowing you to catch up on any missed content. Additionally, you can ask questions in advance for office hours and have them addressed during the live stream. You can also ask questions in Slack.'
In [53]python · cell 21
python
doc_idx['5170565b']['text']Output
'Everything is recorded, so you won’t miss anything. You will be able to ask your questions for office hours in advance and we will cover them during the live stream. Also, you can always ask questions in Slack.'
Cosine similarity metric
In [61]python · cell 23
python
answer_orig = 'Yes, sessions are recorded if you miss one. Everything is recorded, allowing you to catch up on any missed content. Additionally, you can ask questions in advance for office hours and have them addressed during the live stream. You can also ask questions in Slack.'
answer_llm = 'Everything is recorded, so you won’t miss anything. You will be able to ask your questions for office hours in advance and we will cover them during the live stream. Also, you can always ask questions in Slack.'
v_llm = model.encode(answer_llm)
v_orig = model.encode(answer_orig)
v_llm.dot(v_orig)Output
0.759117
In [62]python · cell 24
python
ground_truth[0]Output
{'question': 'Where can I sign up for the course?',
'course': 'machine-learning-zoomcamp',
'document': '0227b872'}In [63]python · cell 25
python
len(ground_truth)Output
1830
In [65]python · cell 26
python
recOutput
{'question': 'Where can I sign up for the course?',
'course': 'machine-learning-zoomcamp',
'document': '0227b872'}In [ ]python · cell 27
python
answers = {}In [89]python · cell 28
python
for i, rec in enumerate(tqdm(ground_truth)):
if i in answers:
continue
answer_llm = rag(rec)
doc_id = rec['document']
original_doc = doc_idx[doc_id]
answer_orig = original_doc['text']
answers[i] = {
'answer_llm': answer_llm,
'answer_orig': answer_orig,
'document': doc_id,
'question': rec['question'],
'course': rec['course'],
}Output
0%| | 0/1830 [00:00<?, ?it/s]
In [107]python · cell 29
python
results_gpt4o = [None] * len(ground_truth)
for i, val in answers.items():
results_gpt4o[i] = val.copy()
results_gpt4o[i].update(ground_truth[i])In [95]python · cell 30
python
import pandas as pdIn [110]python · cell 31
python
df_gpt4o = pd.DataFrame(results_gpt4o)In [112]python · cell 32
python
!mkdir dataIn [113]python · cell 33
python
df_gpt4o.to_csv('data/results-gpt4o.csv', index=False)Evaluating GPT 3.5
In [119]python · cell 35
python
rag(ground_truth[10], model='gpt-3.5-turbo')Output
"No, sessions are recorded so if you miss one, you won't miss anything."
In [120]python · cell 36
python
from tqdm.auto import tqdm
from concurrent.futures import ThreadPoolExecutor
pool = ThreadPoolExecutor(max_workers=6)
def map_progress(pool, seq, f):
results = []
with tqdm(total=len(seq)) as progress:
futures = []
for el in seq:
future = pool.submit(f, el)
future.add_done_callback(lambda p: progress.update())
futures.append(future)
for future in futures:
result = future.result()
results.append(result)
return resultsIn [121]python · cell 37
python
def process_record(rec):
model = 'gpt-3.5-turbo'
answer_llm = rag(rec, model=model)
doc_id = rec['document']
original_doc = doc_idx[doc_id]
answer_orig = original_doc['text']
return {
'answer_llm': answer_llm,
'answer_orig': answer_orig,
'document': doc_id,
'question': rec['question'],
'course': rec['course'],
}In [122]python · cell 38
python
process_record(ground_truth[10])Output
{'answer_llm': 'Yes, sessions are recorded if you miss one. Everything is recorded, so you won’t miss anything, and you can also ask questions for office hours in advance which will be covered during the live stream. You can always ask questions in Slack as well.',
'answer_orig': 'Everything is recorded, so you won’t miss anything. You will be able to ask your questions for office hours in advance and we will cover them during the live stream. Also, you can always ask questions in Slack.',
'document': '5170565b',
'question': 'Are sessions recorded if I miss one?',
'course': 'machine-learning-zoomcamp'}In [123]python · cell 39
python
results_gpt35 = map_progress(pool, ground_truth, process_record)Output
0%| | 0/1830 [00:00<?, ?it/s]
In [124]python · cell 40
python
df_gpt35 = pd.DataFrame(results_gpt35)
df_gpt35.to_csv('data/results-gpt35.csv', index=False)In [125]python · cell 41
python
!head data/results-gpt35.csvOutput
answer_llm,answer_orig,document,question,course You can sign up for the course by going to the course page at http://mlzoomcamp.com/ and scrolling down to access the course materials.,"Machine Learning Zoomcamp FAQ The purpose of this document is to capture frequently asked technical questions. We did this for our data engineering course and it worked quite well. Check this document for inspiration on how to structure your questions and answers: Data Engineering Zoomcamp FAQ In the course GitHub repository there’s a link. Here it is: https://airtable.com/shryxwLd0COOEaqXo work",0227b872,Where can I sign up for the course?,machine-learning-zoomcamp "I am sorry, but there is no direct link provided in the FAQ database for signing up for the course. However, you can find a link in the course GitHub repository at this address: https://airtable.com/shryxwLd0COOEaqXo.","Machine Learning Zoomcamp FAQ The purpose of this document is to capture frequently asked technical questions. We did this for our data engineering course and it worked quite well. Check this document for inspiration on how to structure your questions and answers:
Cosine similarity
A->Q->A' cosine similarity
A -> Q -> A'
cosine(A, A')
gpt-4o
In [128]python · cell 43
python
results_gpt4o = df_gpt4o.to_dict(orient='records')In [130]python · cell 44
python
record = results_gpt4o[0]In [134]python · cell 45
python
def compute_similarity(record):
answer_orig = record['answer_orig']
answer_llm = record['answer_llm']
v_llm = model.encode(answer_llm)
v_orig = model.encode(answer_orig)
return v_llm.dot(v_orig)In [135]python · cell 46
python
similarity = []
for record in tqdm(results_gpt4o):
sim = compute_similarity(record)
similarity.append(sim)Output
0%| | 0/1830 [00:00<?, ?it/s]
In [142]python · cell 47
python
df_gpt4o['cosine'] = similarity
df_gpt4o['cosine'].describe()Output
count 1830.000000 mean 0.679129 std 0.217995 min -0.153426 25% 0.591460 50% 0.734788 75% 0.835390 max 0.995339 Name: cosine, dtype: float64
In [144]python · cell 48
python
import seaborn as snsOutput
Matplotlib is building the font cache; this may take a moment.
gpt-3.5-turbo
In [146]python · cell 50
python
results_gpt35 = df_gpt35.to_dict(orient='records')
similarity_35 = []
for record in tqdm(results_gpt35):
sim = compute_similarity(record)
similarity_35.append(sim)Output
0%| | 0/1830 [00:00<?, ?it/s]
In [147]python · cell 51
python
df_gpt35['cosine'] = similarity_35
df_gpt35['cosine'].describe()Output
count 1830.000000 mean 0.657599 std 0.226062 min -0.168921 25% 0.546504 50% 0.714783 75% 0.817262 max 1.000000 Name: cosine, dtype: float64
In [149]python · cell 52
python
import matplotlib.pyplot as pltgpt-4o-mini
In [151]python · cell 54
python
def process_record_4o_mini(rec):
model = 'gpt-4o-mini'
answer_llm = rag(rec, model=model)
doc_id = rec['document']
original_doc = doc_idx[doc_id]
answer_orig = original_doc['text']
return {
'answer_llm': answer_llm,
'answer_orig': answer_orig,
'document': doc_id,
'question': rec['question'],
'course': rec['course'],
}In [152]python · cell 55
python
process_record_4o_mini(ground_truth[10])Output
{'answer_llm': "Yes, sessions are recorded, so if you miss one, you won't miss anything. You can catch up by watching the recorded sessions later. Additionally, you have the option to ask questions in advance for office hours, which will also be recorded.",
'answer_orig': 'Everything is recorded, so you won’t miss anything. You will be able to ask your questions for office hours in advance and we will cover them during the live stream. Also, you can always ask questions in Slack.',
'document': '5170565b',
'question': 'Are sessions recorded if I miss one?',
'course': 'machine-learning-zoomcamp'}In [ ]python · cell 56
python
results_gpt4omini = []In [157]python · cell 57
python
for record in tqdm(ground_truth):
result = process_record_4o_mini(record)
results_gpt4omini.append(result)Output
0%| | 0/1830 [00:00<?, ?it/s]
In [160]python · cell 58
python
df_gpt4o_mini = pd.DataFrame(results_gpt4omini)
df_gpt4o_mini.to_csv('data/results-gpt4o-mini.csv', index=False)In [161]python · cell 59
python
similarity_4o_mini = []
for record in tqdm(results_gpt4omini):
sim = compute_similarity(record)
similarity_4o_mini.append(sim)Output
0%| | 0/1830 [00:00<?, ?it/s]
In [162]python · cell 60
python
df_gpt4o_mini['cosine'] = similarity_4o_mini
df_gpt4o_mini['cosine'].describe()Output
count 1830.000000 mean 0.680332 std 0.215962 min -0.141910 25% 0.585866 50% 0.733998 75% 0.836750 max 0.982701 Name: cosine, dtype: float64
gpt4o
code
count 1830.000000
mean 0.679129
std 0.217995
min -0.153426
25% 0.591460
50% 0.734788
75% 0.835390
max 0.995339
Name: cosine, dtype: float64In [250]python · cell 62
python
# sns.distplot(df_gpt35['cosine'], label='3.5')
sns.distplot(df_gpt4o['cosine'], label='4o')
sns.distplot(df_gpt4o_mini['cosine'], label='4o-mini')
plt.title("RAG LLM performance")
plt.xlabel("A->Q->A' Cosine Similarity")
plt.legend()Output
C:\Users\alexe\AppData\Local\Temp\ipykernel_8108\4043211035.py:3: UserWarning: `distplot` is a deprecated function and will be removed in seaborn v0.14.0. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms). For a guide to updating your code to use the new functions, please see https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751 sns.distplot(df_gpt4o['cosine'], label='4o') C:\Users\alexe\AppData\Local\Temp\ipykernel_8108\4043211035.py:4: UserWarning: `distplot` is a deprecated function and will be removed in seaborn v0.14.0. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms). For a guide to updating your code to use the new functions, please see https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751 sns.distplot(df_gpt4o_mini['cosine'], label='4o-mini')
<matplotlib.legend.Legend at 0x2d4f39b8140>
<Figure size 640x480 with 1 Axes>
LLM-as-a-Judge
In [234]python · cell 64
python
prompt1_template = """
You are an expert evaluator for a Retrieval-Augmented Generation (RAG) system.
Your task is to analyze the relevance of the generated answer compared to the original answer provided.
Based on the relevance and similarity of the generated answer to the original answer, you will classify
it as "NON_RELEVANT", "PARTLY_RELEVANT", or "RELEVANT".
Here is the data for evaluation:
Original Answer: {answer_orig}
Generated Question: {question}
Generated Answer: {answer_llm}
Please analyze the content and context of the generated answer in relation to the original
answer and provide your evaluation in parsable JSON without using code blocks:
{{
"Relevance": "NON_RELEVANT" | "PARTLY_RELEVANT" | "RELEVANT",
"Explanation": "[Provide a brief explanation for your evaluation]"
}}
""".strip()
prompt2_template = """
You are an expert evaluator for a Retrieval-Augmented Generation (RAG) system.
Your task is to analyze the relevance of the generated answer to the given question.
Based on the relevance of the generated answer, you will classify it
as "NON_RELEVANT", "PARTLY_RELEVANT", or "RELEVANT".
Here is the data for evaluation:
Question: {question}
Generated Answer: {answer_llm}
Please analyze the content and context of the generated answer in relation to the question
and provide your evaluation in parsable JSON without using code blocks:
{{
"Relevance": "NON_RELEVANT" | "PARTLY_RELEVANT" | "RELEVANT",
"Explanation": "[Provide a brief explanation for your evaluation]"
}}
""".strip()In [169]python · cell 65
python
df_sample = df_gpt4o_mini.sample(n=150, random_state=1)In [173]python · cell 66
python
samples = df_sample.to_dict(orient='records')In [208]python · cell 67
python
record = samples[0]
recordOutput
{'answer_llm': 'The syntax for using `precision_recall_fscore_support` in Python is as follows:\n\n```python\nfrom sklearn.metrics import precision_recall_fscore_support\nprecision, recall, fscore, support = precision_recall_fscore_support(y_val, y_val_pred, zero_division=0)\n```',
'answer_orig': 'Scikit-learn offers another way: precision_recall_fscore_support\nExample:\nfrom sklearn.metrics import precision_recall_fscore_support\nprecision, recall, fscore, support = precision_recall_fscore_support(y_val, y_val_pred, zero_division=0)\n(Gopakumar Gopinathan)',
'document': '403bbdd8',
'question': 'What is the syntax for using precision_recall_fscore_support in Python?',
'course': 'machine-learning-zoomcamp',
'cosine': 0.9010756015777588}In [209]python · cell 68
python
prompt = prompt1_template.format(**record)
print(prompt)Output
You are an expert evaluator for a Retrieval-Augmented Generation (RAG) system.
Your task is to analyze the relevance of the generated answer compared to the original answer provided.
Based on the relevance and similarity of the generated answer to the original answer, you will classify
it as "NON_RELEVANT", "PARTLY_RELEVANT", or "RELEVANT".
Here is the data for evaluation:
Original Answer: Scikit-learn offers another way: precision_recall_fscore_support
Example:
from sklearn.metrics import precision_recall_fscore_support
precision, recall, fscore, support = precision_recall_fscore_support(y_val, y_val_pred, zero_division=0)
(Gopakumar Gopinathan)
Generated Question: What is the syntax for using precision_recall_fscore_support in Python?
Generated Answer: The syntax for using `precision_recall_fscore_support` in Python is as follows:
```python
from sklearn.metrics import precision_recall_fscore_support
precision, recall, fscore, support = precision_recall_fscore_support(y_val, y_val_pred, zero_division=0)
```
Please analyze the content and context of the generated answer in relation to the original
answer and provide your evaluation in parsable JSON without using code blocks:
{
"Relevance": "NON_RELEVANT" | "PARTLY_RELEVANT" | "RELEVANT",
"Explanation": "[Provide a brief explanation for your evaluation]"
}
In [210]python · cell 69
python
answer = llm(prompt, model='gpt-4o-mini')In [187]python · cell 70
python
import jsonIn [224]python · cell 71
python
evaluations = []
for record in tqdm(samples):
prompt = prompt1_template.format(**record)
evaluation = llm(prompt, model='gpt-4o-mini')
evaluations.append(evaluation)Output
0%| | 0/150 [00:00<?, ?it/s]
In [225]python · cell 72
python
json_evaluations = []
for i, str_eval in enumerate(evaluations):
json_eval = json.loads(str_eval)
json_evaluations.append(json_eval)In [227]python · cell 73
python
df_evaluations = pd.DataFrame(json_evaluations)In [228]python · cell 74
python
df_evaluations.Relevance.value_counts()Output
Relevance RELEVANT 124 PARTLY_RELEVANT 16 NON_RELEVANT 10 Name: count, dtype: int64
In [231]python · cell 75
python
df_evaluations[df_evaluations.Relevance == 'NON_RELEVANT'] #.to_dict(orient='records')Output
Relevance Explanation 4 NON_RELEVANT The generated answer discusses a pip version e... 11 NON_RELEVANT The generated answer does not address the spec... 27 NON_RELEVANT The generated answer incorrectly states that t... 41 NON_RELEVANT The generated answer provides information abou... 87 NON_RELEVANT The generated answer does not address the orig... 90 NON_RELEVANT The generated answer responds to a question ab... 93 NON_RELEVANT The generated answer does not address the topi... 116 NON_RELEVANT The generated answer discusses the recommended... 138 NON_RELEVANT The generated answer addresses a different iss... 139 NON_RELEVANT The generated answer does not relate to the to...
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Relevance | Explanation | |
|---|---|---|
| 4 | NON_RELEVANT | The generated answer discusses a pip version e... |
| 11 | NON_RELEVANT | The generated answer does not address the spec... |
| 27 | NON_RELEVANT | The generated answer incorrectly states that t... |
| 41 | NON_RELEVANT | The generated answer provides information abou... |
| 87 | NON_RELEVANT | The generated answer does not address the orig... |
| 90 | NON_RELEVANT | The generated answer responds to a question ab... |
| 93 | NON_RELEVANT | The generated answer does not address the topi... |
| 116 | NON_RELEVANT | The generated answer discusses the recommended... |
| 138 | NON_RELEVANT | The generated answer addresses a different iss... |
| 139 | NON_RELEVANT | The generated answer does not relate to the to... |
In [232]python · cell 76
python
sample[4]Output
{'answer_llm': "The cause of the pip version error in this week's serverless deep learning section could be a version conflict in Scikit-Learn. Specifically, if you are using a different version than what was used during the model training, it can lead to warnings and potential breaking code or invalid results. To resolve this, make sure to use the same version of Scikit-Learn that was used for training the model. For instance, if you trained with version 1.1.1, you should use that same version in your virtual environment.",
'answer_orig': 'When running docker build -t dino-dragon-model it returns the above error\nThe most common source of this error in this week is because Alex video shows a version of the wheel with python 8, we need to find a wheel with the version that we are working on. In this case python 9. Another common error is to copy the link, this will also produce the same error, we need to download the raw format:\nhttps://github.com/alexeygrigorev/tflite-aws-lambda/raw/main/tflite/tflite_runtime-2.7.0-cp39-cp39-linux_x86_64.whl\nPastor Soto',
'document': '42c09143',
'question': "What might be the cause of the pip version error in this week's serverless deep learning section?",
'course': 'machine-learning-zoomcamp',
'cosine': 0.309404581785202}In [235]python · cell 77
python
prompt = prompt2_template.format(**record)
print(prompt)Output
You are an expert evaluator for a Retrieval-Augmented Generation (RAG) system.
Your task is to analyze the relevance of the generated answer to the given question.
Based on the relevance of the generated answer, you will classify it
as "NON_RELEVANT", "PARTLY_RELEVANT", or "RELEVANT".
Here is the data for evaluation:
Question: What modification was made to the median_house_value target in the homework?
Generated Answer: The modification made to the `median_house_value` target in the homework was that it was changed to binary format. The values were made discrete as either 0 or 1, instead of remaining as a continuous variable. This change was necessary for the calculation of the mutual information score, which is applicable to categorical or discrete variables rather than continuous ones.
Please analyze the content and context of the generated answer in relation to the question
and provide your evaluation in parsable JSON without using code blocks:
{
"Relevance": "NON_RELEVANT" | "PARTLY_RELEVANT" | "RELEVANT",
"Explanation": "[Provide a brief explanation for your evaluation]"
}
In [236]python · cell 78
python
evaluation = llm(prompt, model='gpt-4o-mini')
print(evaluation)Output
{
"Relevance": "RELEVANT",
"Explanation": "The generated answer directly addresses the modification made to the `median_house_value` target by clarifying that it was changed to a binary format. It explains the significance of the change and its relation to the calculation of the mutual information score, which is pertinent to understanding the reasons behind the modification."
}
In [237]python · cell 79
python
evaluations_2 = []
for record in tqdm(samples):
prompt = prompt2_template.format(**record)
evaluation = llm(prompt, model='gpt-4o-mini')
evaluations_2.append(evaluation)Output
0%| | 0/150 [00:00<?, ?it/s]
In [238]python · cell 80
python
json_evaluations_2 = []
for i, str_eval in enumerate(evaluations_2):
json_eval = json.loads(str_eval)
json_evaluations_2.append(json_eval)In [239]python · cell 81
python
df_evaluations_2 = pd.DataFrame(json_evaluations_2)In [242]python · cell 82
python
df_evaluations_2[df_evaluations_2.Relevance == 'NON_RELEVANT']Output
Relevance Explanation 45 NON_RELEVANT The generated answer does not address the ques... 49 NON_RELEVANT The generated answer explicitly states that th... 139 NON_RELEVANT The generated answer provides information abou...
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Relevance | Explanation | |
|---|---|---|
| 45 | NON_RELEVANT | The generated answer does not address the ques... |
| 49 | NON_RELEVANT | The generated answer explicitly states that th... |
| 139 | NON_RELEVANT | The generated answer provides information abou... |
In [243]python · cell 83
python
samples[45]Output
{'answer_llm': 'The provided context does not include specific commands to start the Docker daemon on Linux. Therefore, I cannot provide an answer based solely on the facts from the context.',
'answer_orig': 'Working on getting Docker installed - when I try running hello-world I am getting the error.\nDocker: Cannot connect to the docker daemon at unix:///var/run/docker.sock. Is the Docker daemon running ?\nSolution description\nIf you’re getting this error on WSL, re-install your docker: remove the docker installation from WSL and install Docker Desktop on your host machine (Windows).\nOn Linux, start the docker daemon with either of these commands:\nsudo dockerd\nsudo service docker start\nAdded by Ugochukwu Onyebuchi',
'document': '4b2a3181',
'question': 'What commands should I use to start the docker daemon on Linux?',
'course': 'machine-learning-zoomcamp',
'cosine': 0.51130211353302}Saving all the data
In [244]python · cell 85
python
df_gpt4o.to_csv('data/results-gpt4o-cosine.csv', index=False)
df_gpt35.to_csv('data/results-gpt35-cosine.csv', index=False)
df_gpt4o_mini.to_csv('data/results-gpt4o-mini-cosine.csv', index=False)In [245]python · cell 86
python
df_evaluations.to_csv('data/evaluations-aqa.csv', index=False)
df_evaluations_2.to_csv('data/evaluations-qa.csv', index=False)In [ ]python · cell 87
python