Chapter 114
rag intro
NotebookPython 3 (ipykernel)31 cells
In [ ]python · cell 1
python
!pip install minsearchIn [1]python · cell 2
python
import minsearchIn [2]python · cell 3
python
import jsonIn [3]python · cell 4
python
with open('documents.json', 'rt') as f_in:
docs_raw = json.load(f_in)In [4]python · cell 5
python
documents = []
for course_dict in docs_raw:
for doc in course_dict['documents']:
doc['course'] = course_dict['course']
documents.append(doc)In [5]python · cell 6
python
documents[0]Output
{'text': "The purpose of this document is to capture frequently asked technical questions\nThe exact day and hour of the course will be 15th Jan 2024 at 17h00. The course will start with the first “Office Hours'' live.1\nSubscribe to course public Google Calendar (it works from Desktop only).\nRegister before the course starts using this link.\nJoin the course Telegram channel with announcements.\nDon’t forget to register in DataTalks.Club's Slack and join the channel.",
'section': 'General course-related questions',
'question': 'Course - When will the course start?',
'course': 'data-engineering-zoomcamp'}In [6]python · cell 7
python
index = minsearch.Index(
text_fields=["question", "text", "section"],
keyword_fields=["course"]
)SELECT * WHERE course = 'data-engineering-zoomcamp';
In [7]python · cell 9
python
q = 'the course has already started, can I still enroll?'In [8]python · cell 10
python
index.fit(documents)Output
<minsearch.Index at 0x7d0d016b8760>
In [9]python · cell 11
python
from openai import OpenAIIn [10]python · cell 12
python
client = OpenAI()In [11]python · cell 13
python
response = client.chat.completions.create(
model='gpt-4o',
messages=[{"role": "user", "content": q}]
)
response.choices[0].message.contentOutput
"It's not uncommon for courses to accept enrollments even after they have started, but policies can vary widely depending on the institution or provider offering the course. Here are a few steps you can take to find out if you can still enroll:\n\n1. **Check the Course Platform**: If the course is offered online, visit the course's webpage for information about late enrollment policies.\n\n2. **Contact the Instructor**: Reach out to the course instructor or lead facilitator. They may be willing to make an exception or provide you with the necessary information.\n\n3. **Reach Out to Administrative Offices**: Contact the academic or administrative office responsible for course enrollments. This might be the registrar's office, student services, or a similar department.\n\n4. **Review Deadlines and Policies**: Look for any publicly available documentation outlining the deadlines and policies regarding late enrollments.\n\n5. **Consider Catching Up**: Be prepared to quickly catch up on any missed material if you are allowed to enroll late. This shows commitment and can make it easier for instructors or administration to accommodate your request.\n\nRemember, clear and courteous communication will always help you in these situations. Good luck!"
In [12]python · cell 14
python
def search(query):
boost = {'question': 3.0, 'section': 0.5}
results = index.search(
query=query,
filter_dict={'course': 'data-engineering-zoomcamp'},
boost_dict=boost,
num_results=5
)
return resultsIn [13]python · cell 15
python
def build_prompt(query, search_results):
prompt_template = """
You're a course teaching assistant. Answer the QUESTION based on the CONTEXT from the FAQ database.
Use only the facts from the CONTEXT when answering the QUESTION.
QUESTION: {question}
CONTEXT:
{context}
""".strip()
context = ""
for doc in search_results:
context = context + f"section: {doc['section']}\nquestion: {doc['question']}\nanswer: {doc['text']}\n\n"
prompt = prompt_template.format(question=query, context=context).strip()
return promptIn [14]python · cell 16
python
def llm(prompt):
response = client.chat.completions.create(
model='gpt-4o',
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.contentIn [15]python · cell 17
python
query = 'how do I run kafka?'
def rag(query):
search_results = search(query)
prompt = build_prompt(query, search_results)
answer = llm(prompt)
return answerIn [16]python · cell 18
python
rag(query)Output
"To run Kafka, follow the relevant instructions based on your use case:\n\n### For Java:\nNavigate to your project directory and use the following command in the terminal to run a Kafka producer/consumer/KStreams, etc.:\n```shell\njava -cp build/libs/<jar_name>-1.0-SNAPSHOT.jar:out src/main/java/org/example/JsonProducer.java\n```\nReplace `<jar_name>` with the actual name of your JAR file.\n\n### For Python:\nIf you're running Python Kafka, ensure you have your virtual environment set up and activate it. Here’s how you can do it:\n\n1. **Create a virtual environment and install required packages (run only once):**\n ```shell\n python -m venv env\n source env/bin/activate\n pip install -r ../requirements.txt\n ```\n\n2. **Activate the virtual environment (run this every time you need it):**\n ```shell\n source env/bin/activate\n ```\n\n3. **Deactivate the virtual environment when done:**\n ```shell\n deactivate\n ```\n\nNote: On Windows, the activation command would be slightly different:\n```shell\nenv\\Scripts\\activate\n```\n\nMake sure that your Docker images are up and running if they are part of your setup."
In [17]python · cell 19
python
rag('the course has already started, can I still enroll?')Output
'Yes, you can still enroll in the course even after it has started. You are eligible to submit homework assignments, but please be mindful of the deadlines for the final projects to ensure you complete everything on time.'
In [18]python · cell 20
python
documents[0]Output
{'text': "The purpose of this document is to capture frequently asked technical questions\nThe exact day and hour of the course will be 15th Jan 2024 at 17h00. The course will start with the first “Office Hours'' live.1\nSubscribe to course public Google Calendar (it works from Desktop only).\nRegister before the course starts using this link.\nJoin the course Telegram channel with announcements.\nDon’t forget to register in DataTalks.Club's Slack and join the channel.",
'section': 'General course-related questions',
'question': 'Course - When will the course start?',
'course': 'data-engineering-zoomcamp'}In [20]python · cell 21
python
from elasticsearch import ElasticsearchIn [21]python · cell 22
python
es_client = Elasticsearch('http://localhost:9200') In [23]python · cell 23
python
index_settings = {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"properties": {
"text": {"type": "text"},
"section": {"type": "text"},
"question": {"type": "text"},
"course": {"type": "keyword"}
}
}
}
index_name = "course-questions"
es_client.indices.create(index=index_name, body=index_settings)Output
ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'course-questions'})In [24]python · cell 24
python
documents[0]Output
{'text': "The purpose of this document is to capture frequently asked technical questions\nThe exact day and hour of the course will be 15th Jan 2024 at 17h00. The course will start with the first “Office Hours'' live.1\nSubscribe to course public Google Calendar (it works from Desktop only).\nRegister before the course starts using this link.\nJoin the course Telegram channel with announcements.\nDon’t forget to register in DataTalks.Club's Slack and join the channel.",
'section': 'General course-related questions',
'question': 'Course - When will the course start?',
'course': 'data-engineering-zoomcamp'}In [25]python · cell 25
python
from tqdm.auto import tqdmOutput
/usr/local/python/3.10.13/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
In [26]python · cell 26
python
for doc in tqdm(documents):
es_client.index(index=index_name, document=doc)Output
100%|██████████████████████████████████| 948/948 [00:28<00:00, 33.07it/s]
In [36]python · cell 27
python
query = 'I just disovered the course. Can I still join it?'In [42]python · cell 28
python
def elastic_search(query):
search_query = {
"size": 5,
"query": {
"bool": {
"must": {
"multi_match": {
"query": query,
"fields": ["question^3", "text", "section"],
"type": "best_fields"
}
},
"filter": {
"term": {
"course": "data-engineering-zoomcamp"
}
}
}
}
}
response = es_client.search(index=index_name, body=search_query)
result_docs = []
for hit in response['hits']['hits']:
result_docs.append(hit['_source'])
return result_docsIn [44]python · cell 29
python
def rag(query):
search_results = elastic_search(query)
prompt = build_prompt(query, search_results)
answer = llm(prompt)
return answerIn [45]python · cell 30
python
rag(query)Output
'Yes, you can still join the course even if you discovered it after the start date. You are eligible to submit the homeworks, but be mindful of the deadlines for turning in the final projects. So make sure not to leave everything for the last minute.'
In [ ]python · cell 31
python