Chapter 107
demo es
NotebookPython 3 (ipykernel)30 cells
Architecture - Semantic Search using Elastic Search
-
Two very important concepts in Elasticsearch are documents and indexes.
-
A document is collection of fields with their associated values.
-
To work with Elasticsearch you have to organize your data into documents, and then add all your documents to an index.
-
Index as a collection of documents that is stored in a highly optimized format designed to perform efficient searches.
Step 1: Prepare documents
In [1]python · cell 5
python
import json
with open('documents.json', 'rt') as f_in:
docs_raw = json.load(f_in)In [3]python · cell 6
python
documents = []
for course_dict in docs_raw:
for doc in course_dict['documents']:
doc['course'] = course_dict['course']
documents.append(doc)
documents[1]Output
{'text': 'GitHub - DataTalksClub data-engineering-zoomcamp#prerequisites',
'section': 'General course-related questions',
'question': 'Course - What are the prerequisites for this course?',
'course': 'data-engineering-zoomcamp'}Step 2: Create Embeddings using Pretrained Models
Sentence Transformers documentation here: https://www.sbert.net/docs/sentence_transformer/pretrained_models.html
In [4]python · cell 9
python
# This is a new library compared to the previous modules.
# Please perform "pip install sentence_transformers==2.7.0"
from sentence_transformers import SentenceTransformer
# if you get an error do the following:
# 1. Uninstall numpy
# 2. Uninstall torch
# 3. pip install numpy==1.26.4
# 4. pip install torch
# run the above cell, it should work
model = SentenceTransformer("all-mpnet-base-v2")Output
c:\Users\balaj\OneDrive\Documents\Course\Vector_DB\.venv\lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
In [6]python · cell 10
python
len(model.encode("This is a simple sentence"))Output
768
In [9]python · cell 11
python
documents[1]Output
{'text': 'GitHub - DataTalksClub data-engineering-zoomcamp#prerequisites',
'section': 'General course-related questions',
'question': 'Course - What are the prerequisites for this course?',
'course': 'data-engineering-zoomcamp'}In [10]python · cell 12
python
#created the dense vector using the pre-trained model
operations = []
for doc in documents:
# Transforming the title into an embedding using the model
doc["text_vector"] = model.encode(doc["text"]).tolist()
operations.append(doc)Step 3: Setup ElasticSearch connection
In [12]python · cell 14
python
from elasticsearch import Elasticsearch
es_client = Elasticsearch('http://localhost:9200')
es_client.info()Output
ObjectApiResponse({'name': 'c89345ae8aca', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'Uj1CMXaIR3-EqjEuEXx5Yw', 'version': {'number': '8.4.3', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': '42f05b9372a9a4a470db3b52817899b99a76ee73', 'build_date': '2022-10-04T07:17:24.662462378Z', 'build_snapshot': False, 'lucene_version': '9.3.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'})Step 4: Create Mappings and Index
-
Mapping is the process of defining how a document, and the fields it contains, are stored and indexed.
-
Each document is a collection of fields, which each have their own data type.
-
We can compare mapping to a database schema in how it describes the fields and properties that documents hold, the datatype of each field (e.g., string, integer, or date), and how those fields should be indexed and stored
In [13]python · cell 17
python
index_settings = {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"properties": {
"text": {"type": "text"},
"section": {"type": "text"},
"question": {"type": "text"},
"course": {"type": "keyword"} ,
"text_vector": {"type": "dense_vector", "dims": 768, "index": True, "similarity": "cosine"},
}
}
}In [14]python · cell 18
python
index_name = "course-questions"
es_client.indices.delete(index=index_name, ignore_unavailable=True)
es_client.indices.create(index=index_name, body=index_settings)Output
ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'course-questions'})Step 5: Add documents into index
In [16]python · cell 20
python
for doc in operations:
try:
es_client.index(index=index_name, document=doc)
except Exception as e:
print(e)Step 6: Create end user query
In [17]python · cell 22
python
search_term = "windows or mac?"
vector_search_term = model.encode(search_term)In [18]python · cell 23
python
query = {
"field": "text_vector",
"query_vector": vector_search_term,
"k": 5,
"num_candidates": 10000,
}In [19]python · cell 24
python
res = es_client.search(index=index_name, knn=query, source=["text", "section", "question", "course"])
res["hits"]["hits"]Output
[{'_index': 'course-questions',
'_id': '-BY3YZABY4SUV-QiWtCV',
'_score': 0.7147919,
'_source': {'question': 'Environment - Is the course [Windows/mac/Linux/...] friendly?',
'course': 'data-engineering-zoomcamp',
'section': 'General course-related questions',
'text': 'Yes! Linux is ideal but technically it should not matter. Students last year used all 3 OSes successfully'}},
{'_index': 'course-questions',
'_id': 'CxY4YZABY4SUV-QiNdSb',
'_score': 0.6134734,
'_source': {'question': 'WSL instructions',
'course': 'mlops-zoomcamp',
'section': 'Module 1: Introduction',
'text': 'If you wish to use WSL on your windows machine, here are the setup instructions:\nCommand: Sudo apt install wget\nGet Anaconda download address here. wget <download address>\nTurn on Docker Desktop WFree Download | AnacondaSL2\nCommand: git clone <github repository address>\nVSCODE on WSL\nJupyter: pip3 install jupyter\nAdded by Gregory Morris (gwm1980@gmail.com)\nAll in all softwares at one shop:\nYou can use anaconda which has all built in services like pycharm, jupyter\nAdded by Khaja Zaffer (khajazaffer@aln.iseg.ulisboa.pt)\nFor windows “wsl --install” in Powershell\nAdded by Vadim Surin (vdmsurin@gmai.com)'}},
{'_index': 'course-questions',
'_id': 'zBY3YZABY4SUV-Qi3dIs',
'_score': 0.60555583,
'_source': {'question': "The answer I get for one of the homework questions doesn't match any of the options. What should I do?",
'course': 'machine-learning-zoomcamp',
'section': '2. Machine Learning for Regression',
'text': 'That’s normal. We all have different environments: our computers have different versions of OS and different versions of libraries — even different versions of Python.\nIf it’s the case, just select the option that’s closest to your answer'}},
{'_index': 'course-questions',
'_id': 'JRY3YZABY4SUV-Qi9dPn',
'_score': 0.60289603,
'_source': {'question': 'How to install WSL on Windows 10 and 11 ?',
'course': 'machine-learning-zoomcamp',
'section': '5. Deploying Machine Learning Models',
'text': 'It is quite simple, and you can follow these instructions here:\nhttps://www.youtube.com/watch?v=qYlgUDKKK5A&ab_channel=NeuralNine\nMake sure that you have “Virtual Machine Platform” feature activated in your Windows “Features”. To do that, search “features” in the research bar and see if the checkbox is selected. You also need to make sure that your system (in the bios) is able to virtualize. This is usually the case.\nIn the Microsoft Store: look for ‘Ubuntu’ or ‘Debian’ (or any linux distribution you want) and install it\nOnce it is downloaded, open the app and choose a username and a password (secured one). When you type your password, nothing will show in the window, which is normal: the writing is invisible.\nYou are now inside of your linux system. You can test some commands such as “pwd”. You are not in your Windows system.\nTo go to your windows system: you need to go back two times with cd ../.. And then go to the “mnt” directory with cd mnt. If you list here your files, you will see your disks. You can move to the desired folder, for example here I moved to the ML_Zoomcamp folder:\nPython should be already installed but you can check it by running sudo apt install python3 command.\nYou can make your actual folder your default folder when you open your Ubuntu terminal with this command : echo "cd ../../mnt/your/folder/path" >> ~/.bashrc\nYou can disable bell sounds (when you type something that does not exist for example) by modifying the inputrc file with this command: sudo vim /etc/inputrc\nYou have to uncomment the set bell-style none line -> to do that, press the “i” keyboard letter (for insert) and go with your keyboard to this line. Delete the # and then press the Escape keyboard touch and finally press “:wq” to write (it saves your modifications) then quit.\nYou can check that your modifications are taken into account by opening a new terminal (you can pin it to your task bar so you do not have to go to the Microsoft app each time).\nYou will need to install pip by running this command sudo apt install python3-pip\nNB: I had this error message when trying to install pipenv (https://github.com/microsoft/WSL/issues/5663):\n/sbin/ldconfig.real: Can\'t link /usr/lib/wsl/lib/libnvoptix_loader.so.1 to libnvoptix.so.1\n/sbin/ldconfig.real: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link\nSo I had to create the following symbolic link:\nsudo ln -s /usr/lib/wsl/lib/libcuda.so.1 /usr/lib64/libcuda.so\n(Mélanie Fouesnard)'}},
{'_index': 'course-questions',
'_id': '6RY3YZABY4SUV-QiVtBc',
'_score': 0.5985867,
'_source': {'question': 'Environment - Should I use my local machine, GCP, or GitHub Codespaces for my environment?',
'course': 'data-engineering-zoomcamp',
'section': 'General course-related questions',
'text': 'You can set it up on your laptop or PC if you prefer to work locally from your laptop or PC.\nYou might face some challenges, especially for Windows users. If you face cnd2\nIf you prefer to work on the local machine, you may start with the week 1 Introduction to Docker and follow through.\nHowever, if you prefer to set up a virtual machine, you may start with these first:\nUsing GitHub Codespaces\nSetting up the environment on a cloudV Mcodespace\nI decided to work on a virtual machine because I have different laptops & PCs for my home & office, so I can work on this boot camp virtually anywhere.'}}]Step 7: Perform Keyword search with Semantic Search (Hybrid/Advanced Search)
In [ ]python · cell 26
python
# Note: I made a minor modification to the query shown in the notebook here
# (compare to the one shown in the video)
# Included "knn" in the search query (to perform a semantic search) along with the filter
knn_query = {
"field": "text_vector",
"query_vector": vector_search_term,
"k": 5,
"num_candidates": 10000
}In [ ]python · cell 27
python
response = es_client.search(
index=index_name,
query={
"match": {"section": "General course-related questions"},
},
knn=knn_query,
size=5
)In [ ]python · cell 28
python
response["hits"]["hits"]