Chapter 21
persistent rag ingest
Notebookllm-zoomcamp (3.12.1)6 cells
In [1]python · cell 1
python
from ingest import load_faq_data
documents = load_faq_data()In [2]python · cell 2
python
docs_llm = [doc for doc in documents if doc['course'] == 'llm-zoomcamp']
len(docs_llm)Output
85
In [ ]python · cell 3
python
from sqlitesearch import TextSearchIndex
index = TextSearchIndex(
text_fields=['question', 'section', 'answer'],
keyword_fields=['course'],
id_field='doc_id', # <-- use the existing id so if it's re-run, it will update existing records instead of creating duplicates
db_path='faq.db'
)In [ ]python · cell 4
python
import time
for doc in docs_llm:
doc['doc_id'] = doc.pop('id') # "id" is already a reserved column in sqlite, need to add the ids from the json as a different field name
index.add(doc)
print(f'Added: {doc["question"][:60]}...')
time.sleep(0.5)Output
Added: I just discovered the course. Can I still join?... Added: Course: I have registered for the LLM Zoomcamp. When can I e... Added: What is the video/zoom link to the stream for the “Office Ho... Added: Cloud alternatives with GPU... Added: Leaderboard: I am not on the leaderboard / how do I know whi... Added: Certificate: Can I follow the course in a self-paced mode an... Added: I missed the first homework - can I still get a certificate?... Added: Homework: Why does the content keep changing?... Added: When will the course be offered next?... Added: Are there any lectures/videos? Where are they?... Added: WSL2: ResponseError: model requires more system memory (X.X ... Added: Server Error (500) When logging in to course homework using ... Added: Why are we not using Langchain in the course?... Added: OpenAI: Error when running OpenAI chat.completions.create co... Added: OpenAI: Error: RateLimitError: Error code: 429 -... Added: OpenAI: Error: 'Cannot import name OpenAI from openai'; How ... Added: OpenAI: How much will I have to spend to use the Open AI API... Added: OpenAI: Do I have to subscribe and pay for Open AI API for t... Added: Authentication: Safe and easy way to store and load API keys... Added: How to store and load API keys using .env file... Added: Authentication: Why is my OPENAI_API_KEY not found in the Ju... Added: OpenSource: I am using Groq, and it doesn't provide a tokeni... Added: OpenSource: Can I use Groq instead of OpenAI?... Added: OpenSource: Can I use open-source alternatives to OpenAI API... Added: OpenAI: "RateLimitError 429 / insufficient_quota" — my accou... Added: Homework: Returning Empty list after filtering my query (HW ... Added: minsearch: SyntaxError "invalid character" when running my d... Added: ModuleNotFoundError on import docx in parse-faq.ipynb... Added: How do I count tokens for a non-OpenAI model (Gemini, Mistra... Added: OpenAI: Why does my token count differ from what OpenAI repo... Added: API keys: how do I set them once and not re-export every ter... Added: Ollama: How to install Ollama?... Added: Connection refused error on prompting the ollam RAG?... Added: Any free models with tool use support?... Added: Agents: "RuntimeError: Already running asyncio in this threa... Added: I passed a float to my tool, but got a validation error sayi... Added: Agents: which non-OpenAI models support tool / function call... Added: Install MCP Inspector... Added: Agents: "AttributeError: 'str' object has no attribute 'outp... Added: Run MCP Inspector... Added: Inspect MCP Server... Added: How to Solve "RuntimeError: Already running asyncio in this ... Added: I am using Azure OpenAI and I am still getting an error of E... Added: What are embeddings?... Added: Warning: 'model "multi-qa-mpnet-base-dot-v1" was made on sen... Added: Why was .dot(...) used directly to compute cosine similarity... Added: Vector search: should I embed the question, the answer, or b... Added: What is the cosine similarity?... Added: Why does cosine similarity reduce to a matrix multiplication... Added: Evaluation: "JSONDecodeError: Expecting value" when generati... Added: Evaluation: Jupyter kernel crashes when embedding the ground... Added: Evaluation: hitting rate limits while generating the ground-... Added: In Windows OS: OSError: [WinError 126] The specified module ... Added: OperationalError when running python prep.py: psycopg2. Oper... Added: How set Pandas to show entire text content in a column. Usef... Added: How to normalize vectors in a Pandas DataFrame column (or Pa... Added: How to compute the quantile or percentile of Pandas DataFram... Added: How can I remove all Docker containers, images, and volumes,... Added: Session State: I want the user to only be able to give feedb... Added: Docker: When trying to run a streamlit app using docker-comp... Added: Is it a group project?... Added: Do we submit 2 projects, what does attempt 1 and 2 mean?... Added: Does the competition count as the capstone?... Added: How is my capstone project going to be evaluated?... Added: When and how will we be assigned projects for review/grading... Added: I’ve already submitted my project. Why can’t I review any pr... Added: How can I find some good ideas or datasets for the project?... Added: Project: do I need an orchestration tool (Airflow, Mage, Kes... Added: Project: how do I evaluate a recommender-style RAG (no obvio... Added: Project: my corpus is large (long PDFs, many paragraphs). Wh... Added: Project: what does "reproducibility" mean — do reviewers nee... Added: Can I use the workshop materials for my own projects or shar... Added: How to set up a new dlt project when loading from cloud?... Added: There is an error when opening the table using `dbtable = db... Added: There is an error when running main(): FileNotFoundError: Ta... Added: How do I know which tables are in the db?... Added: Does DLT have connectors to ClickHouse or StarRocks?... Added: Notebook does not have secret access or 401 Client Error: Un... Added: Error: How to fix requests library only installs v2.28 inste...
In [5]python · cell 5
python
index.close()In [ ]python · cell 6
python