Chapter 21
local rag huggingface faiss
NotebookPython 333 cells
Fully Local RAG with HuggingFace Embeddings and FAISS 🤗
Overview
This notebook demonstrates a fully local, zero-cost RAG pipeline using open-source models from HuggingFace and FAISS for vector search — no paid API key required. Unlike most RAG implementations that rely on OpenAI, this approach runs entirely on your own machine, making it ideal for privacy-sensitive domains such as banking, healthcare, and legal.
Motivation
Most RAG tutorials depend on paid APIs (OpenAI, Anthropic, etc.), which creates barriers for:
- Developers in cost-sensitive or early-stage projects
- Organizations with strict data privacy requirements
- Researchers experimenting without API budgets
This notebook removes those barriers by using:
- 🤗 HuggingFace
BAAI/bge-base-en-v1.5— top-ranked free embedding model (MTEB leaderboard) - ⚡ FAISS — fast local vector similarity search by Meta AI
- 🦜 LangChain LCEL — clean, composable pipeline orchestration
- 🤖 HuggingFace
Zephyr-7b-beta— powerful open-source instruction-tuned LLM
Pipeline
code
PDF Document
│
▼
[ PDF Loader + Text Splitter ] → Chunks
│
▼
[ HuggingFace Embeddings ] → Vectors
│
▼
[ FAISS Vector Store ] ←──── Embedded User Query
│
▼
[ Top-K Retrieved Chunks ]
│
▼
[ Zephyr-7b LLM + Context ] → Grounded AnswerKey Benefits
- ✅ Zero cost — no OpenAI or paid API needed
- ✅ Privacy preserving — data never leaves your machine
- ✅ Production ready — FAISS index can be saved/loaded for repeated use
- ✅ Source attribution — answers cite which document chunks were used
Install Required Packages
In [ ]python · cell 3
python
# Install required packages
!pip install -qU \
langchain \
langchain-community \
langchain-huggingface \
faiss-cpu \
sentence-transformers \
transformers \
huggingface_hub \
accelerate \
pypdfClone Repository and Download Data
In [ ]python · cell 5
python
# Clone the repository to access helper functions and evaluation modules
!git clone https://github.com/NirDiamant/RAG_Techniques.git
import sys
sys.path.append('RAG_Techniques')
# Create data directory and download PDF
import os
os.makedirs('data', exist_ok=True)
!wget -q -O data/Understanding_Climate_Change.pdf \
https://raw.githubusercontent.com/NirDiamant/RAG_Techniques/main/data/Understanding_Climate_Change.pdf
print('Data downloaded successfully.')Set Up HuggingFace Token
Get your free token at: https://huggingface.co/settings/tokens
This is needed to call the HuggingFace Inference API for text generation.
In [ ]python · cell 7
python
import os
from dotenv import load_dotenv
# Load token from .env file, or set it directly below
load_dotenv()
# Option: set directly (not recommended for shared notebooks)
# os.environ["HUGGINGFACEHUB_API_TOKEN"] = "your_token_here"
HF_TOKEN = os.getenv("HUGGINGFACEHUB_API_TOKEN")
assert HF_TOKEN, (
"Please set HUGGINGFACEHUB_API_TOKEN. "
"Get your free token at https://huggingface.co/settings/tokens"
)
print('HuggingFace token loaded successfully.')Import Libraries
In [ ]python · cell 9
python
import warnings
warnings.filterwarnings('ignore')
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_huggingface import HuggingFaceEmbeddings, HuggingFaceEndpoint
from langchain.prompts import PromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
print('Libraries imported successfully.')Define Configuration Parameters
In [ ]python · cell 11
python
# Path to the document
PDF_PATH = "data/Understanding_Climate_Change.pdf"
# Embedding model — top ranked on MTEB leaderboard, runs on CPU
EMBEDDING_MODEL = "BAAI/bge-base-en-v1.5"
# Open-source instruction-tuned LLM via HuggingFace Inference API
LLM_REPO_ID = "HuggingFaceH4/zephyr-7b-beta"
# Chunking parameters — tune based on your document type
CHUNK_SIZE = 1000 # Characters per chunk
CHUNK_OVERLAP = 200 # Overlap to preserve context at chunk boundaries
# Number of chunks to retrieve per query
TOP_K = 3
print(f'Config: chunk_size={CHUNK_SIZE}, overlap={CHUNK_OVERLAP}, top_k={TOP_K}')
print(f'Embedding model: {EMBEDDING_MODEL}')
print(f'LLM: {LLM_REPO_ID}')Load and Split the PDF Document
In [ ]python · cell 13
python
def load_and_split_pdf(path, chunk_size=1000, chunk_overlap=200):
"""
Load a PDF and split it into overlapping chunks.
Args:
path: Path to the PDF file
chunk_size: Max characters per chunk
chunk_overlap: Characters shared between adjacent chunks
Returns:
List of Document chunks with metadata
"""
loader = PyPDFLoader(path)
documents = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=chunk_size,
chunk_overlap=chunk_overlap,
length_function=len,
)
chunks = text_splitter.split_documents(documents)
return chunks
chunks = load_and_split_pdf(PDF_PATH, CHUNK_SIZE, CHUNK_OVERLAP)
print(f'Loaded PDF: {PDF_PATH}')
print(f'Total chunks created: {len(chunks)}')
print(f'\nSample chunk (first 200 chars):')
print(chunks[0].page_content[:200])
print(f'\nMetadata: {chunks[0].metadata}')Create HuggingFace Embeddings
BAAI/bge-base-en-v1.5 is a top performer on the MTEB leaderboard. It runs efficiently on CPU and produces 768-dimensional embeddings. Setting normalize_embeddings=True ensures correct cosine similarity during retrieval.
In [ ]python · cell 15
python
def create_embeddings(model_name=EMBEDDING_MODEL):
"""
Initialize HuggingFace embedding model.
Args:
model_name: HuggingFace model identifier
Returns:
HuggingFaceEmbeddings instance
"""
return HuggingFaceEmbeddings(
model_name=model_name,
model_kwargs={"device": "cpu"}, # Change to 'cuda' if GPU available
encode_kwargs={"normalize_embeddings": True}, # Required for cosine similarity
)
embeddings = create_embeddings()
# Sanity check
test_vec = embeddings.embed_query("What causes climate change?")
print(f'Embedding model loaded: {EMBEDDING_MODEL}')
print(f'Embedding dimension: {len(test_vec)}')Build FAISS Vector Store
FAISS indexes all chunk embeddings locally in memory. For large document sets, save the index to disk to avoid re-embedding on every run.
In [ ]python · cell 17
python
def encode_to_faiss(chunks, embeddings):
"""
Embed document chunks and build a FAISS vector store.
Args:
chunks: List of Document objects
embeddings: Embedding model instance
Returns:
FAISS vector store
"""
vectorstore = FAISS.from_documents(
documents=chunks,
embedding=embeddings,
)
return vectorstore
vectorstore = encode_to_faiss(chunks, embeddings)
print(f'FAISS index built with {vectorstore.index.ntotal} vectors')
# Optional: persist index to disk for reuse
# vectorstore.save_local("faiss_climate_index")
# vectorstore = FAISS.load_local("faiss_climate_index", embeddings, allow_dangerous_deserialization=True)Set Up Retriever and Test It
Always test the retriever independently before attaching the LLM — this isolates retrieval quality from generation quality.
In [ ]python · cell 19
python
retriever = vectorstore.as_retriever(
search_type="similarity",
search_kwargs={"k": TOP_K},
)
# Test retrieval independently
test_query = "What are the main causes of climate change?"
retrieved_docs = retriever.invoke(test_query)
print(f'Query: "{test_query}"')
print(f'Retrieved {len(retrieved_docs)} chunks:\n')
for i, doc in enumerate(retrieved_docs):
print(f' Chunk {i+1} (page {doc.metadata.get("page", "?")}): {doc.page_content[:120]}...')
print()Initialize HuggingFace LLM
zephyr-7b-beta is a strong open-source instruction-tuned model. It runs via the HuggingFace Inference API (free tier available). For fully offline inference, replace with HuggingFacePipeline using a locally downloaded model.
In [ ]python · cell 21
python
llm = HuggingFaceEndpoint(
repo_id=LLM_REPO_ID,
task="text-generation",
max_new_tokens=256,
temperature=0.1, # Low temperature for factual, grounded answers
do_sample=False,
huggingfacehub_api_token=HF_TOKEN,
)
print(f'LLM loaded: {LLM_REPO_ID}')Define RAG Prompt Template
The prompt explicitly instructs the LLM to answer only from retrieved context. This is the key hallucination mitigation technique in RAG — if the answer is not in the context, the model should say so rather than fabricate.
In [ ]python · cell 23
python
RAG_PROMPT_TEMPLATE = """\
You are a helpful assistant. Answer the question using ONLY the context provided below.
If the answer is not contained in the context, respond with:
"I don't have enough information in the provided context to answer this question."
Do not use any prior knowledge or make assumptions.
Context:
{context}
Question: {question}
Answer:"""
prompt = PromptTemplate(
input_variables=["context", "question"],
template=RAG_PROMPT_TEMPLATE,
)
print('Prompt template defined.')Build the Full RAG Chain Using LCEL
LangChain Expression Language (LCEL) allows composing the full RAG pipeline declaratively. Each component is a runnable that can be chained with |.
In [ ]python · cell 25
python
def format_docs(docs):
"""Concatenate retrieved chunks with page source attribution."""
return "\n\n".join(
f"[Page {doc.metadata.get('page', '?')}]\n{doc.page_content}"
for doc in docs
)
# Build RAG chain: retrieve → format → prompt → LLM → parse
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
print('RAG chain built successfully.')Run Queries and Compare with Basic RAG
We test several queries — including an out-of-scope question to verify that the model correctly refuses to answer when context is insufficient.
In [ ]python · cell 27
python
def ask(question, chain=rag_chain):
"""Run a query through the RAG chain and return a clean answer."""
raw = chain.invoke(question)
# Extract only the answer portion (after 'Answer:')
answer = raw.strip().split("Answer:")[-1].strip()
return answer
test_queries = [
"What are the main causes of climate change?",
"What are the effects of climate change on biodiversity?",
"What solutions are proposed to address climate change?",
"What is the GDP of India in 2024?", # Out-of-scope — should be refused
]
for query in test_queries:
print(f"{'='*65}")
print(f"Q: {query}")
print(f"{'-'*65}")
answer = ask(query)
print(f"A: {answer}")
print()Return Answers with Source Attribution
In production RAG systems, showing which source chunks were used builds user trust and enables auditability — especially important in regulated industries like banking and healthcare.
In [ ]python · cell 29
python
# Chain that returns both the answer and the retrieved source documents
rag_chain_with_sources = RunnableParallel(
answer=rag_chain,
sources=retriever,
)
query = "What are the impacts of rising sea levels?"
result = rag_chain_with_sources.invoke(query)
answer = result['answer'].strip().split('Answer:')[-1].strip()
print(f"Q: {query}\n")
print(f"A: {answer}\n")
print("Sources used:")
for doc in result['sources']:
page = doc.metadata.get('page', '?')
preview = doc.page_content[:100].replace('\n', ' ')
print(f" - Page {page}: {preview}...")Evaluate the RAG System
In [ ]python · cell 31
python
# Evaluation using the repo's built-in evaluate_rag helper
try:
from evaluation.evalute_rag import evaluate_rag
evaluate_rag(rag_chain, retriever)
except ImportError:
# Fallback: manual evaluation on known questions
print("Running manual evaluation...\n")
eval_pairs = [
{
"question": "What gases are primarily responsible for the greenhouse effect?",
"expected_keywords": ["carbon dioxide", "CO2", "methane", "greenhouse"],
},
{
"question": "How does deforestation contribute to climate change?",
"expected_keywords": ["carbon", "forest", "absorb", "emissions"],
},
]
for pair in eval_pairs:
answer = ask(pair["question"]).lower()
hits = [kw for kw in pair["expected_keywords"] if kw.lower() in answer]
score = len(hits) / len(pair["expected_keywords"])
print(f"Q: {pair['question']}")
print(f" Keyword hit rate: {score:.0%} ({hits})")
print()Comparison: Local RAG vs Standard RAG
| Aspect | Standard RAG (OpenAI) | Local RAG (This Notebook) |
|---|---|---|
| Cost | ~$0.002/1K tokens | Free |
| Data privacy | Data sent to API | Stays local |
| Setup | API key required | HuggingFace token (free) |
| Embedding model | text-embedding-ada-002 | BAAI/bge-base-en-v1.5 |
| LLM | GPT-3.5/4 | Zephyr-7b-beta |
| Answer quality | Higher (larger model) | Good for most use cases |
| Offline support | ❌ | ✅ (with local model) |
When to Use This Approach
- 🏥 Healthcare / Legal / Finance — data cannot leave your infrastructure
- 💰 Cost-sensitive projects — no per-token billing
- 🔬 Research / prototyping — iterate freely without API budget concerns
- 🌍 Low-connectivity environments — fully offline with local model variant
Next Steps
- Replace
HuggingFaceEndpointwithHuggingFacePipelinefor fully offline inference - Try
MMR(Max Marginal Relevance) retrieval for more diverse results - Combine with reranking (see
reranking.ipynb) for better precision - Scale to millions of docs by migrating from FAISS to Milvus or Pinecone
- Evaluate with RAGAS framework for faithfulness and context relevance metrics