Chapter 37

simple csv rag

NotebookPython 320 cells

🎬 Watch it explained

How Do You Search a Spreadsheet by Meaning? (7 min) — the intuition behind this notebook: turn each row into one labelled line and search the table by meaning, so it answers questions the table never stored.

Simple RAG (Retrieval-Augmented Generation) System for CSV Files

Overview

This code implements a basic Retrieval-Augmented Generation (RAG) system for processing and querying CSV documents. The system encodes the document content into a vector store, which can then be queried to retrieve relevant information.

CSV File Structure and Use Case

The CSV file contains dummy customer data, comprising various attributes like first name, last name, company, etc. This dataset will be utilized for a RAG use case, facilitating the creation of a customer information Q&A system.

Key Components

Loading and spliting csv files.
Vector store creation using FAISS and OpenAI embeddings
Retriever setup for querying the processed documents
Creating a question and answer over the csv data.

Method Details

Document Preprocessing

The csv is loaded using langchain Csvloader
The data is split into chunks.

Vector Store Creation

OpenAI embeddings are used to create vector representations of the text chunks.
A FAISS vector store is created from these embeddings for efficient similarity search.

Retriever Setup

A retriever is configured to fetch the most relevant chunks for a given query.

Benefits of this Approach

Scalability: Can handle large documents by processing them in chunks.
Flexibility: Easy to adjust parameters like chunk size and number of retrieved results.
Efficiency: Utilizes FAISS for fast similarity search in high-dimensional spaces.
Integration with Advanced NLP: Uses OpenAI embeddings for state-of-the-art text representation.

Conclusion

This simple RAG system provides a solid foundation for building more complex information retrieval and question-answering systems. By encoding document content into a searchable vector store, it enables efficient retrieval of relevant information in response to queries. This approach is particularly useful for applications requiring quick access to specific information within a csv file.

import libries

Package Installation and Imports

The cell below installs all necessary packages required to run this notebook.

In [ ]python · cell 5

python

# Install required packages
!pip install faiss-cpu langchain langchain-community langchain-openai pandas python-dotenv

In [8]python · cell 6

python

from langchain_community.document_loaders.csv_loader import CSVLoader
from pathlib import Path
from langchain_openai import ChatOpenAI,OpenAIEmbeddings
import os
from dotenv import load_dotenv

# Load environment variables from a .env file
load_dotenv()

# Set the OpenAI API key environment variable
os.environ["OPENAI_API_KEY"] = os.getenv('OPENAI_API_KEY')

llm = ChatOpenAI(model="gpt-3.5-turbo-0125")

CSV File Structure and Use Case

In [ ]python · cell 8

python

# Download required data files
import os
os.makedirs('data', exist_ok=True)

# Download the PDF document used in this notebook
!wget -O data/Understanding_Climate_Change.pdf https://raw.githubusercontent.com/NirDiamant/RAG_TECHNIQUES/main/data/Understanding_Climate_Change.pdf
!wget -O data/customers-100.csv https://raw.githubusercontent.com/NirDiamant/RAG_TECHNIQUES/main/data/customers-100.csv

In [18]python · cell 9

python

import pandas as pd

file_path = ('data/customers-100.csv') # insert the path of the csv file
data = pd.read_csv(file_path)

#preview the csv file
data.head()

Output

   Index      Customer Id First Name Last Name  \
0      1  DD37Cf93aecA6Dc     Sheryl    Baxter   
1      2  1Ef7b82A4CAAD10    Preston    Lozano   
2      3  6F94879bDAfE5a6        Roy     Berry   
3      4  5Cef8BFA16c5e3c      Linda     Olsen   
4      5  053d585Ab6b3159     Joanna    Bender   

                           Company               City  \
0                  Rasmussen Group       East Leonard   
1                      Vega-Gentry  East Jimmychester   
2                    Murillo-Perry      Isabelborough   
3  Dominguez, Mcmillan and Donovan         Bensonview   
4         Martin, Lang and Andrade     West Priscilla   

                      Country                 Phone 1                Phone 2  \
0                       Chile            229.077.5154       397.884.0519x718   
1                    Djibouti              5153435776       686-620-1820x944   
2         Antigua and Barbuda         +1-539-402-0259    (496)978-3969x58947   
3          Dominican Republic  001-808-617-6467x12895        +1-813-324-8756   
4  Slovakia (Slovak Republic)  001-234-203-0635x76146  001-199-446-3860x3486   

                         Email Subscription Date                      Website  
0     zunigavanessa@smith.info        2020-08-24   http://www.stephenson.com/  
1              vmata@colon.com        2021-04-23        http://www.hobbs.com/  
2          beckycarr@hogan.com        2020-03-25     http://www.lawrence.com/  
3  stanleyblackwell@benson.org        2020-06-02   http://www.good-lyons.com/  
4      colinalvarado@miles.net        2021-04-17  https://goodwin-ingram.com/

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	Index	Customer Id	First Name	Last Name	Company	City	Country	Phone 1	Phone 2	Email	Subscription Date	Website
0	1	DD37Cf93aecA6Dc	Sheryl	Baxter	Rasmussen Group	East Leonard	Chile	229.077.5154	397.884.0519x718	zunigavanessa@smith.info	2020-08-24	http://www.stephenson.com/
1	2	1Ef7b82A4CAAD10	Preston	Lozano	Vega-Gentry	East Jimmychester	Djibouti	5153435776	686-620-1820x944	vmata@colon.com	2021-04-23	http://www.hobbs.com/
2	3	6F94879bDAfE5a6	Roy	Berry	Murillo-Perry	Isabelborough	Antigua and Barbuda	+1-539-402-0259	(496)978-3969x58947	beckycarr@hogan.com	2020-03-25	http://www.lawrence.com/
3	4	5Cef8BFA16c5e3c	Linda	Olsen	Dominguez, Mcmillan and Donovan	Bensonview	Dominican Republic	001-808-617-6467x12895	+1-813-324-8756	stanleyblackwell@benson.org	2020-06-02	http://www.good-lyons.com/
4	5	053d585Ab6b3159	Joanna	Bender	Martin, Lang and Andrade	West Priscilla	Slovakia (Slovak Republic)	001-234-203-0635x76146	001-199-446-3860x3486	colinalvarado@miles.net	2021-04-17	https://goodwin-ingram.com/

load and process csv data

In [19]python · cell 11

python

loader = CSVLoader(file_path=file_path)
docs = loader.load_and_split()

Initiate faiss vector store and openai embedding

In [9]python · cell 13

python

import faiss
from langchain_community.docstore.in_memory import InMemoryDocstore
from langchain_community.vectorstores import FAISS

embeddings = OpenAIEmbeddings()
index = faiss.IndexFlatL2(len(OpenAIEmbeddings().embed_query(" ")))
vector_store = FAISS(
    embedding_function=OpenAIEmbeddings(),
    index=index,
    docstore=InMemoryDocstore(),
    index_to_docstore_id={}
)

Add the splitted csv data to the vector store

In [ ]python · cell 15

python

vector_store.add_documents(documents=docs)

Create the retrieval chain

In [11]python · cell 17

python

from langchain_core.prompts import ChatPromptTemplate
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain

retriever = vector_store.as_retriever()

# Set up system prompt
system_prompt = (
    "You are an assistant for question-answering tasks. "
    "Use the following pieces of retrieved context to answer "
    "the question. If you don't know the answer, say that you "
    "don't know. Use three sentences maximum and keep the "
    "answer concise."
    "\n\n"
    "{context}"
)

prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human", "{input}"),
    
])

# Create the question-answer chain
question_answer_chain = create_stuff_documents_chain(llm, prompt)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)

Query the rag bot with a question based on the CSV data

In [15]python · cell 19

python

answer= rag_chain.invoke({"input": "which company does sheryl Baxter work for?"})
answer['answer']

Output

'Sheryl Baxter works for Rasmussen Group.'

Method Details

Document Preprocessing

The csv is loaded using langchain Csvloader

The data is split into chunks.

Vector Store Creation

OpenAI embeddings are used to create vector representations of the text chunks.

A FAISS vector store is created from these embeddings for efficient similarity search.

Retriever Setup

A retriever is configured to fetch the most relevant chunks for a given query.

Benefits of this Approach

Scalability: Can handle large documents by processing them in chunks.

Flexibility: Easy to adjust parameters like chunk size and number of retrieved results.

Efficiency: Utilizes FAISS for fast similarity search in high-dimensional spaces.

Integration with Advanced NLP: Uses OpenAI embeddings for state-of-the-art text representation.

Conclusion

from langchain_community.document_loaders.csv_loader import CSVLoader from pathlib import Path from langchain_openai import ChatOpenAI,OpenAIEmbeddings import os from dotenv import load_dotenv # Load environment variables from a .env file load_dotenv() # Set the OpenAI API key environment variable os.environ["OPENAI_API_KEY"] = os.getenv('OPENAI_API_KEY') llm = ChatOpenAI(model="gpt-3.5-turbo-0125")

# Download required data files import os os.makedirs('data', exist_ok=True) # Download the PDF document used in this notebook !wget -O data/Understanding_Climate_Change.pdf https://raw.githubusercontent.com/NirDiamant/RAG_TECHNIQUES/main/data/Understanding_Climate_Change.pdf !wget -O data/customers-100.csv https://raw.githubusercontent.com/NirDiamant/RAG_TECHNIQUES/main/data/customers-100.csv

Index Customer Id First Name Last Name \ 0 1 DD37Cf93aecA6Dc Sheryl Baxter 1 2 1Ef7b82A4CAAD10 Preston Lozano 2 3 6F94879bDAfE5a6 Roy Berry 3 4 5Cef8BFA16c5e3c Linda Olsen 4 5 053d585Ab6b3159 Joanna Bender Company City \ 0 Rasmussen Group East Leonard 1 Vega-Gentry East Jimmychester 2 Murillo-Perry Isabelborough 3 Dominguez, Mcmillan and Donovan Bensonview 4 Martin, Lang and Andrade West Priscilla Country Phone 1 Phone 2 \ 0 Chile 229.077.5154 397.884.0519x718 1 Djibouti 5153435776 686-620-1820x944 2 Antigua and Barbuda +1-539-402-0259 (496)978-3969x58947 3 Dominican Republic 001-808-617-6467x12895 +1-813-324-8756 4 Slovakia (Slovak Republic) 001-234-203-0635x76146 001-199-446-3860x3486 Email Subscription Date Website 0 zunigavanessa@smith.info 2020-08-24 http://www.stephenson.com/ 1 vmata@colon.com 2021-04-23 http://www.hobbs.com/ 2 beckycarr@hogan.com 2020-03-25 http://www.lawrence.com/ 3 stanleyblackwell@benson.org 2020-06-02 http://www.good-lyons.com/ 4 colinalvarado@miles.net 2021-04-17 https://goodwin-ingram.com/

Index

Customer Id

First Name

Last Name

Company

City

Country

Phone 1

Phone 2

Subscription Date

Website

DD37Cf93aecA6Dc

Sheryl

Baxter

Rasmussen Group

East Leonard

Chile

229.077.5154

397.884.0519x718

zunigavanessa@smith.info

2020-08-24

http://www.stephenson.com/

1Ef7b82A4CAAD10

Preston

Lozano

Vega-Gentry

East Jimmychester

Djibouti

5153435776

686-620-1820x944

vmata@colon.com

2021-04-23

http://www.hobbs.com/

6F94879bDAfE5a6

Roy

Berry

Murillo-Perry

Isabelborough

Antigua and Barbuda

+1-539-402-0259

(496)978-3969x58947

beckycarr@hogan.com

2020-03-25

http://www.lawrence.com/

5Cef8BFA16c5e3c

Linda

Olsen

Dominguez, Mcmillan and Donovan

Bensonview

Dominican Republic

001-808-617-6467x12895

+1-813-324-8756

stanleyblackwell@benson.org

2020-06-02

http://www.good-lyons.com/

053d585Ab6b3159

Joanna

Bender

Martin, Lang and Andrade

West Priscilla

Slovakia (Slovak Republic)

001-234-203-0635x76146

001-199-446-3860x3486

colinalvarado@miles.net

2021-04-17

https://goodwin-ingram.com/

import faiss from langchain_community.docstore.in_memory import InMemoryDocstore from langchain_community.vectorstores import FAISS embeddings = OpenAIEmbeddings() index = faiss.IndexFlatL2(len(OpenAIEmbeddings().embed_query(" "))) vector_store = FAISS( embedding_function=OpenAIEmbeddings(), index=index, docstore=InMemoryDocstore(), index_to_docstore_id={} )

from langchain_core.prompts import ChatPromptTemplate from langchain.chains import create_retrieval_chain from langchain.chains.combine_documents import create_stuff_documents_chain retriever = vector_store.as_retriever() # Set up system prompt system_prompt = ( "You are an assistant for question-answering tasks. " "Use the following pieces of retrieved context to answer " "the question. If you don't know the answer, say that you " "don't know. Use three sentences maximum and keep the " "answer concise." "\n\n" "{context}" ) prompt = ChatPromptTemplate.from_messages([ ("system", system_prompt), ("human", "{input}"), ]) # Create the question-answer chain question_answer_chain = create_stuff_documents_chain(llm, prompt) rag_chain = create_retrieval_chain(retriever, question_answer_chain)