Chapter 22

memorag

NotebookPython 374 cells

Overview

MemoRAG is a Retrieval-Augmented Generation (RAG) framework that incorporates a memory model as an auxiliary step before the retrieval phase. In doing so, it bridges the gap in contextual understanding and reasoning that standard RAG techniques face when addressing queries with implicit or ambiguous information needs and unstructured external knowledge.

Motivation

Standard RAG techniques rely heavily on lexical or semantic matching between the query and the knowledge base. While this approach works well for clear question answering tasks with structured knowledge, it often falls short when handling queries with implicit or ambiguous information (e.g., describing the relationships between main characters in a novel) or when the knowledge base is unstructured (e.g., fiction books). In such cases, lexical or semantic matching seldom produces the desired outputs.

Key Components

Memory: A compressed representation of the database created by a long-context model, designed to handle and summarize extensive inputs efficiently.
Retriever: A standard RAG retrieval model responsible for selecting relevant context from the knowledge base to support the generator.
Generator: A generative language model that produces responses by combining the query with the retrieved context, similar to standard RAG setups.

Method Details

1. Memory

The memory module serves as an auxiliary component to enhance the retriever’s ability to identify better matches between queries and relevant parts of the database. It takes the original query and the database as inputs and produces staging answers — intermediate outputs like clues, surrogate queries, or key points — which the retriever uses instead of the original query.
Long-term memory is constructed by running a long-context model, such as Qwen2-7B-Instruct or Mistral-7B-Instruct-v0.2, over the entire database. This process generates a compressed representation of the database through an attention mechanism.
The compressed representation is stored as key-value pairs (kv-cache), facilitating efficient and accurate retrieval.
Released memory models include memorag-qwen2-7b-inst and memorag-mistral-7b-inst, derived from Qwen2-7B-Instruct and Mistral-7B-Instruct-v0.2, respectively.

2. Retriever

The retriever is a standard retrieval model, adapted to take processed queries (created by the memory module as staging answers) instead of the original query.
It outputs the retrieved context, which serves as the basis for generating the final answer.

3. Generator

The generator produces the final response by combining the retriever’s output (retrieved context) with the original query.
MemoRAG ensures compatibility and consistency by using the memory module’s underlying model as the default generator.

Benefits of the Approach

Extended Scope of Queries: MemoRAG's preprocessing capabilities enable it to handle complex and long-context tasks that conventional RAG methods struggle with.
Improved Accuracy: By simplifying and adjusting queries before retrieval, MemoRAG enhances performance over standard RAG methods.
Flexibility: Adapts to diverse tasks, datasets, and retrieval scenarios.
Robustness: Improved performance remains consistent across various generators, datasets, and query types.
Efficiency: The use of key-value compression reduces computational overhead.

Conclusion

The memory module in MemoRAG significantly enhances comprehension of both the queries and the database, enabling more effective retrieval. Its ability to preprocess queries, generate staging answers, and leverage long-context memory models ensures high-quality responses, making MemoRAG a significant step forward in the evolution of retrieval-augmented generation.

Implementation

Imports

In [ ]python · cell 5

python

import os
import pandas as pd
from dotenv import load_dotenv
from langchain_community.embeddings import OpenAIEmbeddings
from openai import OpenAI
from helper_functions import *

OpenAI Setup for querying GPT's API

In [ ]python · cell 7

python

load_dotenv()
client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

Memory Module Classes

Helper classes

Used for initial memorization

In [ ]python · cell 10

python

class KeyValuePair(BaseModel):
    """ The key-value pairs are the "holders" of memory
        The keys are topics from the relevant article and the values are details about that topic """
    topic: str
    details: str

class MemoryResponse(BaseModel):
    """ The obtained memory is a list of topic-details pairs """
    pairs: List[KeyValuePair]

MemoryStore

A class for memory creation and knowledge extraction from it when necessary. Once memory is established, the updated retrieval queries can be created from it. Note that the obtained memory can be saved for future use. Below are prompts that are used in the memorization / retrieval queries creation, and then the class itself.

A system prompt for assigning the model with the role of an expert of memorization

In [ ]python · cell 13

python

memory_store_system_prompt = """
    You are an expert at extracting structured key-value pairs from text.
    "Identify key topics, entities, or questions and pair them with relevant, detailed information.
    "Ensure the output is well-structured, avoiding redundancy while preserving meaning.
    """

A prompt instructing the LLM about the process of memory creation

In [ ]python · cell 15

python

memory_store_kv_cache_prompt = """
    You are provided with a long article, chunk by chunk. Read each chunk carefully and extract key topics and their detailed information.
    For each key topic:
    1. Identify the main concept, entity, or potential question
    2. Provide the corresponding detailed information or answer

    Note: the aim is to mimic kv cache memory creation

    Now, the article begins:
    {document}

    The article ends here.

    RESPONSE FORMAT IS IMPORTANT. You must return a JSON object with the following structure:
    {{
        "pairs": [
            {{
                "topic": "topic1",
                "details": "details1"
            }},
            {{
                "topic": "topic2",
                "details": "details2"
            }}
        ]
    }}
    """

A system prompt for assigning the model with the role of an expert for creating queries for info retrieval

In [ ]python · cell 17

python

system_prompt_for_queries_creation = "You are an expert in information retrieval. Your task is to extract specific and relevant information from the provided context to help answer the given question."

A prompt guiding the LLM on how to find clues in the memory that can help answer the given question

In [ ]python · cell 19

python

memorag_span_prompt_for_queries_creation = """
    You are given a question related to an article. To answer it effectively, you need to use specific details from the article. You are not provided with the whole article. Instead, you are provided with specific and relevant information from it.
     Your task is to identify and extract one or more specific clue texts from the provided information that are relevant to the question.

    ### Question: {question}
    ### Information: {relevant_info}
    ### Instructions:
    1. You have a general understanding of the provided information. Your task is to generate one or more specific clues that will help in searching for supporting evidence within the article.
    2. The clues are in the form of text spans that will assist in answering the question.
    3. Only output the clues. If there are multiple clues, separate them with a newline.
    """

A prompt guiding the LLM on how to create surrogate (helper) queries from memory

In [ ]python · cell 21

python

memorag_sur_prompt_for_queries_creation = """
    You are given a question related to an article. To answer it effectively, you need to use specific details from the article.  You are not provided with the whole article. Instead, you are provided with specific and relevant information from it.
    Your task is to generate precise clue questions that can help locate the necessary information for answering the question.

    ### Question: {question}
    ### Information: {relevant_info}
    ### Instructions:
    1. You have a general understanding of the provided information. Your task is to generate one or more specific clues that will help in searching for supporting evidence within the article.
    2. The clues are in the form of precise surrogate questions that clarify the original question.
    3. Only output the clues. If there are multiple clues, separate them with a newline.
    """

The Memory Class - as mentioned above it can "memorize" the text by creating kv pairs, and later on creating adjusted queries that, in turn, will be transferred to the retrieval model

In [ ]python · cell 23

python

class MemoryStore:
    """The MemoryStore class is a realization of the Memory Module discussed in the paper.
    Its 'memorize' method is used to create a prompt cache mimicking a key-value compression of the original text (database).
    This cache can then be used by the 'create_retrieval_prompt' method for creating the processed query to be used later for retrieval"""

    def __init__(self):
        self.embeddings = OpenAIEmbeddings()
        self.store = None
        self.processed_count = 0
        self.json_parse_failures = 0
        self._last_parse_used_fallback = False


    def memorize(self, document: str):
        """Process document into key-value pairs and store them"""
        system_prompt = memory_store_system_prompt
        kv_cache_prompt = memory_store_kv_cache_prompt
        print(f"Processing chunk {self.processed_count + 1}...")

        response = client.chat.completions.create(
            model="gpt-4o-mini", # 4o or o1 should be considered
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": kv_cache_prompt.format(document=document)}
            ],
            response_format={"type": "json_object"},
            seed=42  # For reproducibility
        )

        pairs = self._parse_into_pairs(response.choices[0].message.content)
        if hasattr(self, 'json_parse_failures') and self._last_parse_used_fallback:
            self.json_parse_failures += 1
            if self.json_parse_failures > 0 and self.json_parse_failures % 5 == 0:
               print(f"Warning: JSON parsing has failed {self.json_parse_failures} times")

        print('*'*20)
        print(f'first extracted pairs: {pairs[0:3]}\n\n')

        # Batch process pairs
        texts = []
        metadatas = []

        for topic, details in pairs:
            if not topic or not details:  # Skip empty pairs
                continue

            combined_text = f"Topic: {topic}\nDetails: {details}"
            texts.append(combined_text)
            metadatas.append({"topic": topic})
        print('*'*20)
        print(f'num of extracted texts: {len(texts)}\n\n')
        if self.store is None:
            self.store = FAISS.from_texts(texts, self.embeddings, metadatas=metadatas)
        else:
            existing_docs = self.store.similarity_search("")
            existing_topics = {doc.metadata.get("topic") for doc in existing_docs}

            # Filter out duplicates
            new_texts = []
            new_metadatas = []
            for text, metadata in zip(texts, metadatas):
                if metadata["topic"] not in existing_topics:
                    new_texts.append(text)
                    new_metadatas.append(metadata)

            if new_texts:
                self.store.add_texts(new_texts, metadatas=new_metadatas)

        self.processed_count += 1
        print(f"Processed {self.processed_count} chunks so far\n\n")

    def _parse_into_pairs(self, content: str):
        """Used when LLM returned a valid JSON"""
        self._last_parse_used_fallback = False
        try:
            # Parse JSON and validate with Pydantic
            data = json.loads(content)
            validated_data = MemoryResponse(**data)
            return [(pair.topic, pair.details) for pair in validated_data.pairs]
        except (json.JSONDecodeError, ValueError) as e:
            print(f"Error parsing response: {e}")
            # Fall back to text parsing if JSON fails
            self._last_parse_used_fallback = True
            return self._parse_text_into_pairs(content)

    def _parse_text_into_pairs(self, text: str):
        """Original text parsing method as fallback
        Used when LLM did not return a valid JSON"""
        pairs = []
        lines = text.split('\n')
        current_topic = None
        current_details = []

        for line in lines:
            if line.startswith('Topic:'):
                if current_topic:  # Save previous pair
                    pairs.append((current_topic, ' '.join(current_details)))
                current_topic = line[6:].strip()
                current_details = []
            elif line.startswith('Details:'):
                current_details.append(line[8:].strip())
            elif current_details:  # Continue adding to details if already started
                current_details.append(line.strip())

        # Add the last pair
        if current_topic:
            pairs.append((current_topic, ' '.join(current_details)))

        return pairs

    def create_retrieval_queries(self, query: str) -> str | None:
        """Generate staging answers y = Θ_mem(q, D | θ_mem)
        This should provide rough clues/outline to guide context retrieval"""

        if not self.store:
            return None

        ## consider increasing k to more than 10. maybe try even top 30% or something
        results = self.store.similarity_search_with_score(query, k=min(10, self.store.index.ntotal))
        relevant_info = [
            f"Topic: {doc.metadata['topic']}\nDetails: {doc.page_content}"
            for doc, _ in results
        ]

        memorag_span_prompt = memorag_span_prompt_for_queries_creation
        memorag_sur_prompt = memorag_sur_prompt_for_queries_creation
        system_prompt = system_prompt_for_queries_creation

        text_spans = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": memorag_span_prompt.format(question=query, relevant_info = relevant_info)}
            ]
        ).choices[0].message.content
        surrogate_queries = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": memorag_sur_prompt.format(question=query, relevant_info = relevant_info)}
            ],
            temperature=0.5  # Allow for some creativity in generating clues
        ).choices[0].message.content

        retrieval_query = text_spans.split("\n") + surrogate_queries.split("\n")
        retrieval_query = [q for q in retrieval_query if len(q.split()) > 3]
        retrieval_query.append(query)

        return retrieval_query

    def save_store(self, path: str):
        if self.store:
            self.store.save_local(path)

    def load_store(self, path: str):
        self.store = FAISS.load_local(path, self.embeddings, allow_dangerous_deserialization=True)

Retrieval Function

For extracting relevant context for generation

In [ ]python · cell 25

python

def retrieve_context(retrieval_query: str, vectorstore, k: int = 3) -> List[str]:
    """Retrieve relevant context using the staging answer and the database vectorstore.
    Implements c = Γ(y, D | γ) from the paper"""

    results = vectorstore.similarity_search(retrieval_query, k=k)
    contexts = [doc.page_content for doc in results]

    return contexts

Generation Function

For getting the final answer using the original user's query + context obtained by the retrieval function

In [ ]python · cell 27

python

def generate_answer(query: str, contexts: List[str], temperature: float = 0) -> str:
    """Generate final answer y = Θ(q, c | θ)"""
    prompt = f"""Based ONLY on the provided context, answer the query.

Query: {query}

Context:
{' '.join(contexts)}

Provide a clear and concise answer based solely on the information in the context. If the context doesn't contain sufficient information to fully answer the query, clearly state that the necessary information is not available in the provided context rather than using any external knowledge.
"""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are an expert in generating answers from given text. Provide clear, concise answers."},
            {"role": "user", "content": prompt}
        ],
        temperature=temperature,
        max_tokens=500
    )

    return response.choices[0].message.content

Query Processing Function

for executing the whole pipeline, creating retrival queries from memory, using these queries to retrieve from the DB (vectorstore)

the relevant context which in turn is used for generating the final response to the query

In [ ]python · cell 29

python

def process_query(query: str, memory_store, vectorstore):
    print('*'*20)
    print("\nProcessing Query:", query)
    print("=" * 50)

    # y = Θ_mem(q, D | θ_mem)
    retrieval_queries = memory_store.create_retrieval_queries(query)
    print('*'*20)
    print(f"\n\nRetrieval Queries:\n{retrieval_queries}\n\n")

    # c = Γ(y, D | γ)
    all_contexts = []
    for retrieval_query in retrieval_queries:
        contexts = retrieve_context(retrieval_query, vectorstore, k=3)
        all_contexts.extend(contexts)

    unique_contexts = list(dict.fromkeys(all_contexts))  # Preserves order
    print('*'*20)
    print(f"Retrieved Context Example: {unique_contexts[0]}\n\n")

    # y = Θ(q, c | θ)
    final_answer = generate_answer(query, unique_contexts)
    print('*'*20)
    print(f"\nFinal Answer: {final_answer}")

    return unique_contexts, final_answer

Initialize Components

In [ ]python · cell 31

python

# Initialize vector store
path = "../data/Understanding_Climate_Change.pdf"
climate_vectorstore = encode_pdf(path, chunk_size=1000, chunk_overlap=200)

# # Initialize memory store
climate_memory_store = MemoryStore()

Memorize document chunks and save memory into a file

Note: this cell might run for a long time and take quite a lot of the tokens credit, so be careful. Once done, the memory is saved into a file, and then you can use it for as many queries as you like.

In [ ]python · cell 33

python

# loader = PyPDFLoader(path)
# documents = loader.load()
# document_text = '\n'.join([doc.page_content for doc in documents])
# climate_memory_store.memorize(document_text)
# climate_memory_store.save_store("../data/Understanding_Climate_Change_Memory_Store.faiss")

Usage Examples

Loading the saved memory

In [ ]python · cell 36

python

climate_memory_store.load_store("../data/Understanding_Climate_Change_Memory_Store.faiss")

Running some queries

In [ ]python · cell 38

python

query_1 = "What are the impacts of climate change on biodiversity?"
query_2 = "Please summarize the climate change article"
query_3 = "Describe the social and economic influence of climate change."

for query in [query_1, query_2, query_3]:
    process_query(query, climate_memory_store, climate_vectorstore)

Comparison between MemorRAG and Simple RAG.

Will be done for both short context and long context

Importing the evaluation functions

In [ ]python · cell 41

python

from evaluation.evalute_rag import *

Simple RAG Short Context

Using the climate_vectorstore created above

In [ ]python · cell 44

python

chunks_query_retriever = climate_vectorstore.as_retriever(search_kwargs={"k": 2})

The evaluate_rag function takes questions from the climate change q_a file, generates answers with Simple RAG and then compares them to the ground truth answers from the q_a file. It then evaluates the quality of Simple RAG responses using an LLM by 3 criteria:

Correctness - plain comparison between the final answers and how much they match
Faithfulness - checks if the generated answers indeed align with the retrieved context
Context Relevancy - checks if the retrieved context from the vectorstore is indeed relevant for the given query

In [ ]python · cell 46

python

evaluate_rag(chunks_query_retriever, q_a_file_name = "../data/q_a.json")

MemoRAG Short Context

An adjustment of the original evaluate_rag function to MemoRAG. Note that one of the parameters here is the memory_store. Since the part of generating answers with MemoRAG might be costly, it allows using pre-saved results if they exist. Another advantage of use_saved_results is the option to use results from original MemoRAG run made in Colab.

In [ ]python · cell 49

python

def evaluate_memo_rag(memory_store = None,vectorstore = None, num_questions: int = 5,
                      q_a_file_name = "../data/q_a.json",
                      use_saved_results=True,
                      use_original_memorag=False,
                      original_memorag_answers=None,
                      original_memorag_retrievals=None
                      )-> None:
    """
    Evaluate the MemoRAG system using predefined metrics.

    Args:
        memory_store: MemoryStore object containing the memory model.
        vectorstore: FAISS vector store containing the encoded document chunks.
        num_questions (int): Number of questions to evaluate (default: 5).
        q_a_file_name (str): Path to the JSON file containing questions and answers (default: "../data/q_a.json").
        use_saved_results (bool): Whether to load previously saved results or to save generated results to disk.
        use_original_memorag (bool): Whether to use paper's MemoRAG results or simplified MemoRAG implementation from this notebook.
        original_memorag_answers: Pre-generated answers to use in case use_original_memorag=True.
        original_memorag_retrievals: Pre-generated retrievals to use in case use_original_memorag=True.
    """
    q_a_file_name = q_a_file_name
    with open(q_a_file_name, "r", encoding="utf-8") as json_file:
        q_a = json.load(json_file)

    questions = [qa["question"] for qa in q_a][:num_questions]
    ground_truth_answers = [qa["answer"] for qa in q_a][:num_questions]

    if use_original_memorag:
        """Results of the original MemoRAG were created externally and loaded here"""
        generated_answers = original_memorag_answers
        retrieved_documents = original_memorag_retrievals
        print("Using original MemoRAG results")
    else:
        """When evaluating results of the MemoRAG implementation in this notebook
        there are 2 options - either using pre-saved results or generating new ones
        full_results_path is used either to load pre-saved results or for saving
        the results in case of generation"""
        q_a_file_base = os.path.splitext(q_a_file_name)[0]
        results_path_suffix = "saved_evaluation_results"
        full_results_path = f"{q_a_file_base}_{results_path_suffix}.json"

        if use_saved_results and os.path.exists(full_results_path):
            with open(full_results_path, "r", encoding="utf-8") as f:
                saved_data = json.load(f)
                generated_answers = saved_data["generated_answers"]
                retrieved_documents = saved_data["retrieved_documents"]
            print(f"Loaded previously saved results from {full_results_path}")
        else:
            if memory_store is None or vectorstore is None:
                raise ValueError("memory_store and vectorstore are required when not using saved results or original MemoRAG")

            # Generate answers and retrieve documents for each question
            generated_answers = []
            retrieved_documents = []
            for question in questions:
                contexts, result = process_query(question, memory_store, vectorstore)
                retrieved_documents.append(contexts)
                generated_answers.append(result)

            with open(full_results_path, "w", encoding="utf-8") as f:
                json.dump({
                    "generated_answers": generated_answers,
                    "retrieved_documents": retrieved_documents
                }, f, ensure_ascii=False, indent=2)
            print(f"Saved results to {full_results_path}")

    # Create test cases and evaluate
    test_cases = create_deep_eval_test_cases(questions, ground_truth_answers, generated_answers, retrieved_documents)
    evaluate(
        test_cases=test_cases,
        metrics=[correctness_metric, faithfulness_metric, relevance_metric],
        throttle_value = 5,
        run_async = False,
        ignore_errors = True
    )

Running evaluate_memo_rag on the same climate change q_a as above

In [ ]python · cell 51

python

evaluate_memo_rag(q_a_file_name = "../data/q_a.json", memory_store=climate_memory_store, vectorstore = climate_vectorstore, use_saved_results=False)

Long Context - the book "The Wealth of Nations" by Adam Smith

Taken from Project Gutenberg. Contains ~600 pages.

Initialize Components

In [ ]python · cell 54

python

# Initialize vector store
path = "../data/The_Wealth_of_Nations_Project_Gutenberg.pdf"
wealth_vectorstore = encode_pdf(path, chunk_size=1000, chunk_overlap=200)

# Initialize memory store
wealth_memory_store = MemoryStore()

Memorize document chunks and save memory into a file

In [ ]python · cell 56

python

# # Memorize document chunks and save memory into a file
# loader = PyPDFLoader(path)
# documents = loader.load()
# document_text = '\n'.join([doc.page_content for doc in documents])

# # Split document into smaller chunks
# text_splitter = RecursiveCharacterTextSplitter(
#     chunk_size=50000,
#     chunk_overlap=5000
# )
# chunks = text_splitter.split_text(document_text)
#
# # Memorize each chunk
# for chunk in chunks:
#     wealth_memory_store.memorize(chunk)
# wealth_memory_store.save_store("../data/The_Wealth_of_Nations_Project_Gutenberg_Memory_Store_Long_Context.faiss")

Loading the saved memory

In [ ]python · cell 58

python

wealth_memory_store.load_store("../data/The_Wealth_of_Nations_Project_Gutenberg_Memory_Store_Long_Context.faiss")

Simple RAG Long Context

Using the wealth_vectorstore created above

In [ ]python · cell 61

python

chunks_query_retriever = wealth_vectorstore.as_retriever(search_kwargs={"k": 2})

Using the evaluate_rag function on the q_a file for the long book

In [ ]python · cell 63

python

evaluate_rag(chunks_query_retriever, num_questions = 15, q_a_file_name = "../data/q_a_smith_short.json")

MemoRAG Long Context

Using the adjusted evaluate_memo_rag function on the q_a file for the long book

In [ ]python · cell 66

python

evaluate_memo_rag(num_questions=15, q_a_file_name = "../data/q_a_smith_short.json", memory_store= wealth_memory_store, vectorstore = wealth_vectorstore, use_saved_results=False)

Evaluate original MemoRAG Long Context

Original MemoRAG is rather heavy and had to be run on Colab with an A100 GPU. Its answers to the questions from the Smith q_a file are in the following cell

In [ ]python · cell 69

python

original_memorag_answers_wealth = ["Adam Smith",
"The invisible hand refers to the self-interested actions of individuals in a free market economy that unintentionally lead to economic outcomes.",
"Land, labor, and capital.",
"Labor determines the value of goods through the quantity of labor required to produce them.",
"Adam Smith argues that while individuals act in their self-interest, this pursuit often benefits society more effectively than if they acted solely for personal gain. He illustrates this through examples like trade, where individuals acting in their self-interest actually facilitate economic growth and prosperity for society as a whole.",
"Smith justifies taxation as necessary for the defense of the state, the maintenance of public order, and the provision of public services.",
"Supply and demand determine prices. When supply exceeds demand, prices tend to fall; when demand exceeds supply, prices tend to rise.",
"The division of labor encourages workers to focus on specific tasks, leading to innovation in tools and methods. This innovation enhances productivity and has long-term economic impacts by increasing the quantity of goods produced and the efficiency of production processes.",
"Smith viewed agriculture as the primary source of wealth, while manufacturing contributed less significantly.",
"Foreign trade enhances domestic wealth by allowing countries to specialize in producing goods where they have a comparative advantage. This specialization leads to increased efficiency and productivity. Smith's 'invisible hand' refers to the self-interested actions of individuals and businesses that, despite their self-interest, ultimately contribute to economic growth and efficiency.",
"Capital accumulation increases the quantity of productive labor by enabling more workers to be employed. This leads to a division of labor where each worker specializes in a specific task, enhancing efficiency and productivity.",
"Challenges include the need for security, the complexity of international transactions, and the potential for fraud and corruption.",
"Value is the amount of labor required to produce a good or service.",
"Smith believed that government should not interfere with the natural division of labor, allowing individuals to specialize in their areas of expertise. He argued that government's role is to protect property rights, ensure justice, and provide public works and institutions that facilitate economic growth.",
"Free trade benefits consumers by lowering prices, increasing product variety, and enhancing economic efficiency.",
"Adam Smith argued that monopolies, whether natural or artificial, are inherently detrimental to society and the economy. They stifle competition, lead to higher prices, and reduce overall economic efficiency. Smith believed that monopolies are a form of corruption and that the benefits they confer on a few at the expense of many are unjust. He advocated for free trade and competition as essential to economic prosperity and social justice"
]

Initialize components

In [ ]python · cell 71

python

csv_path = "../data/wealth_retrieved_passages_from_original_memorag.csv"
df = pd.read_csv(csv_path, dtype=str, keep_default_na=False)

# Convert to a 2D list
original_memorag_retrievals_wealth = df.values.tolist()

Using the adjusted evaluate_memo_rag function on the q_a file for the long book with original MemoRAG's answers

In [ ]python · cell 73

python

evaluate_memo_rag(num_questions=15, q_a_file_name = "../data/q_a_smith_short.json", memory_store= wealth_memory_store, vectorstore = wealth_vectorstore, use_saved_results=True, use_original_memorag=True, original_memorag_answers=original_memorag_answers_wealth, original_memorag_retrievals=original_memorag_retrievals_wealth)

In [ ]python · cell 74

python

Chapter 22

memorag

NotebookPython 374 cells

Overview

Motivation

Key Components

Memory: A compressed representation of the database created by a long-context model, designed to handle and summarize extensive inputs efficiently.
Retriever: A standard RAG retrieval model responsible for selecting relevant context from the knowledge base to support the generator.
Generator: A generative language model that produces responses by combining the query with the retrieved context, similar to standard RAG setups.

Method Details

1. Memory

The memory module serves as an auxiliary component to enhance the retriever’s ability to identify better matches between queries and relevant parts of the database. It takes the original query and the database as inputs and produces staging answers — intermediate outputs like clues, surrogate queries, or key points — which the retriever uses instead of the original query.
Long-term memory is constructed by running a long-context model, such as Qwen2-7B-Instruct or Mistral-7B-Instruct-v0.2, over the entire database. This process generates a compressed representation of the database through an attention mechanism.
The compressed representation is stored as key-value pairs (kv-cache), facilitating efficient and accurate retrieval.
Released memory models include memorag-qwen2-7b-inst and memorag-mistral-7b-inst, derived from Qwen2-7B-Instruct and Mistral-7B-Instruct-v0.2, respectively.

2. Retriever

The retriever is a standard retrieval model, adapted to take processed queries (created by the memory module as staging answers) instead of the original query.
It outputs the retrieved context, which serves as the basis for generating the final answer.

3. Generator

The generator produces the final response by combining the retriever’s output (retrieved context) with the original query.
MemoRAG ensures compatibility and consistency by using the memory module’s underlying model as the default generator.

Benefits of the Approach

Extended Scope of Queries: MemoRAG's preprocessing capabilities enable it to handle complex and long-context tasks that conventional RAG methods struggle with.
Improved Accuracy: By simplifying and adjusting queries before retrieval, MemoRAG enhances performance over standard RAG methods.
Flexibility: Adapts to diverse tasks, datasets, and retrieval scenarios.
Robustness: Improved performance remains consistent across various generators, datasets, and query types.
Efficiency: The use of key-value compression reduces computational overhead.

Conclusion

Implementation

Imports

In [ ]python · cell 5

python

import os
import pandas as pd
from dotenv import load_dotenv
from langchain_community.embeddings import OpenAIEmbeddings
from openai import OpenAI
from helper_functions import *

OpenAI Setup for querying GPT's API

In [ ]python · cell 7

python

load_dotenv()
client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

Memory Module Classes

Helper classes

Used for initial memorization

In [ ]python · cell 10

python

class KeyValuePair(BaseModel):
    """ The key-value pairs are the "holders" of memory
        The keys are topics from the relevant article and the values are details about that topic """
    topic: str
    details: str

class MemoryResponse(BaseModel):
    """ The obtained memory is a list of topic-details pairs """
    pairs: List[KeyValuePair]

MemoryStore

A system prompt for assigning the model with the role of an expert of memorization

In [ ]python · cell 13

python

memory_store_system_prompt = """
    You are an expert at extracting structured key-value pairs from text.
    "Identify key topics, entities, or questions and pair them with relevant, detailed information.
    "Ensure the output is well-structured, avoiding redundancy while preserving meaning.
    """

A prompt instructing the LLM about the process of memory creation

In [ ]python · cell 15

python

memory_store_kv_cache_prompt = """
    You are provided with a long article, chunk by chunk. Read each chunk carefully and extract key topics and their detailed information.
    For each key topic:
    1. Identify the main concept, entity, or potential question
    2. Provide the corresponding detailed information or answer

    Note: the aim is to mimic kv cache memory creation

    Now, the article begins:
    {document}

    The article ends here.

    RESPONSE FORMAT IS IMPORTANT. You must return a JSON object with the following structure:
    {{
        "pairs": [
            {{
                "topic": "topic1",
                "details": "details1"
            }},
            {{
                "topic": "topic2",
                "details": "details2"
            }}
        ]
    }}
    """

A system prompt for assigning the model with the role of an expert for creating queries for info retrieval

In [ ]python · cell 17

python

system_prompt_for_queries_creation = "You are an expert in information retrieval. Your task is to extract specific and relevant information from the provided context to help answer the given question."

A prompt guiding the LLM on how to find clues in the memory that can help answer the given question

In [ ]python · cell 19

python

memorag_span_prompt_for_queries_creation = """
    You are given a question related to an article. To answer it effectively, you need to use specific details from the article. You are not provided with the whole article. Instead, you are provided with specific and relevant information from it.
     Your task is to identify and extract one or more specific clue texts from the provided information that are relevant to the question.

    ### Question: {question}
    ### Information: {relevant_info}
    ### Instructions:
    1. You have a general understanding of the provided information. Your task is to generate one or more specific clues that will help in searching for supporting evidence within the article.
    2. The clues are in the form of text spans that will assist in answering the question.
    3. Only output the clues. If there are multiple clues, separate them with a newline.
    """

A prompt guiding the LLM on how to create surrogate (helper) queries from memory

In [ ]python · cell 21

python

memorag_sur_prompt_for_queries_creation = """
    You are given a question related to an article. To answer it effectively, you need to use specific details from the article.  You are not provided with the whole article. Instead, you are provided with specific and relevant information from it.
    Your task is to generate precise clue questions that can help locate the necessary information for answering the question.

    ### Question: {question}
    ### Information: {relevant_info}
    ### Instructions:
    1. You have a general understanding of the provided information. Your task is to generate one or more specific clues that will help in searching for supporting evidence within the article.
    2. The clues are in the form of precise surrogate questions that clarify the original question.
    3. Only output the clues. If there are multiple clues, separate them with a newline.
    """

The Memory Class - as mentioned above it can "memorize" the text by creating kv pairs, and later on creating adjusted queries that, in turn, will be transferred to the retrieval model

In [ ]python · cell 23

python

class MemoryStore:
    """The MemoryStore class is a realization of the Memory Module discussed in the paper.
    Its 'memorize' method is used to create a prompt cache mimicking a key-value compression of the original text (database).
    This cache can then be used by the 'create_retrieval_prompt' method for creating the processed query to be used later for retrieval"""

    def __init__(self):
        self.embeddings = OpenAIEmbeddings()
        self.store = None
        self.processed_count = 0
        self.json_parse_failures = 0
        self._last_parse_used_fallback = False


    def memorize(self, document: str):
        """Process document into key-value pairs and store them"""
        system_prompt = memory_store_system_prompt
        kv_cache_prompt = memory_store_kv_cache_prompt
        print(f"Processing chunk {self.processed_count + 1}...")

        response = client.chat.completions.create(
            model="gpt-4o-mini", # 4o or o1 should be considered
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": kv_cache_prompt.format(document=document)}
            ],
            response_format={"type": "json_object"},
            seed=42  # For reproducibility
        )

        pairs = self._parse_into_pairs(response.choices[0].message.content)
        if hasattr(self, 'json_parse_failures') and self._last_parse_used_fallback:
            self.json_parse_failures += 1
            if self.json_parse_failures > 0 and self.json_parse_failures % 5 == 0:
               print(f"Warning: JSON parsing has failed {self.json_parse_failures} times")

        print('*'*20)
        print(f'first extracted pairs: {pairs[0:3]}\n\n')

        # Batch process pairs
        texts = []
        metadatas = []

        for topic, details in pairs:
            if not topic or not details:  # Skip empty pairs
                continue

            combined_text = f"Topic: {topic}\nDetails: {details}"
            texts.append(combined_text)
            metadatas.append({"topic": topic})
        print('*'*20)
        print(f'num of extracted texts: {len(texts)}\n\n')
        if self.store is None:
            self.store = FAISS.from_texts(texts, self.embeddings, metadatas=metadatas)
        else:
            existing_docs = self.store.similarity_search("")
            existing_topics = {doc.metadata.get("topic") for doc in existing_docs}

            # Filter out duplicates
            new_texts = []
            new_metadatas = []
            for text, metadata in zip(texts, metadatas):
                if metadata["topic"] not in existing_topics:
                    new_texts.append(text)
                    new_metadatas.append(metadata)

            if new_texts:
                self.store.add_texts(new_texts, metadatas=new_metadatas)

        self.processed_count += 1
        print(f"Processed {self.processed_count} chunks so far\n\n")

    def _parse_into_pairs(self, content: str):
        """Used when LLM returned a valid JSON"""
        self._last_parse_used_fallback = False
        try:
            # Parse JSON and validate with Pydantic
            data = json.loads(content)
            validated_data = MemoryResponse(**data)
            return [(pair.topic, pair.details) for pair in validated_data.pairs]
        except (json.JSONDecodeError, ValueError) as e:
            print(f"Error parsing response: {e}")
            # Fall back to text parsing if JSON fails
            self._last_parse_used_fallback = True
            return self._parse_text_into_pairs(content)

    def _parse_text_into_pairs(self, text: str):
        """Original text parsing method as fallback
        Used when LLM did not return a valid JSON"""
        pairs = []
        lines = text.split('\n')
        current_topic = None
        current_details = []

        for line in lines:
            if line.startswith('Topic:'):
                if current_topic:  # Save previous pair
                    pairs.append((current_topic, ' '.join(current_details)))
                current_topic = line[6:].strip()
                current_details = []
            elif line.startswith('Details:'):
                current_details.append(line[8:].strip())
            elif current_details:  # Continue adding to details if already started
                current_details.append(line.strip())

        # Add the last pair
        if current_topic:
            pairs.append((current_topic, ' '.join(current_details)))

        return pairs

    def create_retrieval_queries(self, query: str) -> str | None:
        """Generate staging answers y = Θ_mem(q, D | θ_mem)
        This should provide rough clues/outline to guide context retrieval"""

        if not self.store:
            return None

        ## consider increasing k to more than 10. maybe try even top 30% or something
        results = self.store.similarity_search_with_score(query, k=min(10, self.store.index.ntotal))
        relevant_info = [
            f"Topic: {doc.metadata['topic']}\nDetails: {doc.page_content}"
            for doc, _ in results
        ]

        memorag_span_prompt = memorag_span_prompt_for_queries_creation
        memorag_sur_prompt = memorag_sur_prompt_for_queries_creation
        system_prompt = system_prompt_for_queries_creation

        text_spans = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": memorag_span_prompt.format(question=query, relevant_info = relevant_info)}
            ]
        ).choices[0].message.content
        surrogate_queries = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": memorag_sur_prompt.format(question=query, relevant_info = relevant_info)}
            ],
            temperature=0.5  # Allow for some creativity in generating clues
        ).choices[0].message.content

        retrieval_query = text_spans.split("\n") + surrogate_queries.split("\n")
        retrieval_query = [q for q in retrieval_query if len(q.split()) > 3]
        retrieval_query.append(query)

        return retrieval_query

    def save_store(self, path: str):
        if self.store:
            self.store.save_local(path)

    def load_store(self, path: str):
        self.store = FAISS.load_local(path, self.embeddings, allow_dangerous_deserialization=True)

Retrieval Function

For extracting relevant context for generation

In [ ]python · cell 25

python

def retrieve_context(retrieval_query: str, vectorstore, k: int = 3) -> List[str]:
    """Retrieve relevant context using the staging answer and the database vectorstore.
    Implements c = Γ(y, D | γ) from the paper"""

    results = vectorstore.similarity_search(retrieval_query, k=k)
    contexts = [doc.page_content for doc in results]

    return contexts

Generation Function

For getting the final answer using the original user's query + context obtained by the retrieval function

In [ ]python · cell 27

python

def generate_answer(query: str, contexts: List[str], temperature: float = 0) -> str:
    """Generate final answer y = Θ(q, c | θ)"""
    prompt = f"""Based ONLY on the provided context, answer the query.

Query: {query}

Context:
{' '.join(contexts)}

Provide a clear and concise answer based solely on the information in the context. If the context doesn't contain sufficient information to fully answer the query, clearly state that the necessary information is not available in the provided context rather than using any external knowledge.
"""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are an expert in generating answers from given text. Provide clear, concise answers."},
            {"role": "user", "content": prompt}
        ],
        temperature=temperature,
        max_tokens=500
    )

    return response.choices[0].message.content

Query Processing Function

for executing the whole pipeline, creating retrival queries from memory, using these queries to retrieve from the DB (vectorstore)

the relevant context which in turn is used for generating the final response to the query

In [ ]python · cell 29

python

def process_query(query: str, memory_store, vectorstore):
    print('*'*20)
    print("\nProcessing Query:", query)
    print("=" * 50)

    # y = Θ_mem(q, D | θ_mem)
    retrieval_queries = memory_store.create_retrieval_queries(query)
    print('*'*20)
    print(f"\n\nRetrieval Queries:\n{retrieval_queries}\n\n")

    # c = Γ(y, D | γ)
    all_contexts = []
    for retrieval_query in retrieval_queries:
        contexts = retrieve_context(retrieval_query, vectorstore, k=3)
        all_contexts.extend(contexts)

    unique_contexts = list(dict.fromkeys(all_contexts))  # Preserves order
    print('*'*20)
    print(f"Retrieved Context Example: {unique_contexts[0]}\n\n")

    # y = Θ(q, c | θ)
    final_answer = generate_answer(query, unique_contexts)
    print('*'*20)
    print(f"\nFinal Answer: {final_answer}")

    return unique_contexts, final_answer

Initialize Components

In [ ]python · cell 31

python

# Initialize vector store
path = "../data/Understanding_Climate_Change.pdf"
climate_vectorstore = encode_pdf(path, chunk_size=1000, chunk_overlap=200)

# # Initialize memory store
climate_memory_store = MemoryStore()

Memorize document chunks and save memory into a file

In [ ]python · cell 33

python

# loader = PyPDFLoader(path)
# documents = loader.load()
# document_text = '\n'.join([doc.page_content for doc in documents])
# climate_memory_store.memorize(document_text)
# climate_memory_store.save_store("../data/Understanding_Climate_Change_Memory_Store.faiss")

Usage Examples

Loading the saved memory

In [ ]python · cell 36

python

climate_memory_store.load_store("../data/Understanding_Climate_Change_Memory_Store.faiss")

Running some queries

In [ ]python · cell 38

python

query_1 = "What are the impacts of climate change on biodiversity?"
query_2 = "Please summarize the climate change article"
query_3 = "Describe the social and economic influence of climate change."

for query in [query_1, query_2, query_3]:
    process_query(query, climate_memory_store, climate_vectorstore)

Comparison between MemorRAG and Simple RAG.

Will be done for both short context and long context

Importing the evaluation functions

In [ ]python · cell 41

python

from evaluation.evalute_rag import *

Simple RAG Short Context

Using the climate_vectorstore created above

In [ ]python · cell 44

python

chunks_query_retriever = climate_vectorstore.as_retriever(search_kwargs={"k": 2})

Correctness - plain comparison between the final answers and how much they match
Faithfulness - checks if the generated answers indeed align with the retrieved context
Context Relevancy - checks if the retrieved context from the vectorstore is indeed relevant for the given query

In [ ]python · cell 46

python

evaluate_rag(chunks_query_retriever, q_a_file_name = "../data/q_a.json")

MemoRAG Short Context

In [ ]python · cell 49

python

def evaluate_memo_rag(memory_store = None,vectorstore = None, num_questions: int = 5,
                      q_a_file_name = "../data/q_a.json",
                      use_saved_results=True,
                      use_original_memorag=False,
                      original_memorag_answers=None,
                      original_memorag_retrievals=None
                      )-> None:
    """
    Evaluate the MemoRAG system using predefined metrics.

    Args:
        memory_store: MemoryStore object containing the memory model.
        vectorstore: FAISS vector store containing the encoded document chunks.
        num_questions (int): Number of questions to evaluate (default: 5).
        q_a_file_name (str): Path to the JSON file containing questions and answers (default: "../data/q_a.json").
        use_saved_results (bool): Whether to load previously saved results or to save generated results to disk.
        use_original_memorag (bool): Whether to use paper's MemoRAG results or simplified MemoRAG implementation from this notebook.
        original_memorag_answers: Pre-generated answers to use in case use_original_memorag=True.
        original_memorag_retrievals: Pre-generated retrievals to use in case use_original_memorag=True.
    """
    q_a_file_name = q_a_file_name
    with open(q_a_file_name, "r", encoding="utf-8") as json_file:
        q_a = json.load(json_file)

    questions = [qa["question"] for qa in q_a][:num_questions]
    ground_truth_answers = [qa["answer"] for qa in q_a][:num_questions]

    if use_original_memorag:
        """Results of the original MemoRAG were created externally and loaded here"""
        generated_answers = original_memorag_answers
        retrieved_documents = original_memorag_retrievals
        print("Using original MemoRAG results")
    else:
        """When evaluating results of the MemoRAG implementation in this notebook
        there are 2 options - either using pre-saved results or generating new ones
        full_results_path is used either to load pre-saved results or for saving
        the results in case of generation"""
        q_a_file_base = os.path.splitext(q_a_file_name)[0]
        results_path_suffix = "saved_evaluation_results"
        full_results_path = f"{q_a_file_base}_{results_path_suffix}.json"

        if use_saved_results and os.path.exists(full_results_path):
            with open(full_results_path, "r", encoding="utf-8") as f:
                saved_data = json.load(f)
                generated_answers = saved_data["generated_answers"]
                retrieved_documents = saved_data["retrieved_documents"]
            print(f"Loaded previously saved results from {full_results_path}")
        else:
            if memory_store is None or vectorstore is None:
                raise ValueError("memory_store and vectorstore are required when not using saved results or original MemoRAG")

            # Generate answers and retrieve documents for each question
            generated_answers = []
            retrieved_documents = []
            for question in questions:
                contexts, result = process_query(question, memory_store, vectorstore)
                retrieved_documents.append(contexts)
                generated_answers.append(result)

            with open(full_results_path, "w", encoding="utf-8") as f:
                json.dump({
                    "generated_answers": generated_answers,
                    "retrieved_documents": retrieved_documents
                }, f, ensure_ascii=False, indent=2)
            print(f"Saved results to {full_results_path}")

    # Create test cases and evaluate
    test_cases = create_deep_eval_test_cases(questions, ground_truth_answers, generated_answers, retrieved_documents)
    evaluate(
        test_cases=test_cases,
        metrics=[correctness_metric, faithfulness_metric, relevance_metric],
        throttle_value = 5,
        run_async = False,
        ignore_errors = True
    )

Running evaluate_memo_rag on the same climate change q_a as above

In [ ]python · cell 51

python

evaluate_memo_rag(q_a_file_name = "../data/q_a.json", memory_store=climate_memory_store, vectorstore = climate_vectorstore, use_saved_results=False)

Long Context - the book "The Wealth of Nations" by Adam Smith

Taken from Project Gutenberg. Contains ~600 pages.

Initialize Components

In [ ]python · cell 54

python

# Initialize vector store
path = "../data/The_Wealth_of_Nations_Project_Gutenberg.pdf"
wealth_vectorstore = encode_pdf(path, chunk_size=1000, chunk_overlap=200)

# Initialize memory store
wealth_memory_store = MemoryStore()

Memorize document chunks and save memory into a file

In [ ]python · cell 56

python

# # Memorize document chunks and save memory into a file
# loader = PyPDFLoader(path)
# documents = loader.load()
# document_text = '\n'.join([doc.page_content for doc in documents])

# # Split document into smaller chunks
# text_splitter = RecursiveCharacterTextSplitter(
#     chunk_size=50000,
#     chunk_overlap=5000
# )
# chunks = text_splitter.split_text(document_text)
#
# # Memorize each chunk
# for chunk in chunks:
#     wealth_memory_store.memorize(chunk)
# wealth_memory_store.save_store("../data/The_Wealth_of_Nations_Project_Gutenberg_Memory_Store_Long_Context.faiss")

Loading the saved memory

In [ ]python · cell 58

python

wealth_memory_store.load_store("../data/The_Wealth_of_Nations_Project_Gutenberg_Memory_Store_Long_Context.faiss")

Simple RAG Long Context

Using the wealth_vectorstore created above

In [ ]python · cell 61

python

chunks_query_retriever = wealth_vectorstore.as_retriever(search_kwargs={"k": 2})

Using the evaluate_rag function on the q_a file for the long book

In [ ]python · cell 63

python

evaluate_rag(chunks_query_retriever, num_questions = 15, q_a_file_name = "../data/q_a_smith_short.json")

MemoRAG Long Context

Using the adjusted evaluate_memo_rag function on the q_a file for the long book

In [ ]python · cell 66

python

evaluate_memo_rag(num_questions=15, q_a_file_name = "../data/q_a_smith_short.json", memory_store= wealth_memory_store, vectorstore = wealth_vectorstore, use_saved_results=False)

Evaluate original MemoRAG Long Context

Original MemoRAG is rather heavy and had to be run on Colab with an A100 GPU. Its answers to the questions from the Smith q_a file are in the following cell

In [ ]python · cell 69

python

original_memorag_answers_wealth = ["Adam Smith",
"The invisible hand refers to the self-interested actions of individuals in a free market economy that unintentionally lead to economic outcomes.",
"Land, labor, and capital.",
"Labor determines the value of goods through the quantity of labor required to produce them.",
"Adam Smith argues that while individuals act in their self-interest, this pursuit often benefits society more effectively than if they acted solely for personal gain. He illustrates this through examples like trade, where individuals acting in their self-interest actually facilitate economic growth and prosperity for society as a whole.",
"Smith justifies taxation as necessary for the defense of the state, the maintenance of public order, and the provision of public services.",
"Supply and demand determine prices. When supply exceeds demand, prices tend to fall; when demand exceeds supply, prices tend to rise.",
"The division of labor encourages workers to focus on specific tasks, leading to innovation in tools and methods. This innovation enhances productivity and has long-term economic impacts by increasing the quantity of goods produced and the efficiency of production processes.",
"Smith viewed agriculture as the primary source of wealth, while manufacturing contributed less significantly.",
"Foreign trade enhances domestic wealth by allowing countries to specialize in producing goods where they have a comparative advantage. This specialization leads to increased efficiency and productivity. Smith's 'invisible hand' refers to the self-interested actions of individuals and businesses that, despite their self-interest, ultimately contribute to economic growth and efficiency.",
"Capital accumulation increases the quantity of productive labor by enabling more workers to be employed. This leads to a division of labor where each worker specializes in a specific task, enhancing efficiency and productivity.",
"Challenges include the need for security, the complexity of international transactions, and the potential for fraud and corruption.",
"Value is the amount of labor required to produce a good or service.",
"Smith believed that government should not interfere with the natural division of labor, allowing individuals to specialize in their areas of expertise. He argued that government's role is to protect property rights, ensure justice, and provide public works and institutions that facilitate economic growth.",
"Free trade benefits consumers by lowering prices, increasing product variety, and enhancing economic efficiency.",
"Adam Smith argued that monopolies, whether natural or artificial, are inherently detrimental to society and the economy. They stifle competition, lead to higher prices, and reduce overall economic efficiency. Smith believed that monopolies are a form of corruption and that the benefits they confer on a few at the expense of many are unjust. He advocated for free trade and competition as essential to economic prosperity and social justice"
]

Initialize components

In [ ]python · cell 71

python

csv_path = "../data/wealth_retrieved_passages_from_original_memorag.csv"
df = pd.read_csv(csv_path, dtype=str, keep_default_na=False)

# Convert to a 2D list
original_memorag_retrievals_wealth = df.values.tolist()

Using the adjusted evaluate_memo_rag function on the q_a file for the long book with original MemoRAG's answers

In [ ]python · cell 73

python

evaluate_memo_rag(num_questions=15, q_a_file_name = "../data/q_a_smith_short.json", memory_store= wealth_memory_store, vectorstore = wealth_vectorstore, use_saved_results=True, use_original_memorag=True, original_memorag_answers=original_memorag_answers_wealth, original_memorag_retrievals=original_memorag_retrievals_wealth)

In [ ]python · cell 74

python