Chapter 44
4.句子滑窗检索 Sentence window retrieval
Notebookd2l97 cells
第四章 句子滑窗检索
In [1]python · cell 2
python
import warnings
warnings.filterwarnings('ignore')In [2]python · cell 3
python
import utils
import os
import openai
openai.api_key = utils.get_openai_api_key()Output
✅ In Answer Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` . ✅ In Answer Relevance, input response will be set to __record__.main_output or `Select.RecordOutput` . ✅ In Context Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` . ✅ In Context Relevance, input response will be set to __record__.app.query.rets.source_nodes[:].node.text . ✅ In Groundedness, input source will be set to __record__.app.query.rets.source_nodes[:].node.text . ✅ In Groundedness, input statement will be set to __record__.main_output or `Select.RecordOutput` .
读取数据库
In [3]python · cell 5
python
from llama_index import SimpleDirectoryReader
documents = SimpleDirectoryReader(
input_files=["data/人工智能.pdf"]
).load_data()In [4]python · cell 6
python
print(type(documents), "\n")
print(len(documents), "\n")
print(type(documents[0]))
print(documents[0])Output
<class 'list'> 7 <class 'llama_index.schema.Document'> Doc ID: b03a0e50-2e8a-49bf-82f0-4a3909364809 Text: 2/2/24, 2:43 PM ⼈⼯智能 - 维基百科，⾃由的百科全书 https://zh.wikipedia.org/wiki/ ⼈⼯智能 2/13“⼈⼯智能”的各地常⽤名称 中国⼤陆⼈⼯智能 台湾⼈⼯智慧 港澳⼈⼯智能 新⻢⼈⼯智能、⼈⼯智慧 ⽇韩⼈⼯知能 越南智慧⼈造 [展开] [展开] [展开] [展开] [展开] [展开]⼈⼯智能系列内容 主要⽬标 实现⽅式 ⼈⼯智能哲学 历史 技术 术语⼈⼯智能（英语：artificial intelligence ，缩写为 AI）亦称机器智能，指由⼈制造出来的机器所表现出来的智能。通常⼈⼯ 智能是指⽤普通计算机程序来呈现⼈类智能的技术。该词也指出研究这样的智能系统是否能够实现，以及如何实现。同 时，通过 医学 、神经科学 、机器⼈学 及...
In [5]python · cell 7
python
from llama_index import SimpleDirectoryReader
documents_en = SimpleDirectoryReader(
input_files=["data/eBook-How-to-Build-a-Career-in-AI.pdf"]
).load_data()In [6]python · cell 8
python
print(type(documents_en), "\n")
print(len(documents_en), "\n")
print(type(documents_en[0]))
print(documents_en[0])Output
<class 'list'> 41 <class 'llama_index.schema.Document'> Doc ID: 5ab262c9-a207-4f2d-9513-fc4d5c350cf5 Text: PAGE 1Founder, DeepLearning.AICollected Insights from Andrew Ng How to Build Your Career in AIA Simple Guide
这里通过将 documents 中各个文档的文本连接成一个字符串，然后创建了一个 Document 实例，该实例代表了整个文档集合。
In [7]python · cell 10
python
from llama_index import Document
document = Document(text="\n\n".join([doc.text for doc in documents]))
document_en = Document(text="\n\n".join([doc.text for doc in documents_en]))In [8]python · cell 11
python
# 将中文标点符号替换成英文标点符号，方便后续处理
# 如果是英文文档，可以跳过这一步
# 不处理的话，会导致无法正确切分中文句子，会影响后续sentence_window的大小，导致输入长度大于gpt-3.5-turbo的最大限制
document.text=document.text.replace('。','. ')
document.text=document.text.replace('！','! ')
document.text=document.text.replace('？','? ')一、句子滑窗检索设置
创建了一个名为 node_parser 的解析器对象，指定了窗口大小为3，原始文本元数据键被设置为original_text。这样创建的解析器可以用于从文本中提取节点
In [9]python · cell 14
python
from llama_index.node_parser import SentenceWindowNodeParser
# create the sentence window node parser w/ default settings
node_parser = SentenceWindowNodeParser.from_defaults(
window_size=3,
window_metadata_key="window",
original_text_metadata_key="original_text",
)定义一个中文文本字符串
使用 node_parser 的 get_nodes_from_documents 方法从提供的文本中提取节点。
In [10]python · cell 16
python
text = "你好. 你怎么样? 我很好! "
nodes = node_parser.get_nodes_from_documents([Document(text=text)])In [11]python · cell 17
python
text_en = "hello. how are you? I am fine! "
nodes_en = node_parser.get_nodes_from_documents([Document(text=text_en)])每个单独的词
In [12]python · cell 19
python
print([x.text for x in nodes])
print([x.text for x in nodes_en])Output
['你好. ', '你怎么样? ', '我很好! '] ['hello. ', 'how are you? ', 'I am fine! ']
原整句
In [13]python · cell 21
python
print(nodes[1].metadata["window"])
print(nodes_en[1].metadata["window"])Output
你好. 你怎么样? 我很好! hello. how are you? I am fine!
In [14]python · cell 22
python
text = "你好. 吧台. 猫狗. 老鼠"
text_en2 = 'hello. bar. cat. dog. mouse.'
nodes = node_parser.get_nodes_from_documents([Document(text=text)])
nodes_en2 = node_parser.get_nodes_from_documents([Document(text=text_en2)])In [15]python · cell 23
python
print([x.text for x in nodes])
print([x.text for x in nodes_en])Output
['你好. ', '吧台. ', '猫狗. ', '老鼠'] ['hello. ', 'how are you? ', 'I am fine! ']
In [16]python · cell 24
python
print(nodes[0].metadata["window"])
print(nodes_en2[0].metadata["window"])Output
你好. 吧台. 猫狗. hello. bar. cat.
2.1 创建索引
使用 OpenAI 的 GPT-3.5-turbo 模型创建了一个语言模型的实例，设置了温度参数为0.1。
In [17]python · cell 27
python
from llama_index.llms import OpenAI
llm = OpenAI(model="gpt-3.5-turbo", temperature=0.1)使用 ServiceContext.from_defaults 方法创建了一个 ServiceContext 对象，该对象包含了用于索引构建的服务相关的上下文信息，包括语言模型、嵌入模型以及节点解析器。
In [18]python · cell 29
python
from llama_index import ServiceContext
sentence_context = ServiceContext.from_defaults(
llm=llm,
embed_model="local:BAAI/bge-small-zh-v1.5",
node_parser=node_parser,
)
sentence_context_en = ServiceContext.from_defaults(
llm=llm,
embed_model="local:BAAI/bge-small-en-v1.5",
node_parser=node_parser,
)使用 VectorStoreIndex.from_documents 方法创建了一个 VectorStoreIndex 对象，该对象用于存储和检索与文档相关的向量信息。
In [19]python · cell 31
python
from llama_index import VectorStoreIndex
sentence_index = VectorStoreIndex.from_documents(
[document], service_context=sentence_context
)
from llama_index import VectorStoreIndex
sentence_index_en = VectorStoreIndex.from_documents(
[document_en], service_context=sentence_context_en
)将创建的索引持久化到指定目录（"./sentence_index"）。这样做可以在之后的运行中重新加载索引，而不必重新构建。
In [20]python · cell 33
python
sentence_index.storage_context.persist(persist_dir="./sentence_index")
sentence_index_en.storage_context.persist(persist_dir="./sentence_index_en")检查索引文件是否存在，如果不存在则重新构建,如果存在，它将使用 load_index_from_storage 方法从已有的索引文件中加载索引，而不是重新构建。
In [21]python · cell 35
python
# This block of code is optional to check
# if an index file exist, then it will load it
# if not, it will rebuild it
import os
from llama_index import VectorStoreIndex, StorageContext, load_index_from_storage
from llama_index import load_index_from_storage
if not os.path.exists("./sentence_index"):
sentence_index = VectorStoreIndex.from_documents(
[document], service_context=sentence_context
)
sentence_index.storage_context.persist(persist_dir="./sentence_index")
else:
sentence_index = load_index_from_storage(
StorageContext.from_defaults(persist_dir="./sentence_index"),
service_context=sentence_context
)
if not os.path.exists("./sentence_index_en"):
sentence_index_en = VectorStoreIndex.from_documents(
[document_en], service_context=sentence_context_en
)
sentence_index_en.storage_context.persist(persist_dir="./sentence_index_en")
else:
sentence_index_en = load_index_from_storage(
StorageContext.from_defaults(persist_dir="./sentence_index_en"),
service_context=sentence_context_en
)2.2 创建后处理
使用 MetadataReplacementPostProcessor 类创建了一个后处理器实例，设置了目标元数据键为 window。该后处理器的作用是替换目标元数据键的内容。
In [22]python · cell 38
python
from llama_index.indices.postprocessor import MetadataReplacementPostProcessor
postproc = MetadataReplacementPostProcessor(
target_metadata_key="window"
)使用 NodeWithScore 类，将原始节点列表中的每个节点与一个分数关联，形成带分数的节点列表。
使用 deepcopy 函数创建了原始节点列表的深度拷贝，以便后续比较。
In [23]python · cell 40
python
from llama_index.schema import NodeWithScore
from copy import deepcopy
scored_nodes = [NodeWithScore(node=x, score=1.0) for x in nodes]
nodes_old = [deepcopy(n) for n in nodes]
scored_nodes_en = [NodeWithScore(node=x, score=1.0) for x in nodes_en2]
nodes_old_en = [deepcopy(n) for n in nodes_en2]In [24]python · cell 41
python
print(nodes_old[1].text)
print(nodes_old_en[1].text)Output
吧台. bar.
使用后处理器的 postprocess_nodes 方法，替换了带分数的节点列表中目标元数据键的内容。
In [25]python · cell 43
python
replaced_nodes = postproc.postprocess_nodes(scored_nodes)
replaced_nodes_en = postproc.postprocess_nodes(scored_nodes_en)In [26]python · cell 44
python
print(replaced_nodes[1].text)
print(replaced_nodes_en[1].text)Output
你好. 吧台. 猫狗. 老鼠 hello. bar. cat. dog.
2.3 增设重新排序块
使用 SentenceTransformerRerank 类创建了一个后处理器实例，设置了参数 top_n 为 2，以及使用的模型为 "BAAI/bge-reranker-base"。
In [27]python · cell 47
python
from llama_index.indices.postprocessor import SentenceTransformerRerank
# BAAI/bge-reranker-base
# link: https://huggingface.co/BAAI/bge-reranker-base
rerank = SentenceTransformerRerank(
top_n=2, model="BAAI/bge-reranker-base"
)创建了一个包含查询文本的 QueryBundle 对象，该查询文本为 "我想要只狗."。
创建了一个包含两个带分数的节点的列表，这些节点分别表示包含 "这是只猫" 和 "这是只狗" 文本的文本节点，分数分别为 0.6 和 0.4。
In [28]python · cell 49
python
from llama_index import QueryBundle
from llama_index.schema import TextNode, NodeWithScore
query = QueryBundle("我想要只狗.")
scored_nodes = [
NodeWithScore(node=TextNode(text="这是只猫"), score=0.6),
NodeWithScore(node=TextNode(text="这是只狗"), score=0.4),
]In [29]python · cell 50
python
from llama_index import QueryBundle
from llama_index.schema import TextNode, NodeWithScore
query_en = QueryBundle("I want a dog.")
scored_nodes_en = [
NodeWithScore(node=TextNode(text="This is a cat"), score=0.6),
NodeWithScore(node=TextNode(text="This is a dog"), score=0.4),
]使用 SentenceTransformerRerank 类的 postprocess_nodes 方法，对带分数的节点列表进行重新排名，考虑到查询文本。重新排名的节点将基于预训练的句子转换模型。
In [30]python · cell 52
python
reranked_nodes = rerank.postprocess_nodes(
scored_nodes, query_bundle=query
)
reranked_nodes_en = rerank.postprocess_nodes(
scored_nodes_en, query_bundle=query_en
)输出了重新排名后的节点列表中的文本和分数。这里展示了句子转换模型对节点重新排名的效果。
In [31]python · cell 54
python
print([(x.text, x.score) for x in reranked_nodes])
print([(x.text, x.score) for x in reranked_nodes_en])Output
[('这是只狗', 0.9660425), ('这是只猫', 0.06396222)]
[('This is a dog', 0.9182736), ('This is a cat', 0.0014040753)]
2.4 运行索引引擎
使用 as_query_engine 方法将 sentence_index 转换为查询引擎对象 sentence_window_engine。
在这里，设置了相似性（similarity）的 top k 为 6，并传入了 node_postprocessors 参数，其中包含了之前创建的 postproc 和 rerank 后处理器。
In [32]python · cell 57
python
sentence_window_engine = sentence_index.as_query_engine(
similarity_top_k=6, node_postprocessors=[postproc, rerank]
)
sentence_window_engine_en = sentence_index_en.as_query_engine(
similarity_top_k=6, node_postprocessors=[postproc, rerank]
)使用查询引擎的 query 方法执行了一个查询，查询的内容是 "在人工智能领域建功立业的关键是什么?"。查询引擎将使用之前设置的后处理器进行节点后处理。
In [33]python · cell 59
python
window_response = sentence_window_engine.query(
"在人工智能领域建功立业的关键是什么?"
)In [34]python · cell 60
python
window_response_en = sentence_window_engine_en.query(
"What are the keys to building a career in AI?"
)使用 LLAMA 框架提供的 display_response 函数展示了查询的响应结果。这通常包括与查询匹配的一组节点，以及它们的文本、分数等信息。
这种方式可以在Notebook环境中更好地可视化和理解查询的结果。
In [35]python · cell 62
python
from llama_index.response.notebook_utils import display_response
display_response(window_response)Output
<IPython.core.display.Markdown object>
In [36]python · cell 63
python
from llama_index.response.notebook_utils import display_response
display_response(window_response_en)Output
<IPython.core.display.Markdown object>
二、合并上述操作
documents: 要构建索引的文档列表。
llm: OpenAI 语言模型实例。
embed_model: 嵌入模型的名称或路径。
sentence_window_size: 句子窗口的大小。
save_dir: 持久化索引的目录。
创建一个句子窗口的节点解析器（node_parser）。
创建一个包含语言模型和节点解析器等上下文信息的 ServiceContext。
如果指定的目录中不存在索引，则创建一个基于提供的文档的 VectorStoreIndex 并将其持久化到指定目录。
如果目录中已存在索引文件，则从文件中加载索引。
返回构建的句子窗口索引。
In [37]python · cell 66
python
import os
from llama_index import ServiceContext, VectorStoreIndex, StorageContext
from llama_index.node_parser import SentenceWindowNodeParser
from llama_index.indices.postprocessor import MetadataReplacementPostProcessor
from llama_index.indices.postprocessor import SentenceTransformerRerank
from llama_index import load_index_from_storage
def build_sentence_window_index(
documents,
llm,
embed_model="local:BAAI/bge-small-zh-v1.5",
sentence_window_size=3,
save_dir="sentence_index",
):
# create the sentence window node parser w/ default settings
node_parser = SentenceWindowNodeParser.from_defaults(
window_size=sentence_window_size,
window_metadata_key="window",
original_text_metadata_key="original_text",
)
sentence_context = ServiceContext.from_defaults(
llm=llm,
embed_model=embed_model,
node_parser=node_parser,
)
if not os.path.exists(save_dir):
sentence_index = VectorStoreIndex.from_documents(
documents, service_context=sentence_context
)
sentence_index.storage_context.persist(persist_dir=save_dir)
else:
sentence_index = load_index_from_storage(
StorageContext.from_defaults(persist_dir=save_dir),
service_context=sentence_context,
)
return sentence_index
def build_sentence_window_index_en(
documents_en,
llm,
embed_model="local:BAAI/bge-small-en-v1.5",
sentence_window_size=3,
save_dir="sentence_index_en",
):
# create the sentence window node parser w/ default settings
node_parser = SentenceWindowNodeParser.from_defaults(
window_size=sentence_window_size,
window_metadata_key="window",
original_text_metadata_key="original_text",
)
sentence_context = ServiceContext.from_defaults(
llm=llm,
embed_model=embed_model,
node_parser=node_parser,
)
if not os.path.exists(save_dir):
sentence_index_en = VectorStoreIndex.from_documents(
documents_en, service_context=sentence_context
)
sentence_index_en.storage_context.persist(persist_dir=save_dir)
else:
sentence_index_en = load_index_from_storage(
StorageContext.from_defaults(persist_dir=save_dir),
service_context=sentence_context,
)
return sentence_index_ensentence_index: 已构建的句子窗口索引。
similarity_top_k: 相似性查询的 top k。
rerank_top_n: 重新排名的 top n。
定义了两个后处理器：postproc 用于替换元数据键，rerank 用于使用句子转换模型重新排名节点。
创建一个查询引擎 sentence_window_engine，将句子窗口索引转换为查询引擎，并使用定义的后处理器。
返回构建的查询引擎。
In [38]python · cell 68
python
def get_sentence_window_query_engine(
sentence_index, similarity_top_k=6, rerank_top_n=2
):
# define postprocessors
postproc = MetadataReplacementPostProcessor(target_metadata_key="window")
rerank = SentenceTransformerRerank(
top_n=rerank_top_n, model="BAAI/bge-reranker-base"
)
sentence_window_engine = sentence_index.as_query_engine(
similarity_top_k=similarity_top_k, node_postprocessors=[postproc, rerank]
)
return sentence_window_engine
def get_sentence_window_query_engine_en(
sentence_index_en, similarity_top_k=6, rerank_top_n=2
):
# define postprocessors
postproc = MetadataReplacementPostProcessor(target_metadata_key="window")
rerank = SentenceTransformerRerank(
top_n=rerank_top_n, model="BAAI/bge-reranker-base"
)
sentence_window_engine_en = sentence_index_en.as_query_engine(
similarity_top_k=similarity_top_k, node_postprocessors=[postproc, rerank]
)
return sentence_window_engine_en调用之前定义的 build_sentence_window_index 函数，传入文档列表、语言模型实例和保存目录，以构建句子窗口索引。
In [39]python · cell 70
python
from llama_index.llms import OpenAI
index = build_sentence_window_index(
[document],
llm=OpenAI(model="gpt-3.5-turbo", temperature=0.1),
save_dir="./sentence_index",
)
index_en = build_sentence_window_index_en(
[document_en],
llm=OpenAI(model="gpt-3.5-turbo", temperature=0.1),
save_dir="./sentence_index_en",
)
调用之前定义的 get_sentence_window_query_engine 函数，传入构建的句子窗口索引和相似性 top k，以获取句子窗口的查询引擎。
在这里，similarity_top_k 设置为 6。
In [40]python · cell 72
python
query_engine = get_sentence_window_query_engine(index, similarity_top_k=6)
query_engine_en = get_sentence_window_query_engine(index_en, similarity_top_k=6)三、TruLens评测
从名为 'generated_questions.text' 的文件中读取生成的问题，将其存储在 eval_questions 列表中。
In [41]python · cell 75
python
eval_questions = []
with open('data/generated_questions.txt', 'r') as file:
for line in file:
# Remove newline character and convert to integer
item = line.strip()
eval_questions.append(item)
eval_questions_en = []
with open('data/generated_questions_en.txt', 'r') as file:
for line in file:
# Remove newline character and convert to integer
item = line.strip()
eval_questions.append(item)定义了一个函数 run_evals，该函数接受生成的问题列表、TruLens 记录器和查询引擎作为参数。对于每个问题，使用 TruLens 记录器开始记录，然后使用查询引擎执行查询。
In [42]python · cell 77
python
from trulens_eval import Tru
def run_evals(eval_questions, tru_recorder, query_engine):
for question in eval_questions:
with tru_recorder as recording:
response = query_engine.query(question)
def run_evals_en(eval_questions_en, tru_recorder, query_engine):
for question in eval_questions_en:
with tru_recorder as recording:
response = query_engine.query(question)使用 Tru 类的 reset_database 方法重置 TruLens 数据库。
In [43]python · cell 79
python
from utils import get_prebuilt_trulens_recorder
from trulens_eval import Tru
tru = Tru()
tru.reset_database()Output
🦑 Tru initialized with db url sqlite:///default.sqlite . 🛑 Secret keys may be written to the database. See the `database_redact_keys` option of `Tru` to prevent this.
3.1 滑窗尺寸设置为1
调用之前定义的函数 build_sentence_window_index 和 get_sentence_window_query_engine, 分别构建了句子窗口索引和查询引擎。这里设置了窗口大小为 1，并指定了保存目录为 "sentence_index_1"。
In [44]python · cell 82
python
sentence_index_1 = build_sentence_window_index(
documents,
llm=OpenAI(model="gpt-3.5-turbo", temperature=0.1),
embed_model="local:BAAI/bge-small-zh-v1.5", # "local:BAAI/bge-small-en-v1.5" for english
sentence_window_size=1,
save_dir="sentence_index_1",
)
sentence_window_engine_1 = get_sentence_window_query_engine(
sentence_index_1
)
tru_recorder_1 = get_prebuilt_trulens_recorder(
sentence_window_engine_1,
app_id='sentence window engine 1'
)In [45]python · cell 83
python
sentence_index_1_en = build_sentence_window_index_en(
documents_en,
llm=OpenAI(model="gpt-3.5-turbo", temperature=0.1),
embed_model="local:BAAI/bge-small-en-v1.5", # "local:BAAI/bge-small-en-v1.5" for english
sentence_window_size=1,
save_dir="sentence_index_1_en",
)
sentence_window_engine_1_en = get_sentence_window_query_engine(
sentence_index_1_en
)
tru_recorder_1_en = get_prebuilt_trulens_recorder(
sentence_window_engine_1_en,
app_id='sentence window engine 1_en'
)调用之前定义的评估函数 run_evals，传入生成的问题列表、TruLens 记录器 tru_recorder_1 和构建的查询引擎 sentence_window_engine_1，运行评估任务。
In [46]python · cell 85
python
run_evals(eval_questions, tru_recorder_1, sentence_window_engine_1)
run_evals(eval_questions_en, tru_recorder_1, sentence_window_engine_1_en)Output
openai request failed <class 'openai.RateLimitError'>=Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-3.5-turbo in organization org-me3Y2JVoMQFvYW4UUurcFXXM on tokens per min (TPM): Limit 60000, Used 58250, Requested 1999. Please try again in 249ms. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}. Retries remaining=3.
openai request failed <class 'openai.RateLimitError'>=Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-3.5-turbo in organization org-me3Y2JVoMQFvYW4UUurcFXXM on tokens per min (TPM): Limit 60000, Used 59832, Requested 502. Please try again in 334ms. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}. Retries remaining=3.
openai request failed <class 'openai.RateLimitError'>=Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-3.5-turbo in organization org-me3Y2JVoMQFvYW4UUurcFXXM on tokens per min (TPM): Limit 60000, Used 58502, Requested 1922. Please try again in 424ms. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}. Retries remaining=3.
openai request failed <class 'openai.RateLimitError'>=Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-3.5-turbo in organization org-me3Y2JVoMQFvYW4UUurcFXXM on tokens per min (TPM): Limit 60000, Used 59610, Requested 1723. Please try again in 1.333s. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}. Retries remaining=3.
In [47]python · cell 86
python
tru_recorder_1 = get_prebuilt_trulens_recorder(
sentence_window_engine_1,
app_id='sentence window engine 1'
)
tru_recorder_1_en = get_prebuilt_trulens_recorder(
sentence_window_engine_1_en,
app_id='sentence window engine 1_en'
)查看结果
In [48]python · cell 88
python
records, feedback = tru.get_records_and_feedback(app_ids=[])In [49]python · cell 89
python
records.head()Output
app_id \
0 sentence window engine 1
1 sentence window engine 1
2 sentence window engine 1
3 sentence window engine 1
4 sentence window engine 1
app_json \
0 {"tru_class_info": {"name": "TruLlama", "modul...
1 {"tru_class_info": {"name": "TruLlama", "modul...
2 {"tru_class_info": {"name": "TruLlama", "modul...
3 {"tru_class_info": {"name": "TruLlama", "modul...
4 {"tru_class_info": {"name": "TruLlama", "modul...
type \
0 RetrieverQueryEngine(llama_index.query_engine....
1 RetrieverQueryEngine(llama_index.query_engine....
2 RetrieverQueryEngine(llama_index.query_engine....
3 RetrieverQueryEngine(llama_index.query_engine....
4 RetrieverQueryEngine(llama_index.query_engine....
record_id \
0 record_hash_87b8d0d554e7d74fa19c16f4692d69cf
1 record_hash_b9425d9aa02130eec6c73f7cc6f700f8
2 record_hash_8c266922b3864f14c5aa3fd4fc98923c
3 record_hash_48957156666710cd55d5059c9ed69b56
4 record_hash_c3d563072ae3ef443e6e102c53e1677e
input \
0 "\u4eba\u5de5\u667a\u80fd\u4e2d\u7684\u5148\u9...
1 "\u4eba\u5de5\u667a\u80fd\u7684\u81ea\u6211\u6...
2 "\u7ba1\u7406\u8005\u5982\u4f55\u7ba1\u7406AI\...
3 "\u5f3a\u4eba\u5de5\u667a\u80fd\u662f\u4ec0\u4...
4 "\u4eba\u5de5\u667a\u80fd\u88ab\u6ee5\u7528\u5...
output tags \
0 "\u5148\u9a8c\u77e5\u8bc6\u5728\u4eba\u5de5\u6... -
1 "The self-updating and self-improving capabili... -
2 "Management should consider adjusting their wo... -
3 "\u5f3a\u4eba\u5de5\u667a\u80fd\u662f\u4e00\u7... -
4 "The misuse of artificial intelligence can lea... -
record_json \
0 {"record_id": "record_hash_87b8d0d554e7d74fa19...
1 {"record_id": "record_hash_b9425d9aa02130eec6c...
2 {"record_id": "record_hash_8c266922b3864f14c5a...
3 {"record_id": "record_hash_48957156666710cd55d...
4 {"record_id": "record_hash_c3d563072ae3ef443e6...
cost_json \
0 {"n_requests": 0, "n_successful_requests": 0, ...
1 {"n_requests": 0, "n_successful_requests": 0, ...
2 {"n_requests": 0, "n_successful_requests": 0, ...
3 {"n_requests": 0, "n_successful_requests": 0, ...
4 {"n_requests": 0, "n_successful_requests": 0, ...
perf_json \
0 {"start_time": "2024-03-12T10:30:49.113462", "...
1 {"start_time": "2024-03-12T10:31:02.914647", "...
2 {"start_time": "2024-03-12T10:31:09.505494", "...
3 {"start_time": "2024-03-12T10:31:12.872050", "...
4 {"start_time": "2024-03-12T10:31:21.670386", "...
ts Answer Relevance Context Relevance \
0 2024-03-12T10:31:02.269094 0.9 0.40
1 2024-03-12T10:31:09.293573 1.0 0.60
2 2024-03-12T10:31:12.333074 0.8 0.25
3 2024-03-12T10:31:21.471832 0.8 0.80
4 2024-03-12T10:31:28.646002 0.9 0.50
Groundedness Answer Relevance_calls \
0 1.000000 [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...
1 0.800000 [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...
2 1.000000 [{'args': {'prompt': '管理者如何管理AI？', 'response':...
3 0.333333 [{'args': {'prompt': '强人工智能是什么？', 'response': ...
4 1.000000 [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...
Context Relevance_calls \
0 [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...
1 [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...
2 [{'args': {'prompt': '管理者如何管理AI？', 'response':...
3 [{'args': {'prompt': '强人工智能是什么？', 'response': ...
4 [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...
Groundedness_calls latency total_tokens \
0 [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... 13 0
1 [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... 6 0
2 [{'args': {'source': '任何的科技都会有瓶颈， 摩尔定律 到⽬前也遇到相... 2 0
3 [{'args': {'source': 'The Behavioral and Brain... 8 0
4 [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... 6 0
total_cost
0 0.0
1 0.0
2 0.0
3 0.0
4 0.0
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| app_id | app_json | type | record_id | input | output | tags | record_json | cost_json | perf_json | ts | Answer Relevance | Context Relevance | Groundedness | Answer Relevance_calls | Context Relevance_calls | Groundedness_calls | latency | total_tokens | total_cost | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | sentence window engine 1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.query_engine.... | record_hash_87b8d0d554e7d74fa19c16f4692d69cf | "\u4eba\u5de5\u667a\u80fd\u4e2d\u7684\u5148\u9... | "\u5148\u9a8c\u77e5\u8bc6\u5728\u4eba\u5de5\u6... | - | {"record_id": "record_hash_87b8d0d554e7d74fa19... | {"n_requests": 0, "n_successful_requests": 0, ... | {"start_time": "2024-03-12T10:30:49.113462", "... | 2024-03-12T10:31:02.269094 | 0.9 | 0.40 | 1.000000 | [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're... | [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're... | [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... | 13 | 0 | 0.0 |
| 1 | sentence window engine 1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.query_engine.... | record_hash_b9425d9aa02130eec6c73f7cc6f700f8 | "\u4eba\u5de5\u667a\u80fd\u7684\u81ea\u6211\u6... | "The self-updating and self-improving capabili... | - | {"record_id": "record_hash_b9425d9aa02130eec6c... | {"n_requests": 0, "n_successful_requests": 0, ... | {"start_time": "2024-03-12T10:31:02.914647", "... | 2024-03-12T10:31:09.293573 | 1.0 | 0.60 | 0.800000 | [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人... | [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人... | [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... | 6 | 0 | 0.0 |
| 2 | sentence window engine 1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.query_engine.... | record_hash_8c266922b3864f14c5aa3fd4fc98923c | "\u7ba1\u7406\u8005\u5982\u4f55\u7ba1\u7406AI\... | "Management should consider adjusting their wo... | - | {"record_id": "record_hash_8c266922b3864f14c5a... | {"n_requests": 0, "n_successful_requests": 0, ... | {"start_time": "2024-03-12T10:31:09.505494", "... | 2024-03-12T10:31:12.333074 | 0.8 | 0.25 | 1.000000 | [{'args': {'prompt': '管理者如何管理AI？', 'response':... | [{'args': {'prompt': '管理者如何管理AI？', 'response':... | [{'args': {'source': '任何的科技都会有瓶颈， 摩尔定律 到⽬前也遇到相... | 2 | 0 | 0.0 |
| 3 | sentence window engine 1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.query_engine.... | record_hash_48957156666710cd55d5059c9ed69b56 | "\u5f3a\u4eba\u5de5\u667a\u80fd\u662f\u4ec0\u4... | "\u5f3a\u4eba\u5de5\u667a\u80fd\u662f\u4e00\u7... | - | {"record_id": "record_hash_48957156666710cd55d... | {"n_requests": 0, "n_successful_requests": 0, ... | {"start_time": "2024-03-12T10:31:12.872050", "... | 2024-03-12T10:31:21.471832 | 0.8 | 0.80 | 0.333333 | [{'args': {'prompt': '强人工智能是什么？', 'response': ... | [{'args': {'prompt': '强人工智能是什么？', 'response': ... | [{'args': {'source': 'The Behavioral and Brain... | 8 | 0 | 0.0 |
| 4 | sentence window engine 1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.query_engine.... | record_hash_c3d563072ae3ef443e6e102c53e1677e | "\u4eba\u5de5\u667a\u80fd\u88ab\u6ee5\u7528\u5... | "The misuse of artificial intelligence can lea... | - | {"record_id": "record_hash_c3d563072ae3ef443e6... | {"n_requests": 0, "n_successful_requests": 0, ... | {"start_time": "2024-03-12T10:31:21.670386", "... | 2024-03-12T10:31:28.646002 | 0.9 | 0.50 | 1.000000 | [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons... | [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons... | [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... | 6 | 0 | 0.0 |
3.2 滑窗尺寸设为3
调用之前定义的函数，构建了句子窗口索引、查询引擎和 TruLens 记录器。这里设置了窗口大小为 3，并指定了保存目录为 "sentence_index_3"。
In [ ]python · cell 92
python
sentence_index_3 = build_sentence_window_index(
documents,
llm=OpenAI(model="gpt-3.5-turbo", temperature=0.1),
embed_model="local:BAAI/bge-small-zh-v1.5", # "local:BAAI/bge-small-en-v1.5" for english
sentence_window_size=3,
save_dir="sentence_index_3",
)
sentence_window_engine_3 = get_sentence_window_query_engine(
sentence_index_3
)
tru_recorder_3 = get_prebuilt_trulens_recorder(
sentence_window_engine_3,
app_id='sentence window engine 3'
)In [ ]python · cell 93
python
sentence_index_3_en = build_sentence_window_index_en(
documents_en,
llm=OpenAI(model="gpt-3.5-turbo", temperature=0.1),
embed_model="local:BAAI/bge-small-en-v1.5", # "local:BAAI/bge-small-en-v1.5" for english
sentence_window_size=3,
save_dir="sentence_index_3_en",
)
sentence_window_engine_3_en = get_sentence_window_query_engine(
sentence_index_3_en
)
tru_recorder_3_en = get_prebuilt_trulens_recorder(
sentence_window_engine_3_en,
app_id='sentence window engine 3_en'
)调用 run_evals 函数，传入生成的问题列表 eval_questions、TruLens 记录器 tru_recorder_3 和构建的查询引擎 sentence_window_engine_3，运行评估任务。
In [ ]python · cell 95
python
run_evals(eval_questions, tru_recorder_3, sentence_window_engine_3)
run_evals(eval_questions_en, tru_recorder_3_en, sentence_window_engine_3_en)In [ ]python · cell 96
python
records, feedback = tru.get_records_and_feedback(app_ids=[])In [ ]python · cell 97
python
records.head()Output
app_id \
0 sentence window engine 1
1 sentence window engine 1
2 sentence window engine 1
3 sentence window engine 1
4 sentence window engine 1
app_json \
0 {"tru_class_info": {"name": "TruLlama", "modul...
1 {"tru_class_info": {"name": "TruLlama", "modul...
2 {"tru_class_info": {"name": "TruLlama", "modul...
3 {"tru_class_info": {"name": "TruLlama", "modul...
4 {"tru_class_info": {"name": "TruLlama", "modul...
type \
0 RetrieverQueryEngine(llama_index.query_engine....
1 RetrieverQueryEngine(llama_index.query_engine....
2 RetrieverQueryEngine(llama_index.query_engine....
3 RetrieverQueryEngine(llama_index.query_engine....
4 RetrieverQueryEngine(llama_index.query_engine....
record_id \
0 record_hash_e9815da1c66c0943f4d155a06f94e9c4
1 record_hash_140932cb020d95e0554ecf0489eb42f2
2 record_hash_5bd9fa7d1b1997c136b9d1d4ce3d2684
3 record_hash_cd9a2aa2bdf60288d1b04cdbeac630f3
4 record_hash_736205619e133e5333d36a19a29293fe
input \
0 "\u4eba\u5de5\u667a\u80fd\u4e2d\u7684\u5148\u9...
1 "\u4eba\u5de5\u667a\u80fd\u7684\u81ea\u6211\u6...
2 "\u7ba1\u7406\u8005\u5982\u4f55\u7ba1\u7406AI\...
3 "\u5f3a\u4eba\u5de5\u667a\u80fd\u662f\u4ec0\u4...
4 "\u4eba\u5de5\u667a\u80fd\u88ab\u6ee5\u7528\u5...
output tags \
0 "\u5148\u9a8c\u77e5\u8bc6\u5728\u4eba\u5de5\u6... -
1 "The self-updating and self-improving capabili... -
2 "Management should consider adjusting their wo... -
3 "\u5f3a\u4eba\u5de5\u667a\u80fd\u662f\u4e00\u7... -
4 "The misuse of artificial intelligence can lea... -
record_json \
0 {"record_id": "record_hash_e9815da1c66c0943f4d...
1 {"record_id": "record_hash_140932cb020d95e0554...
2 {"record_id": "record_hash_5bd9fa7d1b1997c136b...
3 {"record_id": "record_hash_cd9a2aa2bdf60288d1b...
4 {"record_id": "record_hash_736205619e133e5333d...
cost_json \
0 {"n_requests": 0, "n_successful_requests": 0, ...
1 {"n_requests": 0, "n_successful_requests": 0, ...
2 {"n_requests": 0, "n_successful_requests": 0, ...
3 {"n_requests": 0, "n_successful_requests": 0, ...
4 {"n_requests": 0, "n_successful_requests": 0, ...
perf_json \
0 {"start_time": "2024-03-10T22:55:51.162029", "...
1 {"start_time": "2024-03-10T22:56:04.085254", "...
2 {"start_time": "2024-03-10T22:56:10.552787", "...
3 {"start_time": "2024-03-10T22:56:13.882046", "...
4 {"start_time": "2024-03-10T22:56:22.140156", "...
ts Answer Relevance Context Relevance \
0 2024-03-10T22:56:03.820730 0.9 0.4
1 2024-03-10T22:56:10.348414 1.0 0.5
2 2024-03-10T22:56:13.694339 0.8 0.3
3 2024-03-10T22:56:21.948222 0.9 0.7
4 2024-03-10T22:56:28.835570 0.9 0.3
Groundedness Answer Relevance_calls \
0 1.0 [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...
1 0.8 [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...
2 0.9 [{'args': {'prompt': '管理者如何管理AI？', 'response':...
3 1.0 [{'args': {'prompt': '强人工智能是什么？', 'response': ...
4 0.0 [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...
Context Relevance_calls \
0 [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...
1 [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...
2 [{'args': {'prompt': '管理者如何管理AI？', 'response':...
3 [{'args': {'prompt': '强人工智能是什么？', 'response': ...
4 [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...
Groundedness_calls latency total_tokens \
0 [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... 12 0
1 [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... 6 0
2 [{'args': {'source': '任何的科技都会有瓶颈， 摩尔定律 到⽬前也遇到相... 3 0
3 [{'args': {'source': 'The Behavioral and Brain... 8 0
4 [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... 6 0
total_cost
0 0.0
1 0.0
2 0.0
3 0.0
4 0.0
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| app_id | app_json | type | record_id | input | output | tags | record_json | cost_json | perf_json | ts | Answer Relevance | Context Relevance | Groundedness | Answer Relevance_calls | Context Relevance_calls | Groundedness_calls | latency | total_tokens | total_cost | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | sentence window engine 1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.query_engine.... | record_hash_e9815da1c66c0943f4d155a06f94e9c4 | "\u4eba\u5de5\u667a\u80fd\u4e2d\u7684\u5148\u9... | "\u5148\u9a8c\u77e5\u8bc6\u5728\u4eba\u5de5\u6... | - | {"record_id": "record_hash_e9815da1c66c0943f4d... | {"n_requests": 0, "n_successful_requests": 0, ... | {"start_time": "2024-03-10T22:55:51.162029", "... | 2024-03-10T22:56:03.820730 | 0.9 | 0.4 | 1.0 | [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're... | [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're... | [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... | 12 | 0 | 0.0 |
| 1 | sentence window engine 1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.query_engine.... | record_hash_140932cb020d95e0554ecf0489eb42f2 | "\u4eba\u5de5\u667a\u80fd\u7684\u81ea\u6211\u6... | "The self-updating and self-improving capabili... | - | {"record_id": "record_hash_140932cb020d95e0554... | {"n_requests": 0, "n_successful_requests": 0, ... | {"start_time": "2024-03-10T22:56:04.085254", "... | 2024-03-10T22:56:10.348414 | 1.0 | 0.5 | 0.8 | [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人... | [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人... | [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... | 6 | 0 | 0.0 |
| 2 | sentence window engine 1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.query_engine.... | record_hash_5bd9fa7d1b1997c136b9d1d4ce3d2684 | "\u7ba1\u7406\u8005\u5982\u4f55\u7ba1\u7406AI\... | "Management should consider adjusting their wo... | - | {"record_id": "record_hash_5bd9fa7d1b1997c136b... | {"n_requests": 0, "n_successful_requests": 0, ... | {"start_time": "2024-03-10T22:56:10.552787", "... | 2024-03-10T22:56:13.694339 | 0.8 | 0.3 | 0.9 | [{'args': {'prompt': '管理者如何管理AI？', 'response':... | [{'args': {'prompt': '管理者如何管理AI？', 'response':... | [{'args': {'source': '任何的科技都会有瓶颈， 摩尔定律 到⽬前也遇到相... | 3 | 0 | 0.0 |
| 3 | sentence window engine 1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.query_engine.... | record_hash_cd9a2aa2bdf60288d1b04cdbeac630f3 | "\u5f3a\u4eba\u5de5\u667a\u80fd\u662f\u4ec0\u4... | "\u5f3a\u4eba\u5de5\u667a\u80fd\u662f\u4e00\u7... | - | {"record_id": "record_hash_cd9a2aa2bdf60288d1b... | {"n_requests": 0, "n_successful_requests": 0, ... | {"start_time": "2024-03-10T22:56:13.882046", "... | 2024-03-10T22:56:21.948222 | 0.9 | 0.7 | 1.0 | [{'args': {'prompt': '强人工智能是什么？', 'response': ... | [{'args': {'prompt': '强人工智能是什么？', 'response': ... | [{'args': {'source': 'The Behavioral and Brain... | 8 | 0 | 0.0 |
| 4 | sentence window engine 1 | {"tru_class_info": {"name": "TruLlama", "modul... | RetrieverQueryEngine(llama_index.query_engine.... | record_hash_736205619e133e5333d36a19a29293fe | "\u4eba\u5de5\u667a\u80fd\u88ab\u6ee5\u7528\u5... | "The misuse of artificial intelligence can lea... | - | {"record_id": "record_hash_736205619e133e5333d... | {"n_requests": 0, "n_successful_requests": 0, ... | {"start_time": "2024-03-10T22:56:22.140156", "... | 2024-03-10T22:56:28.835570 | 0.9 | 0.3 | 0.0 | [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons... | [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons... | [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能 - ... | 6 | 0 | 0.0 |