Chapter 79
第六章 评估
NotebookPython 3 (ipykernel)29 cells
一、评估方法及误差分析
1.1 评估方法
评估一个生成式模型的好坏是相当困难的，因为大部分时候都是开放性问答，没有统一确定的答案，造成评分很难量化。目前来说，常见的评估方式有以下3种：
- 人工评估：人工评估是最为可靠的方案，但是很费时费力。
- 测试集评估：给模型一张试卷（即测试集），让模型去答题，最后计算得分。但测试集的好坏决定了评估的效用。
- Elo评分系统：Elo评分系统为每位选手分配一个数值评分，代表其技能水平。当两位具有不同Elo评分的选手相互对战时，比赛结果会导致他们的评分发生变化。如果评分较低的选手战胜评分较高的选手，他们的评分将会相对较大地提升，而如果评分较高的选手战胜评分较低的选手，评分的变化会较小。评分调整的幅度取决于选手之间评分的差异以及比赛的预期结果。该方法需要多个选手，即多个模型，且需要多次比赛才能比较出模型的好坏，效率较低。
考虑时间和效率，我们一般采用测试集评估。
1.2 测试集标准
一般来说，一个好的测试集需要满足以下几个条件：
- 高质量
- 准确性
- 泛化性
- 没有在训练集中出现过
以下为常见的适用于大语言模型的基准测试集：
- ARC：ARC（Abstraction and Reasoning Corpus）是一个基准测试集，用于评估在有限示例情况下进行抽象推理能力的模型，要求具备一定的“核心知识”概念，如对象、目标状态、计数和基本几何等。
- HellaSwag：一个对常识的测试。
- MMLU：一个多任务测试评分集，涵盖了初等数学、美国历史、计算机科学、法律等等。
- TruthfulQA：TruthfulQA衡量模型复制网上常见虚假信息的倾向。
注意，这些基准测试集都较为泛化，如果你微调训练的模型属于某个特定领域的，那最好还是有该领域的测试集，评估更为准确可靠。
1.3 误差分析
在评估前我们需要知道模型在微调前的表现是怎样的，方便我们判断模型是变好了还是变差了，分析是参数还是数据的问题。
注意测试集里是否存在以下错误：
- 单词拼写错误
- 太长的语句
- 重复啰嗦
如果存在这些错误，可以进行改正以提高测试集的质量。
1.4 微调实用技巧
最后在分享一些微调的实用技巧：
- 定义你的任务；
- 收集和这个任务有关的数据；
- 若数据较少可以尝试生成一些数据；
- 微调一个小模型（如400MB~1B的模型）；
- 改变喂给模型的数据量；
- 评估LLM表现更好了还是更差了；
- 收集更多数据以提升模型性能；
- 增加任务困难度；
- 增加模型规模以增强性能。
二、代码实战
In [6]python · cell 4
python
import os
import pandas as pd
import datasets
from tqdm import tqdm
from utilities import *
from transformers import AutoTokenizer, AutoModelForCausalLM
logger = logging.getLogger(__name__)
global_config = None
# 为防止hugging face报SSL等错误导致无法下载模型和数据，需要设置自己的代理端口：
os.environ['http_proxy'] = 'http://127.0.0.1:7890'
os.environ['https_proxy'] = 'http://127.0.0.1:7890'2.1 加载测试集和模型
这里以lamini_docs测试集为例，第一次加载的话需要下载（需要科学上网）。
In [7]python · cell 6
python
dataset = datasets.load_dataset("lamini/lamini_docs")Output
Found cached dataset parquet (C:/Users/Administrator/.cache/huggingface/datasets/lamini___parquet/lamini--lamini_docs-c73aea7bc123faf0/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
0%| | 0/2 [00:00<?, ?it/s]
In [8]python · cell 7
python
test_dataset = dataset["test"]
print(test_dataset[0]["question"])
print(test_dataset[0]["answer"])Output
Can Lamini generate technical documentation or user manuals for software projects? Yes, Lamini can generate technical documentation and user manuals for software projects. It uses natural language generation techniques to create clear and concise documentation that is easy to understand for both technical and non-technical users. This can save developers a significant amount of time and effort in creating documentation, allowing them to focus on other aspects of their projects.
接下来加载上一章中训练好的模型：
In [9]python · cell 9
python
model_name = "lamini/lamini_docs_finetuned"
tokenizer = AutoTokenizer.from_pretrained(model_name) # 分词器
model = AutoModelForCausalLM.from_pretrained(model_name)Output
Some weights of GPTNeoXForCausalLM were not initialized from the model checkpoint at lamini/lamini_docs_finetuned and are newly initialized: ['gpt_neox.layers.2.attention.bias', 'gpt_neox.layers.2.attention.masked_bias', 'gpt_neox.layers.5.attention.masked_bias', 'gpt_neox.layers.4.attention.bias', 'gpt_neox.layers.1.attention.bias', 'gpt_neox.layers.5.attention.bias', 'gpt_neox.layers.0.attention.masked_bias', 'gpt_neox.layers.3.attention.masked_bias', 'gpt_neox.layers.4.attention.masked_bias', 'gpt_neox.layers.1.attention.masked_bias', 'gpt_neox.layers.0.attention.bias', 'gpt_neox.layers.3.attention.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2.2 定义函数
定义一些函数，方便后续调用。
In [10]python · cell 11
python
def is_exact_match(a:str, b:str) -> bool:
"""
评估模型的回答和实际答案是否完全一致
"""
return a.strip() == b.strip()In [11]python · cell 12
python
def inference(text, model, tokenizer, max_input_tokens=1000, max_output_tokens=100):
"""
模型推理，即模型根据输入的问题生成文字回答。
Args:
text: 问题文本，str
model： 模型
tokenizer: 分词器
max_input_tokens：最大输入长度, int
max_output_tokens: 最大输出长度, int
Return:
generated_text_answer: 模型生成的文字回复，str
"""
# Tokenize
tokenizer.pad_token = tokenizer.eos_token
input_ids = tokenizer.encode(
text,
return_tensors="pt",
truncation=True,
max_length=max_input_tokens
)
# 生成回复
device = model.device
generated_tokens_with_prompt = model.generate(
input_ids = input_ids.to(device),
max_length = max_output_tokens,
pad_token_id = tokenizer.eos_token_id
)
# 解码
generated_text_with_prompt = tokenizer.batch_decode(generated_tokens_with_prompt, skip_special_tokens=True)
# 获得输出
generated_text_answer = generated_text_with_prompt[0][len(text):]
return generated_text_answerIn [12]python · cell 14
python
# 先将模型设为eval模式，不再计算梯度及更新参数
model.eval()Output
GPTNeoXForCausalLM(
(gpt_neox): GPTNeoXModel(
(embed_in): Embedding(50304, 512)
(layers): ModuleList(
(0): GPTNeoXLayer(
(input_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(post_attention_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(attention): GPTNeoXAttention(
(rotary_emb): RotaryEmbedding()
(query_key_value): Linear(in_features=512, out_features=1536, bias=True)
(dense): Linear(in_features=512, out_features=512, bias=True)
)
(mlp): GPTNeoXMLP(
(dense_h_to_4h): Linear(in_features=512, out_features=2048, bias=True)
(dense_4h_to_h): Linear(in_features=2048, out_features=512, bias=True)
(act): GELUActivation()
)
)
(1): GPTNeoXLayer(
(input_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(post_attention_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(attention): GPTNeoXAttention(
(rotary_emb): RotaryEmbedding()
(query_key_value): Linear(in_features=512, out_features=1536, bias=True)
(dense): Linear(in_features=512, out_features=512, bias=True)
)
(mlp): GPTNeoXMLP(
(dense_h_to_4h): Linear(in_features=512, out_features=2048, bias=True)
(dense_4h_to_h): Linear(in_features=2048, out_features=512, bias=True)
(act): GELUActivation()
)
)
(2): GPTNeoXLayer(
(input_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(post_attention_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(attention): GPTNeoXAttention(
(rotary_emb): RotaryEmbedding()
(query_key_value): Linear(in_features=512, out_features=1536, bias=True)
(dense): Linear(in_features=512, out_features=512, bias=True)
)
(mlp): GPTNeoXMLP(
(dense_h_to_4h): Linear(in_features=512, out_features=2048, bias=True)
(dense_4h_to_h): Linear(in_features=2048, out_features=512, bias=True)
(act): GELUActivation()
)
)
(3): GPTNeoXLayer(
(input_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(post_attention_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(attention): GPTNeoXAttention(
(rotary_emb): RotaryEmbedding()
(query_key_value): Linear(in_features=512, out_features=1536, bias=True)
(dense): Linear(in_features=512, out_features=512, bias=True)
)
(mlp): GPTNeoXMLP(
(dense_h_to_4h): Linear(in_features=512, out_features=2048, bias=True)
(dense_4h_to_h): Linear(in_features=2048, out_features=512, bias=True)
(act): GELUActivation()
)
)
(4): GPTNeoXLayer(
(input_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(post_attention_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(attention): GPTNeoXAttention(
(rotary_emb): RotaryEmbedding()
(query_key_value): Linear(in_features=512, out_features=1536, bias=True)
(dense): Linear(in_features=512, out_features=512, bias=True)
)
(mlp): GPTNeoXMLP(
(dense_h_to_4h): Linear(in_features=512, out_features=2048, bias=True)
(dense_4h_to_h): Linear(in_features=2048, out_features=512, bias=True)
(act): GELUActivation()
)
)
(5): GPTNeoXLayer(
(input_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(post_attention_layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
(attention): GPTNeoXAttention(
(rotary_emb): RotaryEmbedding()
(query_key_value): Linear(in_features=512, out_features=1536, bias=True)
(dense): Linear(in_features=512, out_features=512, bias=True)
)
(mlp): GPTNeoXMLP(
(dense_h_to_4h): Linear(in_features=512, out_features=2048, bias=True)
(dense_4h_to_h): Linear(in_features=2048, out_features=512, bias=True)
(act): GELUActivation()
)
)
)
(final_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
)
(embed_out): Linear(in_features=512, out_features=50304, bias=False)
)先用一个问题测试一下模型能否正常推理：
In [13]python · cell 16
python
test_question = test_dataset[0]["question"]
generated_answer = inference(test_question, model, tokenizer) # 模型推理，即生成回复
print(f'test_question: {test_question}')
print(f'generated_answer: {generated_answer}')Output
test_question: Can Lamini generate technical documentation or user manuals for software projects? generated_answer: Yes, Lamini can generate technical documentation or user manuals for software projects. This can be achieved by providing a prompt for a specific technical question or question to the LLM Engine, or by providing a prompt for a specific technical question or question. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini
可以看到lamini模型能够正常回答这个问题。来对比下标准答案：
In [14]python · cell 18
python
answer = test_dataset[0]["answer"]
print(answer)Output
Yes, Lamini can generate technical documentation and user manuals for software projects. It uses natural language generation techniques to create clear and concise documentation that is easy to understand for both technical and non-technical users. This can save developers a significant amount of time and effort in creating documentation, allowing them to focus on other aspects of their projects.
和标准答案还是非常接近的。
In [15]python · cell 20
python
exact_match = is_exact_match(generated_answer, answer)
print(exact_match)Output
False
可惜没法做到和标准答案一字不落地完全匹配。这就是LLM评估的一个难点，要做到百分百匹配标准答案是很困难的，要求过于苛刻，其实只要意思相近就可以了。
2.4 在测试集上进行测试
单个问题测试通过后，我们接下来在测试集上进行测试：
In [20]python · cell 23
python
n = 10
metrics = {'exact_matches': []}
predictions = []
for i, item in tqdm(enumerate(test_dataset)):
print(f"评估第{i+1}个样本: " + str(item))
question = item['question']
answer = item['answer']
try:
predicted_answer = inference(question, model, tokenizer)
except:
continue
predictions.append([predicted_answer, answer])
exact_match = is_exact_match(generated_answer, answer)
metrics['exact_matches'].append(exact_match)
if i > n and n != -1:
break
print('完全匹配个数: ', sum(metrics['exact_matches']))Output
0it [00:00, ?it/s]
评估第1个样本: {'question': 'Can Lamini generate technical documentation or user manuals for software projects?', 'answer': 'Yes, Lamini can generate technical documentation and user manuals for software projects. It uses natural language generation techniques to create clear and concise documentation that is easy to understand for both technical and non-technical users. This can save developers a significant amount of time and effort in creating documentation, allowing them to focus on other aspects of their projects.', 'input_ids': [5804, 418, 4988, 74, 6635, 7681, 10097, 390, 2608, 11595, 84, 323, 3694, 6493, 32, 4374, 13, 418, 4988, 74, 476, 6635, 7681, 10097, 285, 2608, 11595, 84, 323, 3694, 6493, 15, 733, 4648, 3626, 3448, 5978, 5609, 281, 2794, 2590, 285, 44003, 10097, 326, 310, 3477, 281, 2096, 323, 1097, 7681, 285, 1327, 14, 48746, 4212, 15, 831, 476, 5321, 12259, 247, 1534, 2408, 273, 673, 285, 3434, 275, 6153, 10097, 13, 6941, 731, 281, 2770, 327, 643, 7794, 273, 616, 6493, 15], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [5804, 418, 4988, 74, 6635, 7681, 10097, 390, 2608, 11595, 84, 323, 3694, 6493, 32, 4374, 13, 418, 4988, 74, 476, 6635, 7681, 10097, 285, 2608, 11595, 84, 323, 3694, 6493, 15, 733, 4648, 3626, 3448, 5978, 5609, 281, 2794, 2590, 285, 44003, 10097, 326, 310, 3477, 281, 2096, 323, 1097, 7681, 285, 1327, 14, 48746, 4212, 15, 831, 476, 5321, 12259, 247, 1534, 2408, 273, 673, 285, 3434, 275, 6153, 10097, 13, 6941, 731, 281, 2770, 327, 643, 7794, 273, 616, 6493, 15]}
1it [00:01, 1.56s/it]
评估第2个样本: {'question': 'How do I include my API key in the Authorization HTTP header?', 'answer': 'The Authorization HTTP header should include the API key in the following format: Authorization: Bearer <YOUR-KEY-HERE>.', 'input_ids': [2347, 513, 309, 2486, 619, 8990, 2234, 275, 253, 10360, 1320, 17607, 10478, 32, 510, 10360, 1320, 17607, 10478, 943, 2486, 253, 8990, 2234, 275, 253, 1563, 5981, 27, 10360, 1320, 27, 2325, 12287, 654, 58, 11862, 14, 13888, 14, 41, 8147, 13208], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [2347, 513, 309, 2486, 619, 8990, 2234, 275, 253, 10360, 1320, 17607, 10478, 32, 510, 10360, 1320, 17607, 10478, 943, 2486, 253, 8990, 2234, 275, 253, 1563, 5981, 27, 10360, 1320, 27, 2325, 12287, 654, 58, 11862, 14, 13888, 14, 41, 8147, 13208]}
2it [00:03, 1.58s/it]
评估第3个样本: {'question': "Is there a section explaining the code's approach to handling versioning and compatibility?", 'answer': 'Yes, the code includes a version parameter in the FeedbackOperation class constructor, which allows for handling versioning and compatibility.', 'input_ids': [2513, 627, 247, 2593, 15571, 253, 2127, 434, 2746, 281, 10885, 2715, 272, 285, 22862, 32, 4374, 13, 253, 2127, 3797, 247, 2715, 4764, 275, 253, 34600, 2135, 17547, 966, 16757, 13, 534, 4483, 323, 10885, 2715, 272, 285, 22862, 15], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [2513, 627, 247, 2593, 15571, 253, 2127, 434, 2746, 281, 10885, 2715, 272, 285, 22862, 32, 4374, 13, 253, 2127, 3797, 247, 2715, 4764, 275, 253, 34600, 2135, 17547, 966, 16757, 13, 534, 4483, 323, 10885, 2715, 272, 285, 22862, 15]}
3it [00:04, 1.67s/it]
评估第4个样本: {'question': 'Is there a community or support forum available for Lamini users?', 'answer': 'Yes, there is a community forum available for Lamini users. The Lamini community forum can be accessed through the Lamini website and provides a platform for users to ask questions, share ideas, and collaborate with other developers using the library. Additionally, the Lamini team is active on the forum and provides support and guidance to users as needed.', 'input_ids': [2513, 627, 247, 3114, 390, 1329, 12209, 2130, 323, 418, 4988, 74, 4212, 32, 4374, 13, 627, 310, 247, 3114, 12209, 2130, 323, 418, 4988, 74, 4212, 15, 380, 418, 4988, 74, 3114, 12209, 476, 320, 19197, 949, 253, 418, 4988, 74, 4422, 285, 3400, 247, 5147, 323, 4212, 281, 1642, 3533, 13, 3894, 5697, 13, 285, 42124, 342, 643, 12259, 970, 253, 6335, 15, 9157, 13, 253, 418, 4988, 74, 2285, 310, 3939, 327, 253, 12209, 285, 3400, 1329, 285, 12925, 281, 4212, 347, 3058, 15], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [2513, 627, 247, 3114, 390, 1329, 12209, 2130, 323, 418, 4988, 74, 4212, 32, 4374, 13, 627, 310, 247, 3114, 12209, 2130, 323, 418, 4988, 74, 4212, 15, 380, 418, 4988, 74, 3114, 12209, 476, 320, 19197, 949, 253, 418, 4988, 74, 4422, 285, 3400, 247, 5147, 323, 4212, 281, 1642, 3533, 13, 3894, 5697, 13, 285, 42124, 342, 643, 12259, 970, 253, 6335, 15, 9157, 13, 253, 418, 4988, 74, 2285, 310, 3939, 327, 253, 12209, 285, 3400, 1329, 285, 12925, 281, 4212, 347, 3058, 15]}
4it [00:06, 1.66s/it]
评估第5个样本: {'question': 'Can the Lamini library be utilized for text completion or auto-completion tasks, such as filling in missing words or predicting the next word in a sentence?', 'answer': 'The Lamini library is not specifically designed for text completion or auto-completion tasks. However, it can be used for language modeling and generating text based on a given prompt.', 'input_ids': [5804, 253, 418, 4988, 74, 6335, 320, 12845, 323, 2505, 12240, 390, 6753, 14, 45634, 8892, 13, 824, 347, 12868, 275, 5816, 3000, 390, 21565, 253, 1735, 3159, 275, 247, 6197, 32, 510, 418, 4988, 74, 6335, 310, 417, 5742, 4158, 323, 2505, 12240, 390, 6753, 14, 45634, 8892, 15, 1723, 13, 352, 476, 320, 908, 323, 3448, 14053, 285, 11365, 2505, 1754, 327, 247, 1677, 8959, 15], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [5804, 253, 418, 4988, 74, 6335, 320, 12845, 323, 2505, 12240, 390, 6753, 14, 45634, 8892, 13, 824, 347, 12868, 275, 5816, 3000, 390, 21565, 253, 1735, 3159, 275, 247, 6197, 32, 510, 418, 4988, 74, 6335, 310, 417, 5742, 4158, 323, 2505, 12240, 390, 6753, 14, 45634, 8892, 15, 1723, 13, 352, 476, 320, 908, 323, 3448, 14053, 285, 11365, 2505, 1754, 327, 247, 1677, 8959, 15]}
5it [00:07, 1.54s/it]
评估第6个样本: {'question': 'Are there any costs associated with using Lamini for machine learning tasks, and how does the pricing structure work?', 'answer': 'Lamini offers both free and paid plans for using their machine learning services. The free plan includes limited access to their models and data generator, while the paid plans offer more advanced features and higher usage limits. The pricing structure is based on a pay-as-you-go model, where users are charged based on the number of API requests and data processed. Lamini also offers custom enterprise plans for larger organizations with specific needs.', 'input_ids': [6723, 627, 667, 4815, 2330, 342, 970, 418, 4988, 74, 323, 5145, 4715, 8892, 13, 285, 849, 1057, 253, 20910, 2605, 789, 32, 45, 4988, 74, 6131, 1097, 1959, 285, 5087, 5827, 323, 970, 616, 5145, 4715, 3238, 15, 380, 1959, 2098, 3797, 3710, 2289, 281, 616, 3210, 285, 941, 14156, 13, 1223, 253, 5087, 5827, 3959, 625, 7269, 3386, 285, 2169, 10393, 7787, 15, 380, 20910, 2605, 310, 1754, 327, 247, 2075, 14, 284, 14, 5658, 14, 2184, 1566, 13, 835, 4212, 403, 6636, 1754, 327, 253, 1180, 273, 8990, 9762, 285, 941, 11742, 15, 418, 4988, 74, 671, 6131, 2840, 16100, 5827, 323, 4067, 8889, 342, 2173, 3198, 15], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [6723, 627, 667, 4815, 2330, 342, 970, 418, 4988, 74, 323, 5145, 4715, 8892, 13, 285, 849, 1057, 253, 20910, 2605, 789, 32, 45, 4988, 74, 6131, 1097, 1959, 285, 5087, 5827, 323, 970, 616, 5145, 4715, 3238, 15, 380, 1959, 2098, 3797, 3710, 2289, 281, 616, 3210, 285, 941, 14156, 13, 1223, 253, 5087, 5827, 3959, 625, 7269, 3386, 285, 2169, 10393, 7787, 15, 380, 20910, 2605, 310, 1754, 327, 247, 2075, 14, 284, 14, 5658, 14, 2184, 1566, 13, 835, 4212, 403, 6636, 1754, 327, 253, 1180, 273, 8990, 9762, 285, 941, 11742, 15, 418, 4988, 74, 671, 6131, 2840, 16100, 5827, 323, 4067, 8889, 342, 2173, 3198, 15]}
6it [00:09, 1.54s/it]
评估第7个样本: {'question': 'How do I instantiate the LLM engine using the Lamini Python package?', 'answer': 'You can instantiate the LLM engine using the llama module in the Lamini Python package. To do this, you need to import the LLM engine from the llama module, like this: from llama import LLM.', 'input_ids': [2347, 513, 309, 8164, 4513, 253, 21708, 46, 3948, 970, 253, 418, 4988, 74, 13814, 5522, 32, 1394, 476, 8164, 4513, 253, 21708, 46, 3948, 970, 253, 26198, 2902, 6333, 275, 253, 418, 4988, 74, 13814, 5522, 15, 1916, 513, 436, 13, 368, 878, 281, 1395, 253, 21708, 46, 3948, 432, 253, 26198, 2902, 6333, 13, 751, 436, 27, 432, 26198, 2902, 1395, 21708, 46, 15], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [2347, 513, 309, 8164, 4513, 253, 21708, 46, 3948, 970, 253, 418, 4988, 74, 13814, 5522, 32, 1394, 476, 8164, 4513, 253, 21708, 46, 3948, 970, 253, 26198, 2902, 6333, 275, 253, 418, 4988, 74, 13814, 5522, 15, 1916, 513, 436, 13, 368, 878, 281, 1395, 253, 21708, 46, 3948, 432, 253, 26198, 2902, 6333, 13, 751, 436, 27, 432, 26198, 2902, 1395, 21708, 46, 15]}
7it [00:11, 1.55s/it]
评估第8个样本: {'question': 'Does Lamini provide any mechanisms for model compression or optimization to reduce memory footprint?', 'answer': 'Yes, Lamini provides mechanisms for model compression and optimization to reduce memory footprint. These include techniques such as pruning, quantization, and distillation, which can significantly reduce the size of the model while maintaining its performance. Additionally, Lamini offers support for deploying customized LLMs on edge devices with limited resources, such as mobile phones or IoT devices, through techniques such as model quantization and on-device inference.', 'input_ids': [10795, 418, 4988, 74, 2085, 667, 6297, 323, 1566, 13800, 390, 13757, 281, 4796, 3541, 33257, 32, 4374, 13, 418, 4988, 74, 3400, 6297, 323, 1566, 13800, 285, 13757, 281, 4796, 3541, 33257, 15, 2053, 2486, 5609, 824, 347, 819, 25004, 13, 36643, 13, 285, 940, 21755, 13, 534, 476, 3012, 4796, 253, 1979, 273, 253, 1566, 1223, 11850, 697, 3045, 15, 9157, 13, 418, 4988, 74, 6131, 1329, 323, 45021, 32176, 21708, 12822, 327, 5024, 4095, 342, 3710, 5300, 13, 824, 347, 6109, 15169, 390, 37377, 4095, 13, 949, 5609, 824, 347, 1566, 36643, 285, 327, 14, 10933, 17032, 15], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [10795, 418, 4988, 74, 2085, 667, 6297, 323, 1566, 13800, 390, 13757, 281, 4796, 3541, 33257, 32, 4374, 13, 418, 4988, 74, 3400, 6297, 323, 1566, 13800, 285, 13757, 281, 4796, 3541, 33257, 15, 2053, 2486, 5609, 824, 347, 819, 25004, 13, 36643, 13, 285, 940, 21755, 13, 534, 476, 3012, 4796, 253, 1979, 273, 253, 1566, 1223, 11850, 697, 3045, 15, 9157, 13, 418, 4988, 74, 6131, 1329, 323, 45021, 32176, 21708, 12822, 327, 5024, 4095, 342, 3710, 5300, 13, 824, 347, 6109, 15169, 390, 37377, 4095, 13, 949, 5609, 824, 347, 1566, 36643, 285, 327, 14, 10933, 17032, 15]}
8it [00:12, 1.63s/it]
评估第9个样本: {'question': 'How does the performance of LLMs trained using Lamini compare to models fine-tuned with traditional approaches?', 'answer': 'According to the information provided, Lamini allows developers to train high-performing LLMs on large datasets with just a few lines of code from the Lamini library. The optimizations in this library reach far beyond what’s available to developers now, from more challenging optimizations like RLHF to simpler ones like reducing hallucinations. While there is no direct comparison to traditional approaches mentioned, Lamini aims to make training LLMs faster and more accessible to a wider range of developers.', 'input_ids': [2347, 1057, 253, 3045, 273, 21708, 12822, 10166, 970, 418, 4988, 74, 7277, 281, 3210, 4030, 14, 85, 37437, 342, 5899, 7274, 32, 7130, 281, 253, 1491, 2530, 13, 418, 4988, 74, 4483, 12259, 281, 6194, 1029, 14, 468, 14692, 21708, 12822, 327, 1781, 15302, 342, 816, 247, 1643, 3104, 273, 2127, 432, 253, 418, 4988, 74, 6335, 15, 380, 5556, 5904, 275, 436, 6335, 3986, 2080, 4457, 752, 457, 84, 2130, 281, 12259, 1024, 13, 432, 625, 11132, 5556, 5904, 751, 40228, 21996, 281, 19554, 4394, 751, 8493, 33092, 7097, 15, 3900, 627, 310, 642, 1480, 5301, 281, 5899, 7274, 5393, 13, 418, 4988, 74, 13698, 281, 1056, 3733, 21708, 12822, 7938, 285, 625, 12482, 281, 247, 14200, 2491, 273, 12259, 15], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [2347, 1057, 253, 3045, 273, 21708, 12822, 10166, 970, 418, 4988, 74, 7277, 281, 3210, 4030, 14, 85, 37437, 342, 5899, 7274, 32, 7130, 281, 253, 1491, 2530, 13, 418, 4988, 74, 4483, 12259, 281, 6194, 1029, 14, 468, 14692, 21708, 12822, 327, 1781, 15302, 342, 816, 247, 1643, 3104, 273, 2127, 432, 253, 418, 4988, 74, 6335, 15, 380, 5556, 5904, 275, 436, 6335, 3986, 2080, 4457, 752, 457, 84, 2130, 281, 12259, 1024, 13, 432, 625, 11132, 5556, 5904, 751, 40228, 21996, 281, 19554, 4394, 751, 8493, 33092, 7097, 15, 3900, 627, 310, 642, 1480, 5301, 281, 5899, 7274, 5393, 13, 418, 4988, 74, 13698, 281, 1056, 3733, 21708, 12822, 7938, 285, 625, 12482, 281, 247, 14200, 2491, 273, 12259, 15]}
9it [00:14, 1.59s/it]
评估第10个样本: {'question': 'Is there any support or community available to help me if I have questions or need assistance while using Lamini?', 'answer': 'Yes, there is a support community available to assist you with any questions or issues you may have while using Lamini. You can join the Lamini Discord server or reach out to the Lamini team directly for assistance.', 'input_ids': [2513, 627, 667, 1329, 390, 3114, 2130, 281, 1361, 479, 604, 309, 452, 3533, 390, 878, 8385, 1223, 970, 418, 4988, 74, 32, 4374, 13, 627, 310, 247, 1329, 3114, 2130, 281, 10073, 368, 342, 667, 3533, 390, 3374, 368, 778, 452, 1223, 970, 418, 4988, 74, 15, 1422, 476, 6604, 253, 418, 4988, 74, 15292, 636, 4771, 390, 3986, 562, 281, 253, 418, 4988, 74, 2285, 3587, 323, 8385, 15], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [2513, 627, 667, 1329, 390, 3114, 2130, 281, 1361, 479, 604, 309, 452, 3533, 390, 878, 8385, 1223, 970, 418, 4988, 74, 32, 4374, 13, 627, 310, 247, 1329, 3114, 2130, 281, 10073, 368, 342, 667, 3533, 390, 3374, 368, 778, 452, 1223, 970, 418, 4988, 74, 15, 1422, 476, 6604, 253, 418, 4988, 74, 15292, 636, 4771, 390, 3986, 562, 281, 253, 418, 4988, 74, 2285, 3587, 323, 8385, 15]}
10it [00:15, 1.58s/it]
评估第11个样本: {'question': 'Are there any code samples illustrating how to implement custom logging handlers?', 'answer': 'Yes, the Python logging module documentation provides several examples of how to implement custom logging handlers. You can find them in the official documentation here: https://docs.python.org/3/howto/logging-cookbook.html#developing-new-handlers', 'input_ids': [6723, 627, 667, 2127, 3530, 34805, 849, 281, 3359, 2840, 20893, 40093, 32, 4374, 13, 253, 13814, 20893, 6333, 10097, 3400, 2067, 6667, 273, 849, 281, 3359, 2840, 20893, 40093, 15, 1422, 476, 1089, 731, 275, 253, 3565, 10097, 1060, 27, 5987, 1358, 13880, 15, 16659, 15, 2061, 16, 20, 16, 5430, 936, 16, 36193, 14, 29519, 3305, 15, 2974, 4, 16714, 272, 14, 1826, 14, 4608, 10787], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [6723, 627, 667, 2127, 3530, 34805, 849, 281, 3359, 2840, 20893, 40093, 32, 4374, 13, 253, 13814, 20893, 6333, 10097, 3400, 2067, 6667, 273, 849, 281, 3359, 2840, 20893, 40093, 15, 1422, 476, 1089, 731, 275, 253, 3565, 10097, 1060, 27, 5987, 1358, 13880, 15, 16659, 15, 2061, 16, 20, 16, 5430, 936, 16, 36193, 14, 29519, 3305, 15, 2974, 4, 16714, 272, 14, 1826, 14, 4608, 10787]}
11it [00:17, 1.68s/it]
评估第12个样本: {'question': 'Are there any code samples illustrating how to handle authentication and authorization?', 'answer': 'Yes, there is a separate section in the documentation explaining authentication, for more information visit https://lamini-ai.github.io/auth/', 'input_ids': [6723, 627, 667, 2127, 3530, 34805, 849, 281, 6016, 19676, 285, 26239, 32, 4374, 13, 627, 310, 247, 4858, 2593, 275, 253, 10097, 15571, 19676, 13, 323, 625, 1491, 4143, 5987, 1358, 77, 4988, 74, 14, 2284, 15, 7280, 15, 900, 16, 14399, 16], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [6723, 627, 667, 2127, 3530, 34805, 849, 281, 6016, 19676, 285, 26239, 32, 4374, 13, 627, 310, 247, 4858, 2593, 275, 253, 10097, 15571, 19676, 13, 323, 625, 1491, 4143, 5987, 1358, 77, 4988, 74, 14, 2284, 15, 7280, 15, 900, 16, 14399, 16]}
11it [00:19, 1.76s/it]
完全匹配个数: 0
由于这些问题都是开放性问题，所以这里模型的回答和测试集的答案能够完全匹配的个数为0。但是我们可以把模型的答案和测试集的答案放一起进行比较，看看模型的表现如何
In [17]python · cell 25
python
df = pd.DataFrame(predictions, columns=["predicted_answer", "target_answer"])
for i in range(5):
print(f"Q{i+1}: predicted answer: {df['predicted_answer'][i]}")
print(f"Q{i+1}: target answer: {df['target_answer'][i]}\n")Output
Q1: predicted answer: Yes, Lamini can generate technical documentation or user manuals for software projects. This can be achieved by providing a prompt for a specific technical question or question to the LLM Engine, or by providing a prompt for a specific technical question or question. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini Q1: target answer: Yes, Lamini can generate technical documentation and user manuals for software projects. It uses natural language generation techniques to create clear and concise documentation that is easy to understand for both technical and non-technical users. This can save developers a significant amount of time and effort in creating documentation, allowing them to focus on other aspects of their projects. Q2: predicted answer: You can use the Authorization HTTP header to generate a response using the Authorization HTTP header. You can also use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. You can also use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Q2: target answer: The Authorization HTTP header should include the API key in the following format: Authorization: Bearer <YOUR-KEY-HERE>. Q3: predicted answer: Lamini’s LLM Engine is a LLM Engine for developers to use in their applications. It provides a base model for handling versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface Q3: target answer: Yes, the code includes a version parameter in the FeedbackOperation class constructor, which allows for handling versioning and compatibility. Q4: predicted answer: Yes, there is a community or support forum available for Lamini users. This is a great place to start! Lamini is a community of Lamini users, and we are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with Q4: target answer: Yes, there is a community forum available for Lamini users. The Lamini community forum can be accessed through the Lamini website and provides a platform for users to ask questions, share ideas, and collaborate with other developers using the library. Additionally, the Lamini team is active on the forum and provides support and guidance to users as needed. Q5: predicted answer: Yes, the Lamini library can be utilized for text completion or auto-completion tasks. This can be achieved through the use of a combination of techniques such as text completion, and the use of a combination of techniques such as text completion, to generate text-completion results. Additionally, the use of a combination of techniques such as text Q5: target answer: The Lamini library is not specifically designed for text completion or auto-completion tasks. However, it can be used for language modeling and generating text based on a given prompt.
2.5 完整测试集
以上的lamini_docs测试集只有12个样本，仅仅只是lamini_docs_evaluation测试集的一个子集，lamini 官方已经完成了在lamini_docs_evaluation完整测试集上的评估，我们可以下载下来看看：
In [18]python · cell 27
python
evaluation_dataset_path = "lamini/lamini_docs_evaluation"
evaluation_dataset = datasets.load_dataset(evaluation_dataset_path)Output
Found cached dataset parquet (C:/Users/Administrator/.cache/huggingface/datasets/lamini___parquet/lamini--lamini_docs_evaluation-998a25590076dcea/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
0%| | 0/1 [00:00<?, ?it/s]
In [19]python · cell 28
python
df2 = pd.DataFrame(evaluation_dataset)
for i in range(5):
print(f"Q{i+1}: predicted answer: {df2['train'][i]['predicted_answer']}")
print(f"Q{i+1}: target answer: {df2['train'][i]['target_answer']}\n")Output
Q1: predicted answer: Yes, Lamini can generate technical documentation or user manuals for software projects. This can be achieved by providing a prompt for a specific technical question or question to the LLM Engine, or by providing a prompt for a specific technical question or question. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini can be trained on specific technical questions or questions to help users understand the process and provide feedback to the LLM Engine. Additionally, Lamini Q1: target answer: Yes, Lamini can generate technical documentation and user manuals for software projects. It uses natural language generation techniques to create clear and concise documentation that is easy to understand for both technical and non-technical users. This can save developers a significant amount of time and effort in creating documentation, allowing them to focus on other aspects of their projects. Q2: predicted answer: You can use the Authorization HTTP header to generate a response using the Authorization HTTP header. You can also use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. You can also use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header to generate a response using the Authorization HTTP header. Additionally, you can use the Authorization HTTP header Q2: target answer: The Authorization HTTP header should include the API key in the following format: Authorization: Bearer <YOUR-KEY-HERE>. Q3: predicted answer: Lamini’s LLM Engine is a LLM Engine for developers to use in their applications. It provides a base model for handling versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility. Additionally, LLM Engine provides a built-in interface to help developers customize versioning and compatibility Q3: target answer: Yes, the code includes a version parameter in the FeedbackOperation class constructor, which allows for handling versioning and compatibility. Q4: predicted answer: Yes, there is a community or support forum available for Lamini users. This is a great place to start! Lamini is a community of Lamini users, and we are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to Lamini. We are here to help you with questions related to L Q4: target answer: Yes, there is a community forum available for Lamini users. The Lamini community forum can be accessed through the Lamini website and provides a platform for users to ask questions, share ideas, and collaborate with other developers using the library. Additionally, the Lamini team is active on the forum and provides support and guidance to users as needed. Q5: predicted answer: Yes, the Lamini library can be utilized for text completion or auto-completion tasks. This can be achieved through the use of a combination of techniques such as text completion, and the use of a combination of techniques such as text completion, to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, can be utilized to generate text-completion results. Additionally, the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well as the use of a combination of techniques such as text completion, as well Q5: target answer: The Lamini library is not specifically designed for text completion or auto-completion tasks. However, it can be used for language modeling and generating text based on a given prompt.
可以看到，在lamini_docs_evaluation测试集上，微调训练后的模型的表现还是不错的，基本都答对了。
2.6 总结
从以上的实验可以看出，由于聊天问答是一个较为开放性的任务，想要自动化评估微调模型的好坏还是较为困难的，大部分时候还是需要人工来辅助评估。
但也不是没有自动化评估的方法，比如将测试集的问题换成有确定答案的问题，或者换成选择题，再加上一些prompt技巧，还是可以自动化评估的，这个部分就留给读者自己探索了。