Chapter 33
chatglm3 infer
Notebooktransformers12 cells
In [ ]python · cell 1
python
from transformers import AutoTokenizer, AutoModelIn [ ]python · cell 2
python
tokenizer = AutoTokenizer.from_pretrained("d:/Pretrained_models/ZhipuAI/chatglm3-6b/", trust_remote_code=True)
tokenizerIn [ ]python · cell 3
python
import torch
"""
新版本中需要将modeling_chatglm源码中的613行部分进行调整，代码如下：
```
if not kv_caches:
kv_caches = [None for _ in range(self.num_layers)]
else:
kv_caches = kv_caches[1]
```
如果不进行调整，后续chat阶段会报错
"""
model = AutoModel.from_pretrained("d:/Pretrained_models/ZhipuAI/chatglm3-6b/", trust_remote_code=True, low_cpu_mem_usage=True, torch_dtype=torch.half, device_map="auto")In [ ]python · cell 4
python
model.chat(tokenizer, "考试的技巧有哪些？", history=[])In [ ]python · cell 5
python
?model.chatIn [ ]python · cell 6
python
?tokenizer.build_chat_inputIn [ ]python · cell 7
python
tokenizer("<|user|>", add_special_tokens=False)In [ ]python · cell 8
python
tokenizer.get_command("<|user|>")In [ ]python · cell 9
python
tokenizer.build_chat_input("考试的技巧有哪些？", history=[], role="user")In [ ]python · cell 10
python
tokenizer.decode([64790, 64792, 64795, 30910, 13, 30910, 32227, 54530, 33741, 34953,
31514, 64796])数据格式：
[gMASK]sop<|user|> \n Prompt<|assistant|> \n Response eos_token