Chapter 01

第一章 -- 认识LLM

Notebookhands-on-llm15 cells

第一章 -- 认识LLM

In [ ]python · cell 2

python

# %%capture
# !pip install transformers>=4.40.1 accelerate>=0.27.2

In [24]python · cell 3

python

# 检查当前机器是可以使用 GPU，并且已经安装了正确的 CUDA 版本
!nvidia-smi

Output

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Fri Oct 11 07:56:04 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        On  | 00000000:A1:00.0 Off |                  Off |
| 30%   30C    P8              17W / 450W |   2854MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

1.1 Qwen/Qwen2.5-0.5B-Instruct

假定用户已经对机器学习（ML）和深度学习（DL）有一定的了解，第一部就是加载一个模型进行推理预测。可以对模型和 tokenizer 进行分别加载.

In [2]python · cell 5

python

import os
os.environ["HF_HOME"] = "/openbayes/home/huggingface"

In [3]python · cell 6

python

!echo $HF_HOME

Output

/openbayes/home/huggingface

In [4]python · cell 7

python

from transformers import AutoModelForCausalLM, AutoTokenizer


# 加载模型和 tokenizer （第一次加载的时候会进行下载）
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-0.5B-Instruct",
    device_map="cuda",
    torch_dtype="auto",
    trust_remote_code=True,
)
# 查看 模型结构是什么？
print(model)

Output

Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(151936, 896)
    (layers): ModuleList(
      (0-23): 24 x Qwen2DecoderLayer(
        (self_attn): Qwen2SdpaAttention(
          (q_proj): Linear(in_features=896, out_features=896, bias=True)
          (k_proj): Linear(in_features=896, out_features=128, bias=True)
          (v_proj): Linear(in_features=896, out_features=128, bias=True)
          (o_proj): Linear(in_features=896, out_features=896, bias=False)
          (rotary_emb): Qwen2RotaryEmbedding()
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
          (up_proj): Linear(in_features=896, out_features=4864, bias=False)
          (down_proj): Linear(in_features=4864, out_features=896, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): Qwen2RMSNorm()
        (post_attention_layernorm): Qwen2RMSNorm()
      )
    )
    (norm): Qwen2RMSNorm()
  )
  (lm_head): Linear(in_features=896, out_features=151936, bias=False)
)

In [20]python · cell 8

python

# 这里有一个 Special token, 打印看一下是什么token呢？
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
# tokenizer.all_special_tokens
tokenizer.special_tokens_map

Output

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

{'eos_token': '<|im_end|>',
 'pad_token': '<|endoftext|>',
 'additional_special_tokens': ['<|im_start|>',
  '<|im_end|>',
  '<|object_ref_start|>',
  '<|object_ref_end|>',
  '<|box_start|>',
  '<|box_end|>',
  '<|quad_start|>',
  '<|quad_end|>',
  '<|vision_start|>',
  '<|vision_end|>',
  '<|vision_pad|>',
  '<|image_pad|>',
  '<|video_pad|>']}

In [21]python · cell 9

python

# 每一个模型的 special token 都是不一样的（但是大同小异）
# tokenizer = AutoTokenizer.from_pretrained("/openbayes/input/input0")
# tokenizer.special_tokens_map

1.2 基础使用方式

加载模型之后可以怎么使用呢？可以直接使用原始的 model 和 tokenizer 进行推理；

step1: 加载模型
step2: 构建 prompt 和 tokenizer
step3: 推理和解码

刚刚已经加载过模型了，所以直接进行 step2

In [38]python · cell 11

python

prompt = "讲一个猫有关的笑话？"
messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
    {"role": "user", "content": prompt}
]

#  想一下诗变成什么格式
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

print(text)
print("====" * 10)
print(model_inputs)

Output

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
讲一个猫有关的笑话？<|im_end|>
<|im_start|>assistant

========================================
{'input_ids': tensor([[151644,   8948,    198,   2610,    525,   1207,  16948,     11,   3465,
            553,  54364,  14817,     13,   1446,    525,    264,  10950,  17847,
             13, 151645,    198, 151644,    872,    198,  99526,  46944, 100472,
         101063,   9370, 109959,  11319, 151645,    198, 151644,  77091,    198]],
       device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:0')}

In [41]python · cell 12

python

# step3: 推理和解码
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Output

有一只猫对一只小老鼠说：“你真聪明，能帮我找到妈妈！” 小老鼠想了想，回答道：“是啊，我叫‘小红帽’。” 这只猫听了哈哈大笑起来。

1.3 使用 transformers 的 pipeline 简化流程

In [44]python · cell 14

python

from transformers import pipeline


# step1: 生成 pipeline
# return_full_text=False 表示只返回新生成的文本，不包含输入的prompt
# return_full_text=True 则会返回完整文本，包含输入的prompt和生成的新文本
generator = pipeline(
    "text-generation", 
    model=model,
    tokenizer=tokenizer,
    return_full_text=False,  # 只返回新生成的文本部分
    max_new_tokens=500,
    do_sample=False
)

# step2: 构建 prompt
messages = [
    {"role": "user", "content": "写一个和猫有关的笑话."}
]

# step3，输出并解码
output = generator(messages)
print(output[0]["generated_text"])

Output

/output/envs/hands-on-llm/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:515: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.7` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/output/envs/hands-on-llm/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:520: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.8` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(
/output/envs/hands-on-llm/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:537: UserWarning: `do_sample` is set to `False`. However, `top_k` is set to `20` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_k`.
  warnings.warn(

好的，以下是一个关于猫的笑话：

有一天，一只猫在森林里迷路了。它四处张望，但什么也看不见。突然，它看到了一棵树上挂着一个牌子，上面写着“欢迎光临”。猫好奇地走过去，发现牌子后面有一个小洞。

猫小心翼翼地走进洞穴，里面有一只小老鼠正在吃着食物。猫对老鼠说：“你好，我是来自森林的小动物。”老鼠回答道：“我叫米奇，很高兴见到你。”

猫对米奇说：“我也很高兴遇见你，但我需要一些食物来养活自己。”米奇笑了笑，说：“那我就给你煮个饭吧，你尝尝看。”

于是，猫和米奇一起开始烹饪食物。猫用它的爪子扒开泥土，米奇则用他的牙齿咬碎坚果。他们一边吃一边聊，很快就度过了一个愉快的夜晚。

这个故事告诉我们，有时候，我们可能会遇到各种各样的人或事，但只要我们保持耐心、友好，并且乐于分享我们的经历，就有可能结识到新朋友。

In [49]python · cell 15

python

output

Output

[{'generated_text': '好的，以下是一个关于猫的笑话：\n\n有一天，一只猫在森林里迷路了。它四处张望，但什么也看不见。突然，它看到了一棵树上挂着一个牌子，上面写着“欢迎光临”。猫好奇地走过去，发现牌子后面有一个小洞。\n\n猫小心翼翼地走进洞穴，里面有一只小老鼠正在吃着食物。猫对老鼠说：“你好，我是来自森林的小动物。”老鼠回答道：“我叫米奇，很高兴见到你。”\n\n猫对米奇说：“我也很高兴遇见你，但我需要一些食物来养活自己。”米奇笑了笑，说：“那我就给你煮个饭吧，你尝尝看。”\n\n于是，猫和米奇一起开始烹饪食物。猫用它的爪子扒开泥土，米奇则用他的牙齿咬碎坚果。他们一边吃一边聊，很快就度过了一个愉快的夜晚。\n\n这个故事告诉我们，有时候，我们可能会遇到各种各样的人或事，但只要我们保持耐心、友好，并且乐于分享我们的经历，就有可能结识到新朋友。'}]

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Fri Oct 11 07:56:04 2024 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce RTX 4090 On | 00000000:A1:00.0 Off | Off | | 30% 30C P8 17W / 450W | 2854MiB / 24564MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| +---------------------------------------------------------------------------------------+

from transformers import AutoModelForCausalLM, AutoTokenizer # 加载模型和 tokenizer （第一次加载的时候会进行下载） model = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen2.5-0.5B-Instruct", device_map="cuda", torch_dtype="auto", trust_remote_code=True, ) # 查看模型结构是什么？ print(model)

Qwen2ForCausalLM( (model): Qwen2Model( (embed_tokens): Embedding(151936, 896) (layers): ModuleList( (0-23): 24 x Qwen2DecoderLayer( (self_attn): Qwen2SdpaAttention( (q_proj): Linear(in_features=896, out_features=896, bias=True) (k_proj): Linear(in_features=896, out_features=128, bias=True) (v_proj): Linear(in_features=896, out_features=128, bias=True) (o_proj): Linear(in_features=896, out_features=896, bias=False) (rotary_emb): Qwen2RotaryEmbedding() ) (mlp): Qwen2MLP( (gate_proj): Linear(in_features=896, out_features=4864, bias=False) (up_proj): Linear(in_features=896, out_features=4864, bias=False) (down_proj): Linear(in_features=4864, out_features=896, bias=False) (act_fn): SiLU() ) (input_layernorm): Qwen2RMSNorm() (post_attention_layernorm): Qwen2RMSNorm() ) ) (norm): Qwen2RMSNorm() ) (lm_head): Linear(in_features=896, out_features=151936, bias=False) )

# 这里有一个 Special token, 打印看一下是什么token呢？ tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct") # tokenizer.all_special_tokens tokenizer.special_tokens_map

{'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>', '<|object_ref_start|>', '<|object_ref_end|>', '<|box_start|>', '<|box_end|>', '<|quad_start|>', '<|quad_end|>', '<|vision_start|>', '<|vision_end|>', '<|vision_pad|>', '<|image_pad|>', '<|video_pad|>']}

prompt = "讲一个猫有关的笑话？" messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} ] # 想一下诗变成什么格式 text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) print(text) print("====" * 10) print(model_inputs)

<|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|> <|im_start|>user 讲一个猫有关的笑话？<|im_end|> <|im_start|>assistant ======================================== {'input_ids': tensor([[151644, 8948, 198, 2610, 525, 1207, 16948, 11, 3465, 553, 54364, 14817, 13, 1446, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 99526, 46944, 100472, 101063, 9370, 109959, 11319, 151645, 198, 151644, 77091, 198]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:0')}

# step3: 推理和解码 generated_ids = model.generate( **model_inputs, max_new_tokens=512 ) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids) ] response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] print(response)

from transformers import pipeline # step1: 生成 pipeline # return_full_text=False 表示只返回新生成的文本，不包含输入的prompt # return_full_text=True 则会返回完整文本，包含输入的prompt和生成的新文本 generator = pipeline( "text-generation", model=model, tokenizer=tokenizer, return_full_text=False, # 只返回新生成的文本部分 max_new_tokens=500, do_sample=False ) # step2: 构建 prompt messages = [ {"role": "user", "content": "写一个和猫有关的笑话."} ] # step3，输出并解码 output = generator(messages) print(output[0]["generated_text"])

/output/envs/hands-on-llm/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:515: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.7` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. warnings.warn( /output/envs/hands-on-llm/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:520: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.8` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. warnings.warn( /output/envs/hands-on-llm/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:537: UserWarning: `do_sample` is set to `False`. However, `top_k` is set to `20` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_k`. warnings.warn(

好的，以下是一个关于猫的笑话：有一天，一只猫在森林里迷路了。它四处张望，但什么也看不见。突然，它看到了一棵树上挂着一个牌子，上面写着“欢迎光临”。猫好奇地走过去，发现牌子后面有一个小洞。猫小心翼翼地走进洞穴，里面有一只小老鼠正在吃着食物。猫对老鼠说：“你好，我是来自森林的小动物。”老鼠回答道：“我叫米奇，很高兴见到你。” 猫对米奇说：“我也很高兴遇见你，但我需要一些食物来养活自己。”米奇笑了笑，说：“那我就给你煮个饭吧，你尝尝看。” 于是，猫和米奇一起开始烹饪食物。猫用它的爪子扒开泥土，米奇则用他的牙齿咬碎坚果。他们一边吃一边聊，很快就度过了一个愉快的夜晚。这个故事告诉我们，有时候，我们可能会遇到各种各样的人或事，但只要我们保持耐心、友好，并且乐于分享我们的经历，就有可能结识到新朋友。

[{'generated_text': '好的，以下是一个关于猫的笑话：\n\n有一天，一只猫在森林里迷路了。它四处张望，但什么也看不见。突然，它看到了一棵树上挂着一个牌子，上面写着“欢迎光临”。猫好奇地走过去，发现牌子后面有一个小洞。\n\n猫小心翼翼地走进洞穴，里面有一只小老鼠正在吃着食物。猫对老鼠说：“你好，我是来自森林的小动物。”老鼠回答道：“我叫米奇，很高兴见到你。”\n\n猫对米奇说：“我也很高兴遇见你，但我需要一些食物来养活自己。”米奇笑了笑，说：“那我就给你煮个饭吧，你尝尝看。”\n\n于是，猫和米奇一起开始烹饪食物。猫用它的爪子扒开泥土，米奇则用他的牙齿咬碎坚果。他们一边吃一边聊，很快就度过了一个愉快的夜晚。\n\n这个故事告诉我们，有时候，我们可能会遇到各种各样的人或事，但只要我们保持耐心、友好，并且乐于分享我们的经历，就有可能结识到新朋友。'}]