Chapter 02
Chapter 2 - Tokens and Embeddings
Notebookhands-on-llm35 cells
Chapter 2 - Tokens and Embeddings
In [ ]python · cell 2
python
# %%capture
# !pip install transformers>=4.41.2 sentence-transformers>=3.0.1 gensim>=4.3.2 scikit-learn>=1.5.0 accelerate>=0.31.02.1 下载一个开源模型运行看看
和前一章一样，分开加载并且分别解释不同部分的作用
In [4]python · cell 4
python
## step1: 加载模型
from transformers import AutoModelForCausalLM, AutoTokenizer
# 加载模型和 tokenizer
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-0.5B-Instruct",
device_map="cuda",
torch_dtype="auto",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")Output
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
解释原文中为什么要写一个 <|assistant|>？这是因为原文中用得是：microsoft/Phi-3-mini-4k-instruct，它用 <|assistant|> 表示 AI 回答开始。构成的格式如下：
code<|system|> You are a helpful assistant.<|end|> <|user|> How to explain Internet for a medieval knight?<|end|> <|assistant|>
python
# 原文。这里的 <|assistant|> 是为了让模型能
prompt = "帮我写一个请教条，原因是自己生病了。<|assistant|>"
# Tokenize the input prompt
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")Qwen 的模型和它有什么区别呢？
code
<|im_start|>user
讲一个猫有关的笑话？<|im_end|>
<|im_start|>assistantIn [13]python · cell 6
python
## 所以对于 Qwen 可以这么做？
### 中文
prompt = "帮我写一个请教条，原因是自己生病了。<|im_start|>assistant"
# Tokenize the input prompt
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")
# Generate the text
generation_output = model.generate(
input_ids=input_ids,
max_new_tokens=20
)
# Print the output
print(tokenizer.decode(generation_output[0]))Output
帮我写一个请教条，原因是自己生病了。<|im_start|>assistant：尊敬的老师、同学们： 您好！我最近因为身体不适而感到非常疲劳和难受
In [14]python · cell 7
python
print(input_ids)Output
tensor([[108965, 61443, 46944, 116069, 38989, 3837, 107711, 99283, 109281,
34187, 1773, 151644, 77091]], device='cuda:0')
In [17]python · cell 8
python
for id_ in input_ids[0]:
print(tokenizer.decode(id_))Output
帮我 写 一个 请教 条 ， 原因是 自己 生病 了 。 <|im_start|> assistant
In [20]python · cell 9
python
## 英文
prompt = "Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|>"
# Tokenize the input prompt
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")
# Generate the text
generation_output = model.generate(
input_ids=input_ids,
max_new_tokens=20
)
# Print the output
print(tokenizer.decode(generation_output[0]))Output
Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|> Dear Sarah, I hope this message finds you well. I wanted to apologize for the gardening mishap
In [22]python · cell 10
python
for id in input_ids[0]:
print(tokenizer.decode(id))Output
Write an email apolog izing to Sarah for the tragic gardening mish ap . Explain how it happened .< | assistant | >
2.2 对比训练过的 LLM tokenizers
In [24]python · cell 12
python
from transformers import AutoModelForCausalLM, AutoTokenizer
colors_list = [
'102;194;165', '252;141;98', '141;160;203',
'231;138;195', '166;216;84', '255;217;47'
]
def show_tokens(sentence, tokenizer_name):
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
token_ids = tokenizer(sentence).input_ids
for idx, t in enumerate(token_ids):
print(
f'\x1b[0;30;48;2;{colors_list[idx % len(colors_list)]}m' +
tokenizer.decode(t) +
'\x1b[0m',
end=' '
)
text = """
English and CAPITALIZATION
🎵 鸟
show_tokens False None elif == >= else: two tabs:" " Three tabs: " "
12.0*50=600
"""In [25]python · cell 13
python
show_tokens(text, "bert-base-uncased")Output
tokenizer_config.json: 0%| | 0.00/48.0 [00:00<?, ?B/s]
/output/envs/hands-on-llm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1142: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. warnings.warn(
config.json: 0%| | 0.00/570 [00:00<?, ?B/s]
vocab.txt: 0%| | 0.00/232k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/466k [00:00<?, ?B/s]
[0;30;48;2;102;194;165m[CLS][0m [0;30;48;2;252;141;98menglish[0m [0;30;48;2;141;160;203mand[0m [0;30;48;2;231;138;195mcapital[0m [0;30;48;2;166;216;84m##ization[0m [0;30;48;2;255;217;47m[UNK][0m [0;30;48;2;102;194;165m[UNK][0m [0;30;48;2;252;141;98mshow[0m [0;30;48;2;141;160;203m_[0m [0;30;48;2;231;138;195mtoken[0m [0;30;48;2;166;216;84m##s[0m [0;30;48;2;255;217;47mfalse[0m [0;30;48;2;102;194;165mnone[0m [0;30;48;2;252;141;98meli[0m [0;30;48;2;141;160;203m##f[0m [0;30;48;2;231;138;195m=[0m [0;30;48;2;166;216;84m=[0m [0;30;48;2;255;217;47m>[0m [0;30;48;2;102;194;165m=[0m [0;30;48;2;252;141;98melse[0m [0;30;48;2;141;160;203m:[0m [0;30;48;2;231;138;195mtwo[0m [0;30;48;2;166;216;84mtab[0m [0;30;48;2;255;217;47m##s[0m [0;30;48;2;102;194;165m:[0m [0;30;48;2;252;141;98m"[0m [0;30;48;2;141;160;203m"[0m [0;30;48;2;231;138;195mthree[0m [0;30;48;2;166;216;84mtab[0m [0;30;48;2;255;217;47m##s[0m [0;30;48;2;102;194;165m:[0m [0;30;48;2;252;141;98m"[0m [0;30;48;2;141;160;203m"[0m [0;30;48;2;231;138;195m12[0m [0;30;48;2;166;216;84m.[0m [0;30;48;2;255;217;47m0[0m [0;30;48;2;102;194;165m*[0m [0;30;48;2;252;141;98m50[0m [0;30;48;2;141;160;203m=[0m [0;30;48;2;231;138;195m600[0m [0;30;48;2;166;216;84m[SEP][0m
In [26]python · cell 14
python
show_tokens(text, "gpt2")Output
tokenizer_config.json: 0%| | 0.00/26.0 [00:00<?, ?B/s]
config.json: 0%| | 0.00/665 [00:00<?, ?B/s]
vocab.json: 0%| | 0.00/1.04M [00:00<?, ?B/s]
merges.txt: 0%| | 0.00/456k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/1.36M [00:00<?, ?B/s]
[0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98mEnglish[0m [0;30;48;2;141;160;203m and[0m [0;30;48;2;231;138;195m CAP[0m [0;30;48;2;166;216;84mITAL[0m [0;30;48;2;255;217;47mIZ[0m [0;30;48;2;102;194;165mATION[0m [0;30;48;2;252;141;98m [0m [0;30;48;2;141;160;203m�[0m [0;30;48;2;231;138;195m�[0m [0;30;48;2;166;216;84m�[0m [0;30;48;2;255;217;47m �[0m [0;30;48;2;102;194;165m�[0m [0;30;48;2;252;141;98m�[0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195mshow[0m [0;30;48;2;166;216;84m_[0m [0;30;48;2;255;217;47mt[0m [0;30;48;2;102;194;165mok[0m [0;30;48;2;252;141;98mens[0m [0;30;48;2;141;160;203m False[0m [0;30;48;2;231;138;195m None[0m [0;30;48;2;166;216;84m el[0m [0;30;48;2;255;217;47mif[0m [0;30;48;2;102;194;165m ==[0m [0;30;48;2;252;141;98m >=[0m [0;30;48;2;141;160;203m else[0m [0;30;48;2;231;138;195m:[0m [0;30;48;2;166;216;84m two[0m [0;30;48;2;255;217;47m tabs[0m [0;30;48;2;102;194;165m:"[0m [0;30;48;2;252;141;98m [0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195m [0m [0;30;48;2;166;216;84m "[0m [0;30;48;2;255;217;47m Three[0m [0;30;48;2;102;194;165m tabs[0m [0;30;48;2;252;141;98m:[0m [0;30;48;2;141;160;203m "[0m [0;30;48;2;231;138;195m [0m [0;30;48;2;166;216;84m [0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98m [0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195m "[0m [0;30;48;2;166;216;84m [0m [0;30;48;2;255;217;47m12[0m [0;30;48;2;102;194;165m.[0m [0;30;48;2;252;141;98m0[0m [0;30;48;2;141;160;203m*[0m [0;30;48;2;231;138;195m50[0m [0;30;48;2;166;216;84m=[0m [0;30;48;2;255;217;47m600[0m [0;30;48;2;102;194;165m [0m
In [27]python · cell 15
python
show_tokens(text, "Qwen/Qwen2.5-0.5B-Instruct")Output
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98mEnglish[0m [0;30;48;2;141;160;203m and[0m [0;30;48;2;231;138;195m CAPITAL[0m [0;30;48;2;166;216;84mIZATION[0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m🎵[0m [0;30;48;2;252;141;98m �[0m [0;30;48;2;141;160;203m�[0m [0;30;48;2;231;138;195m�[0m [0;30;48;2;166;216;84m [0m [0;30;48;2;255;217;47mshow[0m [0;30;48;2;102;194;165m_tokens[0m [0;30;48;2;252;141;98m False[0m [0;30;48;2;141;160;203m None[0m [0;30;48;2;231;138;195m elif[0m [0;30;48;2;166;216;84m ==[0m [0;30;48;2;255;217;47m >=[0m [0;30;48;2;102;194;165m else[0m [0;30;48;2;252;141;98m:[0m [0;30;48;2;141;160;203m two[0m [0;30;48;2;231;138;195m tabs[0m [0;30;48;2;166;216;84m:"[0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m "[0m [0;30;48;2;252;141;98m Three[0m [0;30;48;2;141;160;203m tabs[0m [0;30;48;2;231;138;195m:[0m [0;30;48;2;166;216;84m "[0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m " [0m [0;30;48;2;252;141;98m1[0m [0;30;48;2;141;160;203m2[0m [0;30;48;2;231;138;195m.[0m [0;30;48;2;166;216;84m0[0m [0;30;48;2;255;217;47m*[0m [0;30;48;2;102;194;165m5[0m [0;30;48;2;252;141;98m0[0m [0;30;48;2;141;160;203m=[0m [0;30;48;2;231;138;195m6[0m [0;30;48;2;166;216;84m0[0m [0;30;48;2;255;217;47m0[0m [0;30;48;2;102;194;165m [0m
2.3 理解 Embedding
2.3.1 原始 word2vec embedding （和 LLM 无关）
In [34]python · cell 18
python
import gensim.downloader as api
# Download embeddings (66MB, glove, trained on wikipedia, vector size: 50)
# Other options include "word2vec-google-news-300"
# More options at https://github.com/RaRe-Technologies/gensim-data
model = api.load("glove-wiki-gigaword-50")
model.most_similar([model['king']], topn=11)Output
[=================---------------------------------] 35.5% 23.4/66.0MB downloaded
IOPub message rate exceeded. The Jupyter server will temporarily stop sending output to the client in order to avoid crashing it. To change this limit, set the config variable `--ServerApp.iopub_msg_rate_limit`. Current values: ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec) ServerApp.rate_limit_window=3.0 (secs)
[==================================================] 100.0% 66.0/66.0MB downloaded
[('king', 1.0000001192092896),
('prince', 0.8236179351806641),
('queen', 0.7839043140411377),
('ii', 0.7746230363845825),
('emperor', 0.7736247777938843),
('son', 0.766719400882721),
('uncle', 0.7627150416374207),
('kingdom', 0.7542161345481873),
('throne', 0.7539914846420288),
('brother', 0.7492411136627197),
('ruler', 0.7434253692626953)]2.3.2 上下文相关的 Embedding
Contextualized Word Embeddings From a Language Model (Like BERT)，最早应该是 ElMO， 然后是 bert
In [28]python · cell 20
python
from transformers import AutoModel, AutoTokenizer
# Load a tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# Load a language model
model = AutoModel.from_pretrained("bert-base-uncased")
# Tokenize the sentence
tokens = tokenizer('Hello world', return_tensors='pt')
# Process the tokens
output = model(**tokens)[0]Output
model.safetensors: 0%| | 0.00/440M [00:00<?, ?B/s]
In [31]python · cell 21
python
outputOutput
tensor([[[-0.1689, 0.1361, -0.1394, ..., -0.6251, 0.0522, 0.3671],
[-0.3633, 0.1412, 0.8800, ..., 0.1043, 0.2888, 0.3727],
[-0.6986, -0.6988, 0.0645, ..., -0.2210, 0.0099, -0.5940],
[ 0.8310, 0.1237, -0.1512, ..., 0.1031, -0.6779, -0.2629]]],
grad_fn=<NativeLayerNormBackward0>)In [29]python · cell 22
python
output.shapeOutput
torch.Size([1, 4, 768])
In [30]python · cell 23
python
for token in tokens['input_ids'][0]:
print(tokenizer.decode(token))Output
[CLS] hello world [SEP]
2.3.3 文本向量（句向量）
Text Embeddings (For Sentences and Whole Documents)
In [32]python · cell 25
python
from sentence_transformers import SentenceTransformer
# Load model
model = SentenceTransformer('sentence-transformers/all-distilroberta-v1')
# Convert text to text embeddings
vector = model.encode("测试一个小模型的 embedding 能力")Output
modules.json: 0%| | 0.00/349 [00:00<?, ?B/s]
config_sentence_transformers.json: 0%| | 0.00/116 [00:00<?, ?B/s]
README.md: 0%| | 0.00/10.3k [00:00<?, ?B/s]
sentence_bert_config.json: 0%| | 0.00/53.0 [00:00<?, ?B/s]
/output/envs/hands-on-llm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1142: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. warnings.warn(
config.json: 0%| | 0.00/653 [00:00<?, ?B/s]
model.safetensors: 0%| | 0.00/328M [00:00<?, ?B/s]
tokenizer_config.json: 0%| | 0.00/333 [00:00<?, ?B/s]
vocab.json: 0%| | 0.00/798k [00:00<?, ?B/s]
merges.txt: 0%| | 0.00/456k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/1.36M [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/239 [00:00<?, ?B/s]
1_Pooling/config.json: 0%| | 0.00/190 [00:00<?, ?B/s]
In [33]python · cell 26
python
vector.shapeOutput
(768,)
2.3.4 基于 embeddings的歌曲推理
In [38]python · cell 28
python
import pandas as pd
from urllib import request
# Get the playlist dataset file
data = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/train.txt')
# Parse the playlist dataset file. Skip the first two lines as
# they only contain metadata
lines = data.read().decode("utf-8").split('\n')[2:]
# Remove playlists with only one song
playlists = [s.rstrip().split() for s in lines if len(s.split()) > 1]
# Load song metadata
songs_file = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/song_hash.txt')
songs_file = songs_file.read().decode("utf-8").split('\n')
songs = [s.rstrip().split('\t') for s in songs_file]
songs_df = pd.DataFrame(data=songs, columns = ['id', 'title', 'artist'])
songs_df = songs_df.set_index('id')In [39]python · cell 29
python
print( 'Playlist #1:\n ', playlists[0], '\n')
print( 'Playlist #2:\n ', playlists[1])Output
Playlist #1: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '2', '42', '43', '44', '45', '46', '47', '48', '20', '49', '8', '50', '51', '52', '53', '54', '55', '56', '57', '25', '58', '59', '60', '61', '62', '3', '63', '64', '65', '66', '46', '47', '67', '2', '48', '68', '69', '70', '57', '50', '71', '72', '53', '73', '25', '74', '59', '20', '46', '75', '76', '77', '59', '20', '43'] Playlist #2: ['78', '79', '80', '3', '62', '81', '14', '82', '48', '83', '84', '17', '85', '86', '87', '88', '74', '89', '90', '91', '4', '73', '62', '92', '17', '53', '59', '93', '94', '51', '50', '27', '95', '48', '96', '97', '98', '99', '100', '57', '101', '102', '25', '103', '3', '104', '105', '106', '107', '47', '108', '109', '110', '111', '112', '113', '25', '63', '62', '114', '115', '84', '116', '117', '118', '119', '120', '121', '122', '123', '50', '70', '71', '124', '17', '85', '14', '82', '48', '125', '47', '46', '72', '53', '25', '73', '4', '126', '59', '74', '20', '43', '127', '128', '129', '13', '82', '48', '130', '131', '132', '133', '134', '135', '136', '137', '59', '46', '138', '43', '20', '139', '140', '73', '57', '70', '141', '3', '1', '74', '142', '143', '144', '145', '48', '13', '25', '146', '50', '147', '126', '59', '20', '148', '149', '150', '151', '152', '56', '153', '154', '155', '156', '157', '158', '159', '160', '161', '162', '163', '164', '165', '166', '167', '168', '169', '170', '171', '172', '173', '174', '175', '60', '176', '51', '177', '178', '179', '180', '181', '182', '183', '184', '185', '57', '186', '187', '188', '189', '190', '191', '46', '192', '193', '194', '195', '196', '197', '198', '25', '199', '200', '49', '201', '100', '202', '203', '204', '205', '206', '207', '32', '208', '209', '210']
In [40]python · cell 30
python
from gensim.models import Word2Vec
# Train our Word2Vec model
model = Word2Vec(
playlists, vector_size=32, window=20, negative=50, min_count=1, workers=4
)In [41]python · cell 31
python
song_id = 2172
# Ask the model for songs similar to song #2172
model.wv.most_similar(positive=str(song_id))Output
[('3167', 0.9990279674530029),
('3094', 0.9980515241622925),
('2976', 0.9978846907615662),
('10084', 0.9973754286766052),
('2704', 0.9972402453422546),
('6624', 0.9970847368240356),
('5586', 0.9969120621681213),
('2640', 0.9966724514961243),
('6658', 0.9966153502464294),
('2849', 0.9962793588638306)]In [42]python · cell 32
python
print(songs_df.iloc[2172])Output
title Fade To Black artist Metallica Name: 2172 , dtype: object
In [43]python · cell 33
python
import numpy as np
def print_recommendations(song_id):
similar_songs = np.array(
model.wv.most_similar(positive=str(song_id),topn=5)
)[:,0]
return songs_df.iloc[similar_songs]
# Extract recommendations
print_recommendations(2172)Output
title artist id 3167 Unchained Van Halen 3094 Breaking The Law Judas Priest 2976 I Don't Know Ozzy Osbourne 10084 Detroit Rock City Kiss 2704 Over The Mountain Ozzy Osbourne
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| title | artist | |
|---|---|---|
| id | ||
| 3167 | Unchained | Van Halen |
| 3094 | Breaking The Law | Judas Priest |
| 2976 | I Don't Know | Ozzy Osbourne |
| 10084 | Detroit Rock City | Kiss |
| 2704 | Over The Mountain | Ozzy Osbourne |
In [47]python · cell 34
python
print(songs_df.iloc[842])
print_recommendations(842)Output
title California Love (w\/ Dr. Dre & Roger Troutman) artist 2Pac Name: 842 , dtype: object
title artist id 330 Hate It Or Love It (w\/ 50 Cent) The Game 5788 Drop It Like It's Hot (w\/ Pharrell) Snoop Dogg 886 Heartless Kanye West 413 If I Ruled The World (Imagine That) (w\/ Laury... Nas 5668 How We Do (w\/ 50 Cent) The Game
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| title | artist | |
|---|---|---|
| id | ||
| 330 | Hate It Or Love It (w\/ 50 Cent) | The Game |
| 5788 | Drop It Like It's Hot (w\/ Pharrell) | Snoop Dogg |
| 886 | Heartless | Kanye West |
| 413 | If I Ruled The World (Imagine That) (w\/ Laury... | Nas |
| 5668 | How We Do (w\/ 50 Cent) | The Game |
In [ ]python · cell 35
python