Chapter 02
Chapter 2 Tokens and Token Embeddings
NotebookPython 3 (ipykernel)41 cells
This notebook is for Chapter 2 of the Hands-On Large Language Models book by Jay Alammar and Maarten Grootendorst.
[OPTIONAL] - Installing Packages on
If you are viewing this notebook on Google Colab (or any other cloud vendor), you need to uncomment and run the following codeblock to install the dependencies for this chapter:
💡 NOTE: We will want to use a GPU to run the examples in this notebook. In Google Colab, go to Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4.
In [ ]python · cell 3
python
# %%capture
# !pip install --upgrade transformers==4.41.2 sentence-transformers==3.0.1 gensim==4.3.2 scikit-learn==1.5.0 accelerate==0.31.0 peft==0.11.1 scipy==1.10.1 numpy==1.26.4Downloading and Running An LLM
The first step is to load our model onto the GPU for faster inference. Note that we load the model and tokenizer separately and keep them as such so that we can explore them separately.
In [ ]python · cell 5
python
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
"microsoft/Phi-3-mini-4k-instruct",
device_map="cuda",
torch_dtype="auto",
trust_remote_code=False,
)
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")Output
`flash-attention` package not found, consider installing for better performance: No module named 'flash_attn'. Current `flash-attention` does not support `window_size`. Either upgrade or use `attn_implementation='eager'`.
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
In [ ]python · cell 6
python
prompt = "Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|>"
# Tokenize the input prompt
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")
# Generate the text
generation_output = model.generate(
input_ids=input_ids,
max_new_tokens=20
)
# Print the output
print(tokenizer.decode(generation_output[0]))Output
WARNING:transformers_modules.microsoft.Phi-3-mini-4k-instruct.ff07dc01615f8113924aed013115ab2abd32115b.modeling_phi3:You are not running the flash-attention implementation, expect numerical differences.
<s> Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|> Subject: My Sincere Apologies for the Gardening Mishap Dear
In [ ]python · cell 7
python
print(input_ids)Output
tensor([[ 1, 14350, 385, 4876, 27746, 5281, 304, 19235, 363, 278,
25305, 293, 16423, 292, 286, 728, 481, 29889, 12027, 7420,
920, 372, 9559, 29889, 32001]], device='cuda:0')
In [ ]python · cell 8
python
for id in input_ids[0]:
print(tokenizer.decode(id))Output
<s> Write an email apolog izing to Sarah for the trag ic garden ing m ish ap . Exp lain how it happened . <|assistant|>
In [ ]python · cell 9
python
generation_outputOutput
tensor([[ 1, 14350, 385, 4876, 27746, 5281, 304, 19235, 363, 278,
25305, 293, 16423, 292, 286, 728, 481, 29889, 12027, 7420,
920, 372, 9559, 29889, 32001, 3323, 622, 29901, 1619, 317,
3742, 406, 6225, 11763, 363, 278, 19906, 292, 341, 728,
481, 13, 13, 29928, 799]], device='cuda:0')In [ ]python · cell 10
python
print(tokenizer.decode(3323))
print(tokenizer.decode(622))
print(tokenizer.decode([3323, 622]))
print(tokenizer.decode(29901))Output
Sub ject Subject :
Comparing Trained LLM Tokenizers
In [ ]python · cell 12
python
from transformers import AutoModelForCausalLM, AutoTokenizer
colors_list = [
'102;194;165', '252;141;98', '141;160;203',
'231;138;195', '166;216;84', '255;217;47'
]
def show_tokens(sentence, tokenizer_name):
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
token_ids = tokenizer(sentence).input_ids
for idx, t in enumerate(token_ids):
print(
f'\x1b[0;30;48;2;{colors_list[idx % len(colors_list)]}m' +
tokenizer.decode(t) +
'\x1b[0m',
end=' '
)In [ ]python · cell 13
python
text = """
English and CAPITALIZATION
🎵 鸟
show_tokens False None elif == >= else: two tabs:" " Three tabs: " "
12.0*50=600
"""In [ ]python · cell 14
python
show_tokens(text, "bert-base-uncased")Output
[0;30;48;2;102;194;165m[CLS][0m [0;30;48;2;252;141;98menglish[0m [0;30;48;2;141;160;203mand[0m [0;30;48;2;231;138;195mcapital[0m [0;30;48;2;166;216;84m##ization[0m [0;30;48;2;255;217;47m[UNK][0m [0;30;48;2;102;194;165m[UNK][0m [0;30;48;2;252;141;98mshow[0m [0;30;48;2;141;160;203m_[0m [0;30;48;2;231;138;195mtoken[0m [0;30;48;2;166;216;84m##s[0m [0;30;48;2;255;217;47mfalse[0m [0;30;48;2;102;194;165mnone[0m [0;30;48;2;252;141;98meli[0m [0;30;48;2;141;160;203m##f[0m [0;30;48;2;231;138;195m=[0m [0;30;48;2;166;216;84m=[0m [0;30;48;2;255;217;47m>[0m [0;30;48;2;102;194;165m=[0m [0;30;48;2;252;141;98melse[0m [0;30;48;2;141;160;203m:[0m [0;30;48;2;231;138;195mtwo[0m [0;30;48;2;166;216;84mtab[0m [0;30;48;2;255;217;47m##s[0m [0;30;48;2;102;194;165m:[0m [0;30;48;2;252;141;98m"[0m [0;30;48;2;141;160;203m/[0m [0;30;48;2;231;138;195mt[0m [0;30;48;2;166;216;84m/[0m [0;30;48;2;255;217;47mt[0m [0;30;48;2;102;194;165m"[0m [0;30;48;2;252;141;98mthree[0m [0;30;48;2;141;160;203mtab[0m [0;30;48;2;231;138;195m##s[0m [0;30;48;2;166;216;84m:[0m [0;30;48;2;255;217;47m"[0m [0;30;48;2;102;194;165m"[0m [0;30;48;2;252;141;98m12[0m [0;30;48;2;141;160;203m.[0m [0;30;48;2;231;138;195m0[0m [0;30;48;2;166;216;84m*[0m [0;30;48;2;255;217;47m50[0m [0;30;48;2;102;194;165m=[0m [0;30;48;2;252;141;98m600[0m [0;30;48;2;141;160;203m[SEP][0m
In [ ]python · cell 15
python
show_tokens(text, "bert-base-cased")Output
tokenizer_config.json: 0%| | 0.00/49.0 [00:00<?, ?B/s]
/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. warnings.warn(
config.json: 0%| | 0.00/570 [00:00<?, ?B/s]
vocab.txt: 0%| | 0.00/213k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/436k [00:00<?, ?B/s]
[0;30;48;2;102;194;165m[CLS][0m [0;30;48;2;252;141;98mEnglish[0m [0;30;48;2;141;160;203mand[0m [0;30;48;2;231;138;195mCA[0m [0;30;48;2;166;216;84m##PI[0m [0;30;48;2;255;217;47m##TA[0m [0;30;48;2;102;194;165m##L[0m [0;30;48;2;252;141;98m##I[0m [0;30;48;2;141;160;203m##Z[0m [0;30;48;2;231;138;195m##AT[0m [0;30;48;2;166;216;84m##ION[0m [0;30;48;2;255;217;47m[UNK][0m [0;30;48;2;102;194;165m[UNK][0m [0;30;48;2;252;141;98mshow[0m [0;30;48;2;141;160;203m_[0m [0;30;48;2;231;138;195mtoken[0m [0;30;48;2;166;216;84m##s[0m [0;30;48;2;255;217;47mF[0m [0;30;48;2;102;194;165m##als[0m [0;30;48;2;252;141;98m##e[0m [0;30;48;2;141;160;203mNone[0m [0;30;48;2;231;138;195mel[0m [0;30;48;2;166;216;84m##if[0m [0;30;48;2;255;217;47m=[0m [0;30;48;2;102;194;165m=[0m [0;30;48;2;252;141;98m>[0m [0;30;48;2;141;160;203m=[0m [0;30;48;2;231;138;195melse[0m [0;30;48;2;166;216;84m:[0m [0;30;48;2;255;217;47mtwo[0m [0;30;48;2;102;194;165mta[0m [0;30;48;2;252;141;98m##bs[0m [0;30;48;2;141;160;203m:[0m [0;30;48;2;231;138;195m"[0m [0;30;48;2;166;216;84m"[0m [0;30;48;2;255;217;47mThree[0m [0;30;48;2;102;194;165mta[0m [0;30;48;2;252;141;98m##bs[0m [0;30;48;2;141;160;203m:[0m [0;30;48;2;231;138;195m"[0m [0;30;48;2;166;216;84m"[0m [0;30;48;2;255;217;47m12[0m [0;30;48;2;102;194;165m.[0m [0;30;48;2;252;141;98m0[0m [0;30;48;2;141;160;203m*[0m [0;30;48;2;231;138;195m50[0m [0;30;48;2;166;216;84m=[0m [0;30;48;2;255;217;47m600[0m [0;30;48;2;102;194;165m[SEP][0m
In [ ]python · cell 16
python
show_tokens(text, "gpt2")Output
tokenizer_config.json: 0%| | 0.00/26.0 [00:00<?, ?B/s]
config.json: 0%| | 0.00/665 [00:00<?, ?B/s]
vocab.json: 0%| | 0.00/1.04M [00:00<?, ?B/s]
merges.txt: 0%| | 0.00/456k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/1.36M [00:00<?, ?B/s]
[0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98mEnglish[0m [0;30;48;2;141;160;203m and[0m [0;30;48;2;231;138;195m CAP[0m [0;30;48;2;166;216;84mITAL[0m [0;30;48;2;255;217;47mIZ[0m [0;30;48;2;102;194;165mATION[0m [0;30;48;2;252;141;98m [0m [0;30;48;2;141;160;203m�[0m [0;30;48;2;231;138;195m�[0m [0;30;48;2;166;216;84m�[0m [0;30;48;2;255;217;47m �[0m [0;30;48;2;102;194;165m�[0m [0;30;48;2;252;141;98m�[0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195mshow[0m [0;30;48;2;166;216;84m_[0m [0;30;48;2;255;217;47mt[0m [0;30;48;2;102;194;165mok[0m [0;30;48;2;252;141;98mens[0m [0;30;48;2;141;160;203m False[0m [0;30;48;2;231;138;195m None[0m [0;30;48;2;166;216;84m el[0m [0;30;48;2;255;217;47mif[0m [0;30;48;2;102;194;165m ==[0m [0;30;48;2;252;141;98m >=[0m [0;30;48;2;141;160;203m else[0m [0;30;48;2;231;138;195m:[0m [0;30;48;2;166;216;84m two[0m [0;30;48;2;255;217;47m tabs[0m [0;30;48;2;102;194;165m:"[0m [0;30;48;2;252;141;98m [0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195m [0m [0;30;48;2;166;216;84m "[0m [0;30;48;2;255;217;47m Three[0m [0;30;48;2;102;194;165m tabs[0m [0;30;48;2;252;141;98m:[0m [0;30;48;2;141;160;203m "[0m [0;30;48;2;231;138;195m [0m [0;30;48;2;166;216;84m [0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98m [0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195m "[0m [0;30;48;2;166;216;84m [0m [0;30;48;2;255;217;47m12[0m [0;30;48;2;102;194;165m.[0m [0;30;48;2;252;141;98m0[0m [0;30;48;2;141;160;203m*[0m [0;30;48;2;231;138;195m50[0m [0;30;48;2;166;216;84m=[0m [0;30;48;2;255;217;47m600[0m [0;30;48;2;102;194;165m [0m
In [ ]python · cell 17
python
show_tokens(text, "google/flan-t5-small")Output
tokenizer_config.json: 0%| | 0.00/2.54k [00:00<?, ?B/s]
spiece.model: 0%| | 0.00/792k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/2.42M [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/2.20k [00:00<?, ?B/s]
[0;30;48;2;102;194;165mEnglish[0m [0;30;48;2;252;141;98mand[0m [0;30;48;2;141;160;203mCA[0m [0;30;48;2;231;138;195mPI[0m [0;30;48;2;166;216;84mTAL[0m [0;30;48;2;255;217;47mIZ[0m [0;30;48;2;102;194;165mATION[0m [0;30;48;2;252;141;98m[0m [0;30;48;2;141;160;203m<unk>[0m [0;30;48;2;231;138;195m[0m [0;30;48;2;166;216;84m<unk>[0m [0;30;48;2;255;217;47mshow[0m [0;30;48;2;102;194;165m_[0m [0;30;48;2;252;141;98mto[0m [0;30;48;2;141;160;203mken[0m [0;30;48;2;231;138;195ms[0m [0;30;48;2;166;216;84mFal[0m [0;30;48;2;255;217;47ms[0m [0;30;48;2;102;194;165me[0m [0;30;48;2;252;141;98mNone[0m [0;30;48;2;141;160;203m[0m [0;30;48;2;231;138;195me[0m [0;30;48;2;166;216;84ml[0m [0;30;48;2;255;217;47mif[0m [0;30;48;2;102;194;165m=[0m [0;30;48;2;252;141;98m=[0m [0;30;48;2;141;160;203m>[0m [0;30;48;2;231;138;195m=[0m [0;30;48;2;166;216;84melse[0m [0;30;48;2;255;217;47m:[0m [0;30;48;2;102;194;165mtwo[0m [0;30;48;2;252;141;98mtab[0m [0;30;48;2;141;160;203ms[0m [0;30;48;2;231;138;195m:[0m [0;30;48;2;166;216;84m"[0m [0;30;48;2;255;217;47m"[0m [0;30;48;2;102;194;165mThree[0m [0;30;48;2;252;141;98mtab[0m [0;30;48;2;141;160;203ms[0m [0;30;48;2;231;138;195m:[0m [0;30;48;2;166;216;84m"[0m [0;30;48;2;255;217;47m"[0m [0;30;48;2;102;194;165m12.[0m [0;30;48;2;252;141;98m0[0m [0;30;48;2;141;160;203m*[0m [0;30;48;2;231;138;195m50[0m [0;30;48;2;166;216;84m=[0m [0;30;48;2;255;217;47m600[0m [0;30;48;2;102;194;165m[0m [0;30;48;2;252;141;98m</s>[0m
In [ ]python · cell 18
python
# The official is `tiktoken` but this the same tokenizer on the HF platform
show_tokens(text, "Xenova/gpt-4")Output
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98mEnglish[0m [0;30;48;2;141;160;203m and[0m [0;30;48;2;231;138;195m CAPITAL[0m [0;30;48;2;166;216;84mIZATION[0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m�[0m [0;30;48;2;252;141;98m�[0m [0;30;48;2;141;160;203m�[0m [0;30;48;2;231;138;195m �[0m [0;30;48;2;166;216;84m�[0m [0;30;48;2;255;217;47m�[0m [0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98mshow[0m [0;30;48;2;141;160;203m_tokens[0m [0;30;48;2;231;138;195m False[0m [0;30;48;2;166;216;84m None[0m [0;30;48;2;255;217;47m elif[0m [0;30;48;2;102;194;165m ==[0m [0;30;48;2;252;141;98m >=[0m [0;30;48;2;141;160;203m else[0m [0;30;48;2;231;138;195m:[0m [0;30;48;2;166;216;84m two[0m [0;30;48;2;255;217;47m tabs[0m [0;30;48;2;102;194;165m:"[0m [0;30;48;2;252;141;98m [0m [0;30;48;2;141;160;203m "[0m [0;30;48;2;231;138;195m Three[0m [0;30;48;2;166;216;84m tabs[0m [0;30;48;2;255;217;47m:[0m [0;30;48;2;102;194;165m "[0m [0;30;48;2;252;141;98m [0m [0;30;48;2;141;160;203m " [0m [0;30;48;2;231;138;195m12[0m [0;30;48;2;166;216;84m.[0m [0;30;48;2;255;217;47m0[0m [0;30;48;2;102;194;165m*[0m [0;30;48;2;252;141;98m50[0m [0;30;48;2;141;160;203m=[0m [0;30;48;2;231;138;195m600[0m [0;30;48;2;166;216;84m [0m
In [ ]python · cell 19
python
# You need to request access before being able to use this tokenizer
show_tokens(text, "bigcode/starcoder2-15b")Output
tokenizer_config.json: 0%| | 0.00/7.88k [00:00<?, ?B/s]
vocab.json: 0%| | 0.00/777k [00:00<?, ?B/s]
merges.txt: 0%| | 0.00/442k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/2.06M [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/958 [00:00<?, ?B/s]
[0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98mEnglish[0m [0;30;48;2;141;160;203m and[0m [0;30;48;2;231;138;195m CAPITAL[0m [0;30;48;2;166;216;84mIZATION[0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m�[0m [0;30;48;2;252;141;98m�[0m [0;30;48;2;141;160;203m�[0m [0;30;48;2;231;138;195m [0m [0;30;48;2;166;216;84m�[0m [0;30;48;2;255;217;47m�[0m [0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98mshow[0m [0;30;48;2;141;160;203m_[0m [0;30;48;2;231;138;195mtokens[0m [0;30;48;2;166;216;84m False[0m [0;30;48;2;255;217;47m None[0m [0;30;48;2;102;194;165m elif[0m [0;30;48;2;252;141;98m ==[0m [0;30;48;2;141;160;203m >=[0m [0;30;48;2;231;138;195m else[0m [0;30;48;2;166;216;84m:[0m [0;30;48;2;255;217;47m two[0m [0;30;48;2;102;194;165m tabs[0m [0;30;48;2;252;141;98m:"[0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195m "[0m [0;30;48;2;166;216;84m Three[0m [0;30;48;2;255;217;47m tabs[0m [0;30;48;2;102;194;165m:[0m [0;30;48;2;252;141;98m "[0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195m "[0m [0;30;48;2;166;216;84m [0m [0;30;48;2;255;217;47m1[0m [0;30;48;2;102;194;165m2[0m [0;30;48;2;252;141;98m.[0m [0;30;48;2;141;160;203m0[0m [0;30;48;2;231;138;195m*[0m [0;30;48;2;166;216;84m5[0m [0;30;48;2;255;217;47m0[0m [0;30;48;2;102;194;165m=[0m [0;30;48;2;252;141;98m6[0m [0;30;48;2;141;160;203m0[0m [0;30;48;2;231;138;195m0[0m [0;30;48;2;166;216;84m [0m
In [ ]python · cell 20
python
show_tokens(text, "facebook/galactica-1.3b")Output
tokenizer_config.json: 0%| | 0.00/166 [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/2.14M [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/3.00 [00:00<?, ?B/s]
[0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98mEnglish[0m [0;30;48;2;141;160;203m and[0m [0;30;48;2;231;138;195m CAP[0m [0;30;48;2;166;216;84mITAL[0m [0;30;48;2;255;217;47mIZATION[0m [0;30;48;2;102;194;165m [0m [0;30;48;2;252;141;98m�[0m [0;30;48;2;141;160;203m�[0m [0;30;48;2;231;138;195m�[0m [0;30;48;2;166;216;84m�[0m [0;30;48;2;255;217;47m �[0m [0;30;48;2;102;194;165m�[0m [0;30;48;2;252;141;98m�[0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195mshow[0m [0;30;48;2;166;216;84m_[0m [0;30;48;2;255;217;47mtokens[0m [0;30;48;2;102;194;165m False[0m [0;30;48;2;252;141;98m None[0m [0;30;48;2;141;160;203m elif[0m [0;30;48;2;231;138;195m [0m [0;30;48;2;166;216;84m==[0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m>[0m [0;30;48;2;252;141;98m=[0m [0;30;48;2;141;160;203m else[0m [0;30;48;2;231;138;195m:[0m [0;30;48;2;166;216;84m two[0m [0;30;48;2;255;217;47m t[0m [0;30;48;2;102;194;165mabs[0m [0;30;48;2;252;141;98m:[0m [0;30;48;2;141;160;203m"[0m [0;30;48;2;231;138;195m [0m [0;30;48;2;166;216;84m"[0m [0;30;48;2;255;217;47m Three[0m [0;30;48;2;102;194;165m t[0m [0;30;48;2;252;141;98mabs[0m [0;30;48;2;141;160;203m:[0m [0;30;48;2;231;138;195m [0m [0;30;48;2;166;216;84m"[0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m"[0m [0;30;48;2;252;141;98m [0m [0;30;48;2;141;160;203m1[0m [0;30;48;2;231;138;195m2[0m [0;30;48;2;166;216;84m.[0m [0;30;48;2;255;217;47m0[0m [0;30;48;2;102;194;165m*[0m [0;30;48;2;252;141;98m5[0m [0;30;48;2;141;160;203m0[0m [0;30;48;2;231;138;195m=[0m [0;30;48;2;166;216;84m6[0m [0;30;48;2;255;217;47m0[0m [0;30;48;2;102;194;165m0[0m [0;30;48;2;252;141;98m [0m
In [ ]python · cell 21
python
show_tokens(text, "microsoft/Phi-3-mini-4k-instruct")Output
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[0;30;48;2;102;194;165m<s>[0m [0;30;48;2;252;141;98m[0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195mEnglish[0m [0;30;48;2;166;216;84mand[0m [0;30;48;2;255;217;47mC[0m [0;30;48;2;102;194;165mAP[0m [0;30;48;2;252;141;98mIT[0m [0;30;48;2;141;160;203mAL[0m [0;30;48;2;231;138;195mIZ[0m [0;30;48;2;166;216;84mATION[0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m�[0m [0;30;48;2;252;141;98m�[0m [0;30;48;2;141;160;203m�[0m [0;30;48;2;231;138;195m�[0m [0;30;48;2;166;216;84m[0m [0;30;48;2;255;217;47m�[0m [0;30;48;2;102;194;165m�[0m [0;30;48;2;252;141;98m�[0m [0;30;48;2;141;160;203m [0m [0;30;48;2;231;138;195mshow[0m [0;30;48;2;166;216;84m_[0m [0;30;48;2;255;217;47mto[0m [0;30;48;2;102;194;165mkens[0m [0;30;48;2;252;141;98mFalse[0m [0;30;48;2;141;160;203mNone[0m [0;30;48;2;231;138;195melif[0m [0;30;48;2;166;216;84m==[0m [0;30;48;2;255;217;47m>=[0m [0;30;48;2;102;194;165melse[0m [0;30;48;2;252;141;98m:[0m [0;30;48;2;141;160;203mtwo[0m [0;30;48;2;231;138;195mtabs[0m [0;30;48;2;166;216;84m:"[0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m"[0m [0;30;48;2;252;141;98mThree[0m [0;30;48;2;141;160;203mtabs[0m [0;30;48;2;231;138;195m:[0m [0;30;48;2;166;216;84m"[0m [0;30;48;2;255;217;47m [0m [0;30;48;2;102;194;165m"[0m [0;30;48;2;252;141;98m [0m [0;30;48;2;141;160;203m1[0m [0;30;48;2;231;138;195m2[0m [0;30;48;2;166;216;84m.[0m [0;30;48;2;255;217;47m0[0m [0;30;48;2;102;194;165m*[0m [0;30;48;2;252;141;98m5[0m [0;30;48;2;141;160;203m0[0m [0;30;48;2;231;138;195m=[0m [0;30;48;2;166;216;84m6[0m [0;30;48;2;255;217;47m0[0m [0;30;48;2;102;194;165m0[0m [0;30;48;2;252;141;98m [0m
Contextualized Word Embeddings From a Language Model (Like BERT)
In [ ]python · cell 23
python
from transformers import AutoModel, AutoTokenizer
# Load a tokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-base")
# Load a language model
model = AutoModel.from_pretrained("microsoft/deberta-v3-xsmall")
# Tokenize the sentence
tokens = tokenizer('Hello world', return_tensors='pt')
# Process the tokens
output = model(**tokens)[0]Output
tokenizer_config.json: 0%| | 0.00/52.0 [00:00<?, ?B/s]
/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. warnings.warn(
config.json: 0%| | 0.00/474 [00:00<?, ?B/s]
vocab.json: 0%| | 0.00/899k [00:00<?, ?B/s]
merges.txt: 0%| | 0.00/456k [00:00<?, ?B/s]
config.json: 0%| | 0.00/578 [00:00<?, ?B/s]
pytorch_model.bin: 0%| | 0.00/241M [00:00<?, ?B/s]
In [ ]python · cell 24
python
output.shapeOutput
torch.Size([1, 4, 384])
In [ ]python · cell 25
python
for token in tokens['input_ids'][0]:
print(tokenizer.decode(token))Output
[CLS] Hello world [SEP]
In [ ]python · cell 26
python
outputOutput
tensor([[[-3.4816, 0.0861, -0.1819, ..., -0.0612, -0.3911, 0.3017],
[ 0.1898, 0.3208, -0.2315, ..., 0.3714, 0.2478, 0.8048],
[ 0.2071, 0.5036, -0.0485, ..., 1.2175, -0.2292, 0.8582],
[-3.4278, 0.0645, -0.1427, ..., 0.0658, -0.4367, 0.3834]]],
grad_fn=<NativeLayerNormBackward0>)Text Embeddings (For Sentences and Whole Documents)
In [ ]python · cell 28
python
from sentence_transformers import SentenceTransformer
# Load model
model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
# Convert text to text embeddings
vector = model.encode("Best movie ever!")Output
modules.json: 0%| | 0.00/349 [00:00<?, ?B/s]
config_sentence_transformers.json: 0%| | 0.00/116 [00:00<?, ?B/s]
README.md: 0%| | 0.00/10.6k [00:00<?, ?B/s]
sentence_bert_config.json: 0%| | 0.00/53.0 [00:00<?, ?B/s]
/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. warnings.warn(
config.json: 0%| | 0.00/571 [00:00<?, ?B/s]
model.safetensors: 0%| | 0.00/438M [00:00<?, ?B/s]
tokenizer_config.json: 0%| | 0.00/363 [00:00<?, ?B/s]
vocab.txt: 0%| | 0.00/232k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/466k [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/239 [00:00<?, ?B/s]
1_Pooling/config.json: 0%| | 0.00/190 [00:00<?, ?B/s]
In [ ]python · cell 29
python
vector.shapeOutput
(768,)
Word Embeddings Beyond LLMs
In [ ]python · cell 31
python
import gensim.downloader as api
# Download embeddings (66MB, glove, trained on wikipedia, vector size: 50)
# Other options include "word2vec-google-news-300"
# More options at https://github.com/RaRe-Technologies/gensim-data
model = api.load("glove-wiki-gigaword-50")Output
[==================================================] 100.0% 66.0/66.0MB downloaded
In [ ]python · cell 32
python
model.most_similar([model['king']], topn=11)Output
[('king', 1.0000001192092896),
('prince', 0.8236179351806641),
('queen', 0.7839043140411377),
('ii', 0.7746230363845825),
('emperor', 0.7736247777938843),
('son', 0.766719400882721),
('uncle', 0.7627150416374207),
('kingdom', 0.7542161345481873),
('throne', 0.7539914846420288),
('brother', 0.7492411136627197),
('ruler', 0.7434253692626953)]Recommending songs by embeddings
In [ ]python · cell 34
python
import pandas as pd
from urllib import request
# Get the playlist dataset file
data = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/train.txt')
# Parse the playlist dataset file. Skip the first two lines as
# they only contain metadata
lines = data.read().decode("utf-8").split('\n')[2:]
# Remove playlists with only one song
playlists = [s.rstrip().split() for s in lines if len(s.split()) > 1]
# Load song metadata
songs_file = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/song_hash.txt')
songs_file = songs_file.read().decode("utf-8").split('\n')
songs = [s.rstrip().split('\t') for s in songs_file]
songs_df = pd.DataFrame(data=songs, columns = ['id', 'title', 'artist'])
songs_df = songs_df.set_index('id')In [ ]python · cell 35
python
print( 'Playlist #1:\n ', playlists[0], '\n')
print( 'Playlist #2:\n ', playlists[1])Output
Playlist #1: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '2', '42', '43', '44', '45', '46', '47', '48', '20', '49', '8', '50', '51', '52', '53', '54', '55', '56', '57', '25', '58', '59', '60', '61', '62', '3', '63', '64', '65', '66', '46', '47', '67', '2', '48', '68', '69', '70', '57', '50', '71', '72', '53', '73', '25', '74', '59', '20', '46', '75', '76', '77', '59', '20', '43'] Playlist #2: ['78', '79', '80', '3', '62', '81', '14', '82', '48', '83', '84', '17', '85', '86', '87', '88', '74', '89', '90', '91', '4', '73', '62', '92', '17', '53', '59', '93', '94', '51', '50', '27', '95', '48', '96', '97', '98', '99', '100', '57', '101', '102', '25', '103', '3', '104', '105', '106', '107', '47', '108', '109', '110', '111', '112', '113', '25', '63', '62', '114', '115', '84', '116', '117', '118', '119', '120', '121', '122', '123', '50', '70', '71', '124', '17', '85', '14', '82', '48', '125', '47', '46', '72', '53', '25', '73', '4', '126', '59', '74', '20', '43', '127', '128', '129', '13', '82', '48', '130', '131', '132', '133', '134', '135', '136', '137', '59', '46', '138', '43', '20', '139', '140', '73', '57', '70', '141', '3', '1', '74', '142', '143', '144', '145', '48', '13', '25', '146', '50', '147', '126', '59', '20', '148', '149', '150', '151', '152', '56', '153', '154', '155', '156', '157', '158', '159', '160', '161', '162', '163', '164', '165', '166', '167', '168', '169', '170', '171', '172', '173', '174', '175', '60', '176', '51', '177', '178', '179', '180', '181', '182', '183', '184', '185', '57', '186', '187', '188', '189', '190', '191', '46', '192', '193', '194', '195', '196', '197', '198', '25', '199', '200', '49', '201', '100', '202', '203', '204', '205', '206', '207', '32', '208', '209', '210']
In [ ]python · cell 36
python
from gensim.models import Word2Vec
# Train our Word2Vec model
model = Word2Vec(
playlists, vector_size=32, window=20, negative=50, min_count=1, workers=4
)In [ ]python · cell 37
python
song_id = 2172
# Ask the model for songs similar to song #2172
model.wv.most_similar(positive=str(song_id))Output
[('2849', 0.9979680776596069),
('2640', 0.9964019060134888),
('3167', 0.9963980317115784),
('5549', 0.9959008693695068),
('2715', 0.9958351850509644),
('3117', 0.9954560995101929),
('2987', 0.9953479766845703),
('2881', 0.9951083660125732),
('2886', 0.9950577616691589),
('3094', 0.994985044002533)]In [ ]python · cell 38
python
print(songs_df.iloc[2172])Output
title Fade To Black artist Metallica Name: 2172 , dtype: object
In [ ]python · cell 39
python
import numpy as np
def print_recommendations(song_id):
similar_songs = np.array(
model.wv.most_similar(positive=str(song_id),topn=5)
)[:,0]
return songs_df.iloc[similar_songs]
# Extract recommendations
print_recommendations(2172)Output
title artist id 2849 Run To The Hills Iron Maiden 2640 Red Barchetta Rush 3167 Unchained Van Halen 5549 November Rain Guns N' Roses 2715 Rainbow In The Dark Dio
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| title | artist | |
|---|---|---|
| id | ||
| 2849 | Run To The Hills | Iron Maiden |
| 2640 | Red Barchetta | Rush |
| 3167 | Unchained | Van Halen |
| 5549 | November Rain | Guns N' Roses |
| 2715 | Rainbow In The Dark | Dio |
.colab-df-container {
display:flex;
gap: 12px;
}
.colab-df-convert {
background-color: #E8F0FE;
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: #1967D2;
height: 32px;
padding: 0 0 0 0;
width: 32px;
}
.colab-df-convert:hover {
background-color: #E2EBFA;
box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: #174EA6;
}
.colab-df-buttons div {
margin-bottom: 4px;
}
[theme=dark] .colab-df-convert {
background-color: #3B4455;
fill: #D2E3FC;
}
[theme=dark] .colab-df-convert:hover {
background-color: #434B5C;
box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);
filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));
fill: #FFFFFF;
}
.colab-df-quickchart {
--bg-color: #E8F0FE;
--fill-color: #1967D2;
--hover-bg-color: #E2EBFA;
--hover-fill-color: #174EA6;
--disabled-fill-color: #AAA;
--disabled-bg-color: #DDD;
}
[theme=dark] .colab-df-quickchart {
--bg-color: #3B4455;
--fill-color: #D2E3FC;
--hover-bg-color: #434B5C;
--hover-fill-color: #FFFFFF;
--disabled-bg-color: #3B4455;
--disabled-fill-color: #666;
}
.colab-df-quickchart {
background-color: var(--bg-color);
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: var(--fill-color);
height: 32px;
padding: 0;
width: 32px;
}
.colab-df-quickchart:hover {
background-color: var(--hover-bg-color);
box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: var(--button-hover-fill-color);
}
.colab-df-quickchart-complete:disabled,
.colab-df-quickchart-complete:disabled:hover {
background-color: var(--disabled-bg-color);
fill: var(--disabled-fill-color);
box-shadow: none;
}
.colab-df-spinner {
border: 2px solid var(--fill-color);
border-color: transparent;
border-bottom-color: var(--fill-color);
animation:
spin 1s steps(1) infinite;
}
@keyframes spin {
0% {
border-color: transparent;
border-bottom-color: var(--fill-color);
border-left-color: var(--fill-color);
}
20% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
}
30% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
border-right-color: var(--fill-color);
}
40% {
border-color: transparent;
border-right-color: var(--fill-color);
border-top-color: var(--fill-color);
}
60% {
border-color: transparent;
border-right-color: var(--fill-color);
}
80% {
border-color: transparent;
border-right-color: var(--fill-color);
border-bottom-color: var(--fill-color);
}
90% {
border-color: transparent;
border-bottom-color: var(--fill-color);
}
}
In [ ]python · cell 40
python
print_recommendations(2172)Output
title Fade To Black artist Metallica Name: 2172 , dtype: object ['2849' '2640' '3167' '5549' '2715']
title artist id 2849 Run To The Hills Iron Maiden 2640 Red Barchetta Rush 3167 Unchained Van Halen 5549 November Rain Guns N' Roses 2715 Rainbow In The Dark Dio
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| title | artist | |
|---|---|---|
| id | ||
| 2849 | Run To The Hills | Iron Maiden |
| 2640 | Red Barchetta | Rush |
| 3167 | Unchained | Van Halen |
| 5549 | November Rain | Guns N' Roses |
| 2715 | Rainbow In The Dark | Dio |
.colab-df-container {
display:flex;
gap: 12px;
}
.colab-df-convert {
background-color: #E8F0FE;
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: #1967D2;
height: 32px;
padding: 0 0 0 0;
width: 32px;
}
.colab-df-convert:hover {
background-color: #E2EBFA;
box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: #174EA6;
}
.colab-df-buttons div {
margin-bottom: 4px;
}
[theme=dark] .colab-df-convert {
background-color: #3B4455;
fill: #D2E3FC;
}
[theme=dark] .colab-df-convert:hover {
background-color: #434B5C;
box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);
filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));
fill: #FFFFFF;
}
.colab-df-quickchart {
--bg-color: #E8F0FE;
--fill-color: #1967D2;
--hover-bg-color: #E2EBFA;
--hover-fill-color: #174EA6;
--disabled-fill-color: #AAA;
--disabled-bg-color: #DDD;
}
[theme=dark] .colab-df-quickchart {
--bg-color: #3B4455;
--fill-color: #D2E3FC;
--hover-bg-color: #434B5C;
--hover-fill-color: #FFFFFF;
--disabled-bg-color: #3B4455;
--disabled-fill-color: #666;
}
.colab-df-quickchart {
background-color: var(--bg-color);
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: var(--fill-color);
height: 32px;
padding: 0;
width: 32px;
}
.colab-df-quickchart:hover {
background-color: var(--hover-bg-color);
box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: var(--button-hover-fill-color);
}
.colab-df-quickchart-complete:disabled,
.colab-df-quickchart-complete:disabled:hover {
background-color: var(--disabled-bg-color);
fill: var(--disabled-fill-color);
box-shadow: none;
}
.colab-df-spinner {
border: 2px solid var(--fill-color);
border-color: transparent;
border-bottom-color: var(--fill-color);
animation:
spin 1s steps(1) infinite;
}
@keyframes spin {
0% {
border-color: transparent;
border-bottom-color: var(--fill-color);
border-left-color: var(--fill-color);
}
20% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
}
30% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
border-right-color: var(--fill-color);
}
40% {
border-color: transparent;
border-right-color: var(--fill-color);
border-top-color: var(--fill-color);
}
60% {
border-color: transparent;
border-right-color: var(--fill-color);
}
80% {
border-color: transparent;
border-right-color: var(--fill-color);
border-bottom-color: var(--fill-color);
}
90% {
border-color: transparent;
border-bottom-color: var(--fill-color);
}
}
In [ ]python · cell 41
python
print_recommendations(842)Output
title California Love (w\/ Dr. Dre & Roger Troutman) artist 2Pac Name: 842 , dtype: object ['5668' '413' '5661' '330' '886']
title artist id 5668 How We Do (w\/ 50 Cent) The Game 413 If I Ruled The World (Imagine That) (w\/ Laury... Nas 5661 Sweet Dreams Beyonce 330 Hate It Or Love It (w\/ 50 Cent) The Game 886 Heartless Kanye West
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| title | artist | |
|---|---|---|
| id | ||
| 5668 | How We Do (w\/ 50 Cent) | The Game |
| 413 | If I Ruled The World (Imagine That) (w\/ Laury... | Nas |
| 5661 | Sweet Dreams | Beyonce |
| 330 | Hate It Or Love It (w\/ 50 Cent) | The Game |
| 886 | Heartless | Kanye West |
.colab-df-container {
display:flex;
gap: 12px;
}
.colab-df-convert {
background-color: #E8F0FE;
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: #1967D2;
height: 32px;
padding: 0 0 0 0;
width: 32px;
}
.colab-df-convert:hover {
background-color: #E2EBFA;
box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: #174EA6;
}
.colab-df-buttons div {
margin-bottom: 4px;
}
[theme=dark] .colab-df-convert {
background-color: #3B4455;
fill: #D2E3FC;
}
[theme=dark] .colab-df-convert:hover {
background-color: #434B5C;
box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);
filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));
fill: #FFFFFF;
}
.colab-df-quickchart {
--bg-color: #E8F0FE;
--fill-color: #1967D2;
--hover-bg-color: #E2EBFA;
--hover-fill-color: #174EA6;
--disabled-fill-color: #AAA;
--disabled-bg-color: #DDD;
}
[theme=dark] .colab-df-quickchart {
--bg-color: #3B4455;
--fill-color: #D2E3FC;
--hover-bg-color: #434B5C;
--hover-fill-color: #FFFFFF;
--disabled-bg-color: #3B4455;
--disabled-fill-color: #666;
}
.colab-df-quickchart {
background-color: var(--bg-color);
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: var(--fill-color);
height: 32px;
padding: 0;
width: 32px;
}
.colab-df-quickchart:hover {
background-color: var(--hover-bg-color);
box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: var(--button-hover-fill-color);
}
.colab-df-quickchart-complete:disabled,
.colab-df-quickchart-complete:disabled:hover {
background-color: var(--disabled-bg-color);
fill: var(--disabled-fill-color);
box-shadow: none;
}
.colab-df-spinner {
border: 2px solid var(--fill-color);
border-color: transparent;
border-bottom-color: var(--fill-color);
animation:
spin 1s steps(1) infinite;
}
@keyframes spin {
0% {
border-color: transparent;
border-bottom-color: var(--fill-color);
border-left-color: var(--fill-color);
}
20% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
}
30% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
border-right-color: var(--fill-color);
}
40% {
border-color: transparent;
border-right-color: var(--fill-color);
border-top-color: var(--fill-color);
}
60% {
border-color: transparent;
border-right-color: var(--fill-color);
}
80% {
border-color: transparent;
border-right-color: var(--fill-color);
border-bottom-color: var(--fill-color);
}
90% {
border-color: transparent;
border-bottom-color: var(--fill-color);
}
}