Chapter 05
Chapter 5 Text Clustering and Topic Modeling
NotebookPython 3 (ipykernel)58 cells
This notebook is for Chapter 5 of the Hands-On Large Language Models book by Jay Alammar and Maarten Grootendorst.
[OPTIONAL] - Installing Packages on
If you are viewing this notebook on Google Colab (or any other cloud vendor), you need to uncomment and run the following codeblock to install the dependencies for this chapter:
💡 NOTE: We will want to use a GPU to run the examples in this notebook. In Google Colab, go to Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4.
In [ ]python · cell 3
python
# %%capture
# !pip install bertopic datasets openai datamapplotArXiv Articles: Computation and Language
In [ ]python · cell 5
python
# Load data from huggingface
from datasets import load_dataset
dataset = load_dataset("maartengr/arxiv_nlp")["train"]
# Extract metadata
abstracts = list(dataset["Abstracts"])
titles = list(dataset["Titles"])Output
/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: The secret `HF_TOKEN` does not exist in your Colab secrets. To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session. You will be able to reuse this secret in all of your notebooks. Please note that authentication is recommended but still optional to access public models or datasets. warnings.warn(
A Common Pipeline for Text Clustering
1. Embedding Documents
In [ ]python · cell 8
python
from sentence_transformers import SentenceTransformer
# Create an embedding for each abstract
embedding_model = SentenceTransformer('thenlper/gte-small')
embeddings = embedding_model.encode(abstracts, show_progress_bar=True)Output
Batches: 0%| | 0/1405 [00:00<?, ?it/s]
In [ ]python · cell 9
python
# Check the dimensions of the resulting embeddings
embeddings.shapeOutput
(44949, 384)
2. Reducing the Dimensionality of Embeddings
In [ ]python · cell 11
python
from umap import UMAP
# We reduce the input embeddings from 384 dimenions to 5 dimenions
umap_model = UMAP(
n_components=5, min_dist=0.0, metric='cosine', random_state=42
)
reduced_embeddings = umap_model.fit_transform(embeddings)Output
/usr/local/lib/python3.10/dist-packages/umap/umap_.py:1945: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")
3. Cluster the Reduced Embeddings
In [ ]python · cell 13
python
from hdbscan import HDBSCAN
# We fit the model and extract the clusters
hdbscan_model = HDBSCAN(
min_cluster_size=50, metric='euclidean', cluster_selection_method='eom'
).fit(reduced_embeddings)
clusters = hdbscan_model.labels_
# How many clusters did we generate?
len(set(clusters))Output
/usr/local/lib/python3.10/dist-packages/joblib/externals/loky/backend/fork_exec.py:38: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock. pid = os.fork()
156
Inspecting the Clusters
Manually inspect the first three documents in cluster 0:
In [ ]python · cell 16
python
import numpy as np
# Print first three documents in cluster 0
cluster = 0
for index in np.where(clusters==cluster)[0][:3]:
print(abstracts[index][:300] + "... \n")Output
This works aims to design a statistical machine translation from English text to American Sign Language (ASL). The system is based on Moses tool with some modifications and the results are synthesized through a 3D avatar for interpretation. First, we translate the input text to gloss, a written fo... Researches on signed languages still strongly dissociate lin- guistic issues related on phonological and phonetic aspects, and gesture studies for recognition and synthesis purposes. This paper focuses on the imbrication of motion and meaning for the analysis, synthesis and evaluation of sign lang... Modern computational linguistic software cannot produce important aspects of sign language translation. Using some researches we deduce that the majority of automatic sign language translation systems ignore many aspects when they generate animation; therefore the interpretation lost the truth inf...
Next, we reduce our embeddings to 2-dimensions so that we can plot them and get a rough understanding of the generated clusters.
In [ ]python · cell 18
python
import pandas as pd
# Reduce 384-dimensional embeddings to 2 dimensions for easier visualization
reduced_embeddings = UMAP(
n_components=2, min_dist=0.0, metric='cosine', random_state=42
).fit_transform(embeddings)
# Create dataframe
df = pd.DataFrame(reduced_embeddings, columns=["x", "y"])
df["title"] = titles
df["cluster"] = [str(c) for c in clusters]
# Select outliers and non-outliers (clusters)
clusters_df = df.loc[df.cluster != "-1", :]
outliers_df = df.loc[df.cluster == "-1", :]Output
/usr/local/lib/python3.10/dist-packages/umap/umap_.py:1945: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")
Static Plot
In [ ]python · cell 20
python
import matplotlib.pyplot as plt
# Plot outliers and non-outliers seperately
plt.scatter(outliers_df.x, outliers_df.y, alpha=0.05, s=2, c="grey")
plt.scatter(
clusters_df.x, clusters_df.y, c=clusters_df.cluster.astype(int),
alpha=0.6, s=2, cmap='tab20b'
)
plt.axis('off')
# plt.savefig("matplotlib.png", dpi=300) # Uncomment to save the graph as a .pngOutput
<Figure size 640x480 with 1 Axes>
[省略较大 image/png 输出]
From Text Clustering to Topic Modeling
BERTopic: A Modular Topic Modeling Framework
In [ ]python · cell 23
python
from bertopic import BERTopic
# Train our model with our previously defined models
topic_model = BERTopic(
embedding_model=embedding_model,
umap_model=umap_model,
hdbscan_model=hdbscan_model,
verbose=True
).fit(abstracts, embeddings)Output
2024-04-24 10:38:31,273 - BERTopic - Dimensionality - Fitting the dimensionality reduction algorithm 2024-04-24 10:39:22,540 - BERTopic - Dimensionality - Completed ✓ 2024-04-24 10:39:22,543 - BERTopic - Cluster - Start clustering the reduced embeddings 2024-04-24 10:39:24,548 - BERTopic - Cluster - Completed ✓ 2024-04-24 10:39:24,563 - BERTopic - Representation - Extracting topics from clusters using representation models. 2024-04-24 10:39:34,185 - BERTopic - Representation - Completed ✓
Now, let's start exploring the topics that we got by running the code above.
In [ ]python · cell 25
python
topic_model.get_topic_info()Output
Topic Count Name \
0 -1 14520 -1_the_of_and_to
1 0 2290 0_speech_asr_recognition_end
2 1 1403 1_medical_clinical_biomedical_patient
3 2 1156 2_sentiment_aspect_analysis_reviews
4 3 986 3_translation_nmt_machine_neural
.. ... ... ...
151 150 54 150_coherence_discourse_paragraph_text
152 151 54 151_prompt_prompts_optimization_prompting
153 152 53 152_sentence_sts_embeddings_similarity
154 153 53 153_counseling_mental_health_therapy
155 154 50 154_backdoor_attacks_attack_triggers
Representation \
0 [the, of, and, to, in, we, that, language, for...
1 [speech, asr, recognition, end, acoustic, spea...
2 [medical, clinical, biomedical, patient, healt...
3 [sentiment, aspect, analysis, reviews, opinion...
4 [translation, nmt, machine, neural, bleu, engl...
.. ...
151 [coherence, discourse, paragraph, text, cohesi...
152 [prompt, prompts, optimization, prompting, llm...
153 [sentence, sts, embeddings, similarity, embedd...
154 [counseling, mental, health, therapy, psychoth...
155 [backdoor, attacks, attack, triggers, poisoned...
Representative_Docs
0 [ This paper presents new state-of-the-art mo...
1 [ End-to-end models have achieved impressive ...
2 [ Recent years have seen particular interest ...
3 [ Aspect-based sentiment analysis of review t...
4 [ Neural machine translation (NMT) becomes a ...
.. ...
151 [ While there has been significant progress t...
152 [ Prompt optimization aims to find the best p...
153 [ Sentence representation at the semantic lev...
154 [ Mental health care poses an increasingly se...
155 [ Deep neural networks (DNNs) and natural lan...
[156 rows x 5 columns]
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Topic | Count | Name | Representation | Representative_Docs | |
|---|---|---|---|---|---|
| 0 | -1 | 14520 | -1_the_of_and_to | [the, of, and, to, in, we, that, language, for... | [ This paper presents new state-of-the-art mo... |
| 1 | 0 | 2290 | 0_speech_asr_recognition_end | [speech, asr, recognition, end, acoustic, spea... | [ End-to-end models have achieved impressive ... |
| 2 | 1 | 1403 | 1_medical_clinical_biomedical_patient | [medical, clinical, biomedical, patient, healt... | [ Recent years have seen particular interest ... |
| 3 | 2 | 1156 | 2_sentiment_aspect_analysis_reviews | [sentiment, aspect, analysis, reviews, opinion... | [ Aspect-based sentiment analysis of review t... |
| 4 | 3 | 986 | 3_translation_nmt_machine_neural | [translation, nmt, machine, neural, bleu, engl... | [ Neural machine translation (NMT) becomes a ... |
| ... | ... | ... | ... | ... | ... |
| 151 | 150 | 54 | 150_coherence_discourse_paragraph_text | [coherence, discourse, paragraph, text, cohesi... | [ While there has been significant progress t... |
| 152 | 151 | 54 | 151_prompt_prompts_optimization_prompting | [prompt, prompts, optimization, prompting, llm... | [ Prompt optimization aims to find the best p... |
| 153 | 152 | 53 | 152_sentence_sts_embeddings_similarity | [sentence, sts, embeddings, similarity, embedd... | [ Sentence representation at the semantic lev... |
| 154 | 153 | 53 | 153_counseling_mental_health_therapy | [counseling, mental, health, therapy, psychoth... | [ Mental health care poses an increasingly se... |
| 155 | 154 | 50 | 154_backdoor_attacks_attack_triggers | [backdoor, attacks, attack, triggers, poisoned... | [ Deep neural networks (DNNs) and natural lan... |
156 rows × 5 columns
.colab-df-container {
display:flex;
gap: 12px;
}
.colab-df-convert {
background-color: #E8F0FE;
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: #1967D2;
height: 32px;
padding: 0 0 0 0;
width: 32px;
}
.colab-df-convert:hover {
background-color: #E2EBFA;
box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: #174EA6;
}
.colab-df-buttons div {
margin-bottom: 4px;
}
[theme=dark] .colab-df-convert {
background-color: #3B4455;
fill: #D2E3FC;
}
[theme=dark] .colab-df-convert:hover {
background-color: #434B5C;
box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);
filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));
fill: #FFFFFF;
}
.colab-df-quickchart {
--bg-color: #E8F0FE;
--fill-color: #1967D2;
--hover-bg-color: #E2EBFA;
--hover-fill-color: #174EA6;
--disabled-fill-color: #AAA;
--disabled-bg-color: #DDD;
}
[theme=dark] .colab-df-quickchart {
--bg-color: #3B4455;
--fill-color: #D2E3FC;
--hover-bg-color: #434B5C;
--hover-fill-color: #FFFFFF;
--disabled-bg-color: #3B4455;
--disabled-fill-color: #666;
}
.colab-df-quickchart {
background-color: var(--bg-color);
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: var(--fill-color);
height: 32px;
padding: 0;
width: 32px;
}
.colab-df-quickchart:hover {
background-color: var(--hover-bg-color);
box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: var(--button-hover-fill-color);
}
.colab-df-quickchart-complete:disabled,
.colab-df-quickchart-complete:disabled:hover {
background-color: var(--disabled-bg-color);
fill: var(--disabled-fill-color);
box-shadow: none;
}
.colab-df-spinner {
border: 2px solid var(--fill-color);
border-color: transparent;
border-bottom-color: var(--fill-color);
animation:
spin 1s steps(1) infinite;
}
@keyframes spin {
0% {
border-color: transparent;
border-bottom-color: var(--fill-color);
border-left-color: var(--fill-color);
}
20% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
}
30% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
border-right-color: var(--fill-color);
}
40% {
border-color: transparent;
border-right-color: var(--fill-color);
border-top-color: var(--fill-color);
}
60% {
border-color: transparent;
border-right-color: var(--fill-color);
}
80% {
border-color: transparent;
border-right-color: var(--fill-color);
border-bottom-color: var(--fill-color);
}
90% {
border-color: transparent;
border-bottom-color: var(--fill-color);
}
}
Hundreds of topics were generated using the default model! To get the top 10 keywords per topic as well as their c-TF-IDF weights, we can use the get_topic() function:
In [ ]python · cell 27
python
topic_model.get_topic(0)Output
[('speech', 0.028177697715245358),
('asr', 0.018971184497453525),
('recognition', 0.013457745472471012),
('end', 0.00980445092749381),
('acoustic', 0.009452082794507863),
('speaker', 0.0068822647060204885),
('audio', 0.006807649923681604),
('the', 0.0063343444687017645),
('error', 0.006320144717019838),
('automatic', 0.006290216996043161)]We can use the find_topics() function to search for specific topics based on a search term. Let’s search for a topic about topic modeling:
In [ ]python · cell 29
python
topic_model.find_topics("topic modeling")Output
([22, -1, 1, 47, 32], [0.95456535, 0.91173744, 0.9074769, 0.9067007, 0.90510106])
It returns that topic 22 has a relatively high similarity (0.95) with our search term. If we then inspect the topic, we can see that it is indeed a topic about topic modeling:
In [ ]python · cell 31
python
topic_model.get_topic(22)Output
[('topic', 0.06634619076655907),
('topics', 0.035308535091932707),
('lda', 0.016386314730705634),
('latent', 0.013372311924864435),
('document', 0.012973600191120576),
('documents', 0.012383715497143821),
('modeling', 0.011978375291037142),
('dirichlet', 0.010078277589545706),
('word', 0.008505619415413312),
('allocation', 0.007930890698168108)]That seems like a topic that is, in part, characterized by the classic LDA technique. Let's see if the BERTopic paper was also assigned to topic 22:
In [ ]python · cell 33
python
topic_model.topics_[titles.index('BERTopic: Neural topic modeling with a class-based TF-IDF procedure')]Output
22
It is! We expected it might be because there are non-LDA specific words in the topic describtion such as "clustering" and "topic".
Visualizations
Visualize Documents
In [ ]python · cell 37
python
# Visualize topics and documents
fig = topic_model.visualize_documents(
titles,
reduced_embeddings=reduced_embeddings,
width=1200,
hide_annotations=True
)
# Update fonts of legend for easier visualization
fig.update_layout(font=dict(size=16))Output
Output hidden; open in https://colab.research.google.com to view.
In [ ]python · cell 38
python
# Visualize barchart with ranked keywords
topic_model.visualize_barchart()
# Visualize relationships between topics
topic_model.visualize_heatmap(n_clusters=30)
# Visualize the potential hierarchical structure of topics
topic_model.visualize_hierarchy()Output
Representation Models
In these examples that follow, we will update our topic representations after having trained our model. This allows for quick iteration. If, however, you want to use a representation model at the start of training, you will need to run it as follows:
python
from bertopic.representation import KeyBERTInspired
from bertopic import BERTopic
# Create your representation model
representation_model = KeyBERTInspired()
# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)To use the representation models, we are first going to duplicate our topic model such that easily show the differences between a model with and without representation model.
In [ ]python · cell 42
python
# Save original representations
from copy import deepcopy
original_topics = deepcopy(topic_model.topic_representations_)In [ ]python · cell 43
python
def topic_differences(model, original_topics, nr_topics=5):
"""Show the differences in topic representations between two models """
df = pd.DataFrame(columns=["Topic", "Original", "Updated"])
for topic in range(nr_topics):
# Extract top 5 words per topic per model
og_words = " | ".join(list(zip(*original_topics[topic]))[0][:5])
new_words = " | ".join(list(zip(*model.get_topic(topic)))[0][:5])
df.loc[len(df)] = [topic, og_words, new_words]
return dfKeyBERTInspired
In [ ]python · cell 45
python
from bertopic.representation import KeyBERTInspired
# Update our topic representations using KeyBERTInspired
representation_model = KeyBERTInspired()
topic_model.update_topics(abstracts, representation_model=representation_model)
# Show topic differences
topic_differences(topic_model, original_topics)Output
Topic Original \
0 0 speech | asr | recognition | end | acoustic
1 1 medical | clinical | biomedical | patient | he...
2 2 sentiment | aspect | analysis | reviews | opinion
3 3 translation | nmt | machine | neural | bleu
4 4 summarization | summaries | summary | abstract...
Updated
0 speech | encoder | phonetic | language | trans...
1 nlp | ehr | clinical | biomedical | language
2 aspect | sentiment | aspects | sentiments | cl...
3 translation | translating | translate | transl...
4 summarization | summarizers | summaries | summ...
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Topic | Original | Updated | |
|---|---|---|---|
| 0 | 0 | speech | asr | recognition | end | acoustic | speech | encoder | phonetic | language | trans... |
| 1 | 1 | medical | clinical | biomedical | patient | he... | nlp | ehr | clinical | biomedical | language |
| 2 | 2 | sentiment | aspect | analysis | reviews | opinion | aspect | sentiment | aspects | sentiments | cl... |
| 3 | 3 | translation | nmt | machine | neural | bleu | translation | translating | translate | transl... |
| 4 | 4 | summarization | summaries | summary | abstract... | summarization | summarizers | summaries | summ... |
.colab-df-container {
display:flex;
gap: 12px;
}
.colab-df-convert {
background-color: #E8F0FE;
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: #1967D2;
height: 32px;
padding: 0 0 0 0;
width: 32px;
}
.colab-df-convert:hover {
background-color: #E2EBFA;
box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: #174EA6;
}
.colab-df-buttons div {
margin-bottom: 4px;
}
[theme=dark] .colab-df-convert {
background-color: #3B4455;
fill: #D2E3FC;
}
[theme=dark] .colab-df-convert:hover {
background-color: #434B5C;
box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);
filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));
fill: #FFFFFF;
}
.colab-df-quickchart {
--bg-color: #E8F0FE;
--fill-color: #1967D2;
--hover-bg-color: #E2EBFA;
--hover-fill-color: #174EA6;
--disabled-fill-color: #AAA;
--disabled-bg-color: #DDD;
}
[theme=dark] .colab-df-quickchart {
--bg-color: #3B4455;
--fill-color: #D2E3FC;
--hover-bg-color: #434B5C;
--hover-fill-color: #FFFFFF;
--disabled-bg-color: #3B4455;
--disabled-fill-color: #666;
}
.colab-df-quickchart {
background-color: var(--bg-color);
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: var(--fill-color);
height: 32px;
padding: 0;
width: 32px;
}
.colab-df-quickchart:hover {
background-color: var(--hover-bg-color);
box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: var(--button-hover-fill-color);
}
.colab-df-quickchart-complete:disabled,
.colab-df-quickchart-complete:disabled:hover {
background-color: var(--disabled-bg-color);
fill: var(--disabled-fill-color);
box-shadow: none;
}
.colab-df-spinner {
border: 2px solid var(--fill-color);
border-color: transparent;
border-bottom-color: var(--fill-color);
animation:
spin 1s steps(1) infinite;
}
@keyframes spin {
0% {
border-color: transparent;
border-bottom-color: var(--fill-color);
border-left-color: var(--fill-color);
}
20% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
}
30% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
border-right-color: var(--fill-color);
}
40% {
border-color: transparent;
border-right-color: var(--fill-color);
border-top-color: var(--fill-color);
}
60% {
border-color: transparent;
border-right-color: var(--fill-color);
}
80% {
border-color: transparent;
border-right-color: var(--fill-color);
border-bottom-color: var(--fill-color);
}
90% {
border-color: transparent;
border-bottom-color: var(--fill-color);
}
}
Maximal Marginal Relevance
In [ ]python · cell 47
python
from bertopic.representation import MaximalMarginalRelevance
# Update our topic representations to MaximalMarginalRelevance
representation_model = MaximalMarginalRelevance(diversity=0.5)
topic_model.update_topics(abstracts, representation_model=representation_model)
# Show topic differences
topic_differences(topic_model, original_topics)Output
Topic Original \
0 0 speech | asr | recognition | end | acoustic
1 1 medical | clinical | biomedical | patient | he...
2 2 sentiment | aspect | analysis | reviews | opinion
3 3 translation | nmt | machine | neural | bleu
4 4 summarization | summaries | summary | abstract...
Updated
0 speech | asr | error | model | training
1 clinical | biomedical | patient | healthcare |...
2 sentiment | analysis | reviews | absa | polarity
3 translation | nmt | bleu | parallel | multilin...
4 summarization | document | extractive | rouge ...
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Topic | Original | Updated | |
|---|---|---|---|
| 0 | 0 | speech | asr | recognition | end | acoustic | speech | asr | error | model | training |
| 1 | 1 | medical | clinical | biomedical | patient | he... | clinical | biomedical | patient | healthcare |... |
| 2 | 2 | sentiment | aspect | analysis | reviews | opinion | sentiment | analysis | reviews | absa | polarity |
| 3 | 3 | translation | nmt | machine | neural | bleu | translation | nmt | bleu | parallel | multilin... |
| 4 | 4 | summarization | summaries | summary | abstract... | summarization | document | extractive | rouge ... |
.colab-df-container {
display:flex;
gap: 12px;
}
.colab-df-convert {
background-color: #E8F0FE;
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: #1967D2;
height: 32px;
padding: 0 0 0 0;
width: 32px;
}
.colab-df-convert:hover {
background-color: #E2EBFA;
box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: #174EA6;
}
.colab-df-buttons div {
margin-bottom: 4px;
}
[theme=dark] .colab-df-convert {
background-color: #3B4455;
fill: #D2E3FC;
}
[theme=dark] .colab-df-convert:hover {
background-color: #434B5C;
box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);
filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));
fill: #FFFFFF;
}
.colab-df-quickchart {
--bg-color: #E8F0FE;
--fill-color: #1967D2;
--hover-bg-color: #E2EBFA;
--hover-fill-color: #174EA6;
--disabled-fill-color: #AAA;
--disabled-bg-color: #DDD;
}
[theme=dark] .colab-df-quickchart {
--bg-color: #3B4455;
--fill-color: #D2E3FC;
--hover-bg-color: #434B5C;
--hover-fill-color: #FFFFFF;
--disabled-bg-color: #3B4455;
--disabled-fill-color: #666;
}
.colab-df-quickchart {
background-color: var(--bg-color);
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: var(--fill-color);
height: 32px;
padding: 0;
width: 32px;
}
.colab-df-quickchart:hover {
background-color: var(--hover-bg-color);
box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: var(--button-hover-fill-color);
}
.colab-df-quickchart-complete:disabled,
.colab-df-quickchart-complete:disabled:hover {
background-color: var(--disabled-bg-color);
fill: var(--disabled-fill-color);
box-shadow: none;
}
.colab-df-spinner {
border: 2px solid var(--fill-color);
border-color: transparent;
border-bottom-color: var(--fill-color);
animation:
spin 1s steps(1) infinite;
}
@keyframes spin {
0% {
border-color: transparent;
border-bottom-color: var(--fill-color);
border-left-color: var(--fill-color);
}
20% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
}
30% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
border-right-color: var(--fill-color);
}
40% {
border-color: transparent;
border-right-color: var(--fill-color);
border-top-color: var(--fill-color);
}
60% {
border-color: transparent;
border-right-color: var(--fill-color);
}
80% {
border-color: transparent;
border-right-color: var(--fill-color);
border-bottom-color: var(--fill-color);
}
90% {
border-color: transparent;
border-bottom-color: var(--fill-color);
}
}
Text Generation
Flan-T5
In [ ]python · cell 50
python
from transformers import pipeline
from bertopic.representation import TextGeneration
prompt = """I have a topic that contains the following documents:
[DOCUMENTS]
The topic is described by the following keywords: '[KEYWORDS]'.
Based on the documents and keywords, what is this topic about?"""
# Update our topic representations using Flan-T5
generator = pipeline('text2text-generation', model='google/flan-t5-small')
representation_model = TextGeneration(
generator, prompt=prompt, doc_length=50, tokenizer="whitespace"
)
topic_model.update_topics(abstracts, representation_model=representation_model)
# Show topic differences
topic_differences(topic_model, original_topics)Output
config.json: 0%| | 0.00/1.40k [00:00<?, ?B/s]
model.safetensors: 0%| | 0.00/308M [00:00<?, ?B/s]
generation_config.json: 0%| | 0.00/147 [00:00<?, ?B/s]
tokenizer_config.json: 0%| | 0.00/2.54k [00:00<?, ?B/s]
spiece.model: 0%| | 0.00/792k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/2.42M [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/2.20k [00:00<?, ?B/s]
100%|██████████| 156/156 [01:28<00:00, 1.76it/s]
Topic Original \
0 0 speech | asr | recognition | end | acoustic
1 1 medical | clinical | biomedical | patient | he...
2 2 sentiment | aspect | analysis | reviews | opinion
3 3 translation | nmt | machine | neural | bleu
4 4 summarization | summaries | summary | abstract...
Updated
0 Speech-to-description | | | |
1 Science/Tech | | | |
2 Review | | | |
3 Attention-based neural machine translation | ...
4 Summarization | | | |
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Topic | Original | Updated | |
|---|---|---|---|
| 0 | 0 | speech | asr | recognition | end | acoustic | Speech-to-description | | | | |
| 1 | 1 | medical | clinical | biomedical | patient | he... | Science/Tech | | | | |
| 2 | 2 | sentiment | aspect | analysis | reviews | opinion | Review | | | | |
| 3 | 3 | translation | nmt | machine | neural | bleu | Attention-based neural machine translation | ... |
| 4 | 4 | summarization | summaries | summary | abstract... | Summarization | | | | |
.colab-df-container {
display:flex;
gap: 12px;
}
.colab-df-convert {
background-color: #E8F0FE;
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: #1967D2;
height: 32px;
padding: 0 0 0 0;
width: 32px;
}
.colab-df-convert:hover {
background-color: #E2EBFA;
box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: #174EA6;
}
.colab-df-buttons div {
margin-bottom: 4px;
}
[theme=dark] .colab-df-convert {
background-color: #3B4455;
fill: #D2E3FC;
}
[theme=dark] .colab-df-convert:hover {
background-color: #434B5C;
box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);
filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));
fill: #FFFFFF;
}
.colab-df-quickchart {
--bg-color: #E8F0FE;
--fill-color: #1967D2;
--hover-bg-color: #E2EBFA;
--hover-fill-color: #174EA6;
--disabled-fill-color: #AAA;
--disabled-bg-color: #DDD;
}
[theme=dark] .colab-df-quickchart {
--bg-color: #3B4455;
--fill-color: #D2E3FC;
--hover-bg-color: #434B5C;
--hover-fill-color: #FFFFFF;
--disabled-bg-color: #3B4455;
--disabled-fill-color: #666;
}
.colab-df-quickchart {
background-color: var(--bg-color);
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: var(--fill-color);
height: 32px;
padding: 0;
width: 32px;
}
.colab-df-quickchart:hover {
background-color: var(--hover-bg-color);
box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: var(--button-hover-fill-color);
}
.colab-df-quickchart-complete:disabled,
.colab-df-quickchart-complete:disabled:hover {
background-color: var(--disabled-bg-color);
fill: var(--disabled-fill-color);
box-shadow: none;
}
.colab-df-spinner {
border: 2px solid var(--fill-color);
border-color: transparent;
border-bottom-color: var(--fill-color);
animation:
spin 1s steps(1) infinite;
}
@keyframes spin {
0% {
border-color: transparent;
border-bottom-color: var(--fill-color);
border-left-color: var(--fill-color);
}
20% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
}
30% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
border-right-color: var(--fill-color);
}
40% {
border-color: transparent;
border-right-color: var(--fill-color);
border-top-color: var(--fill-color);
}
60% {
border-color: transparent;
border-right-color: var(--fill-color);
}
80% {
border-color: transparent;
border-right-color: var(--fill-color);
border-bottom-color: var(--fill-color);
}
90% {
border-color: transparent;
border-bottom-color: var(--fill-color);
}
}
OpenAI
In [ ]python · cell 52
python
import openai
from bertopic.representation import OpenAI
prompt = """
I have a topic that contains the following documents:
[DOCUMENTS]
The topic is described by the following keywords: [KEYWORDS]
Based on the information above, extract a short topic label in the following format:
topic: <short topic label>
"""
# Update our topic representations using GPT-3.5
client = openai.OpenAI(api_key="YOUR_KEY_HERE")
representation_model = OpenAI(
client, model="gpt-3.5-turbo", exponential_backoff=True, chat=True, prompt=prompt
)
topic_model.update_topics(abstracts, representation_model=representation_model)
# Show topic differences
topic_differences(topic_model, original_topics)Output
100%|██████████| 156/156 [02:13<00:00, 1.17it/s]
Topic Original \
0 0 speech | asr | recognition | end | acoustic
1 1 medical | clinical | biomedical | patient | he...
2 2 sentiment | aspect | analysis | reviews | opinion
3 3 translation | nmt | machine | neural | bleu
4 4 summarization | summaries | summary | abstract...
Updated
0 Leveraging External Data for Improving Low-Res...
1 Improved Representation Learning for Biomedica...
2 "Advancements in Aspect-Based Sentiment Analys...
3 Neural Machine Translation Enhancements
4 Document Summarization Techniques
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
| Topic | Original | Updated | |
|---|---|---|---|
| 0 | 0 | speech | asr | recognition | end | acoustic | Leveraging External Data for Improving Low-Res... |
| 1 | 1 | medical | clinical | biomedical | patient | he... | Improved Representation Learning for Biomedica... |
| 2 | 2 | sentiment | aspect | analysis | reviews | opinion | "Advancements in Aspect-Based Sentiment Analys... |
| 3 | 3 | translation | nmt | machine | neural | bleu | Neural Machine Translation Enhancements |
| 4 | 4 | summarization | summaries | summary | abstract... | Document Summarization Techniques |
.colab-df-container {
display:flex;
gap: 12px;
}
.colab-df-convert {
background-color: #E8F0FE;
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: #1967D2;
height: 32px;
padding: 0 0 0 0;
width: 32px;
}
.colab-df-convert:hover {
background-color: #E2EBFA;
box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: #174EA6;
}
.colab-df-buttons div {
margin-bottom: 4px;
}
[theme=dark] .colab-df-convert {
background-color: #3B4455;
fill: #D2E3FC;
}
[theme=dark] .colab-df-convert:hover {
background-color: #434B5C;
box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);
filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));
fill: #FFFFFF;
}
.colab-df-quickchart {
--bg-color: #E8F0FE;
--fill-color: #1967D2;
--hover-bg-color: #E2EBFA;
--hover-fill-color: #174EA6;
--disabled-fill-color: #AAA;
--disabled-bg-color: #DDD;
}
[theme=dark] .colab-df-quickchart {
--bg-color: #3B4455;
--fill-color: #D2E3FC;
--hover-bg-color: #434B5C;
--hover-fill-color: #FFFFFF;
--disabled-bg-color: #3B4455;
--disabled-fill-color: #666;
}
.colab-df-quickchart {
background-color: var(--bg-color);
border: none;
border-radius: 50%;
cursor: pointer;
display: none;
fill: var(--fill-color);
height: 32px;
padding: 0;
width: 32px;
}
.colab-df-quickchart:hover {
background-color: var(--hover-bg-color);
box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);
fill: var(--button-hover-fill-color);
}
.colab-df-quickchart-complete:disabled,
.colab-df-quickchart-complete:disabled:hover {
background-color: var(--disabled-bg-color);
fill: var(--disabled-fill-color);
box-shadow: none;
}
.colab-df-spinner {
border: 2px solid var(--fill-color);
border-color: transparent;
border-bottom-color: var(--fill-color);
animation:
spin 1s steps(1) infinite;
}
@keyframes spin {
0% {
border-color: transparent;
border-bottom-color: var(--fill-color);
border-left-color: var(--fill-color);
}
20% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
}
30% {
border-color: transparent;
border-left-color: var(--fill-color);
border-top-color: var(--fill-color);
border-right-color: var(--fill-color);
}
40% {
border-color: transparent;
border-right-color: var(--fill-color);
border-top-color: var(--fill-color);
}
60% {
border-color: transparent;
border-right-color: var(--fill-color);
}
80% {
border-color: transparent;
border-right-color: var(--fill-color);
border-bottom-color: var(--fill-color);
}
90% {
border-color: transparent;
border-bottom-color: var(--fill-color);
}
}
In [ ]python · cell 53
python
# Visualize topics and documents
fig = topic_model.visualize_document_datamap(
titles,
topics=list(range(20)),
reduced_embeddings=reduced_embeddings,
width=1200,
label_font_size=11,
label_wrap_width=20,
use_medoids=True,
)
plt.savefig("datamapplot.png", dpi=300)Output
<Figure size 1200x1200 with 1 Axes>
[省略较大 image/png 输出]
BONUS: Word Cloud
Make sure to pip install wordcloud first in order to follow this bonus:
First, we need to make sure that each topic is described by a bit more words than just 10 as that would make for a much more interesting wordcloud.
In [ ]python · cell 56
python
topic_model.update_topics(abstracts, top_n_words=500)Then, we can run the following code to generate the wordcloud for our topic modeling topic:
In [ ]python · cell 58
python
from wordcloud import WordCloud
import matplotlib.pyplot as plt
def create_wordcloud(model, topic):
plt.figure(figsize=(10,5))
text = {word: value for word, value in model.get_topic(topic)}
wc = WordCloud(background_color="white", max_words=1000, width=1600, height=800)
wc.generate_from_frequencies(text)
plt.imshow(wc, interpolation="bilinear")
plt.axis("off")
plt.show()
# Show wordcloud
create_wordcloud(topic_model, topic=17)Output
<Figure size 1000x500 with 1 Axes>
[省略较大 image/png 输出]