Chapter 43
Introduction to Gemini Multimodal Embeddings
NotebookPython 348 cells
In [ ]python · cell 1
python
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.Introduction to Gemini Multimodal Embeddings
Objectives
In this notebook, you will explore:
- Multimodal Embeddings with Gemini Embeddings Model (Texts, Images, Video, Audio, and PDFs)
- Setting embeddings dimensionality (truncation)
- Building various use cases with embeddings:
- Product search with text and images
- Semantic Similarity Analysis with text
- Multimodal Document Similarity Analysis with PDFs
- Using
task_typefor optimized Retrieval Augmented Generation (RAG)
Gemini Multimodal Embeddings
Gemini Embedding 2 is Google's multimodal embedding model for high-performance embedding generation, specifically designed to power complex retrieval and advanced analytics.
The model processes a wide array of input modalities including text, images, documents, audio, and video to generate dense, 3072-dimensional vectors. These embeddings are positioned within a unified semantic space, ensuring that disparate data types with similar conceptual meanings are represented by mathematically proximal vectors.
This notebook demonstrates how to use the Multimodal Embeddings API for generating high-dimensional vector representations.
Use cases
- Image classification & search: Search relevant or similar images, or classify images based on labels.
- Video content search: Search relevant videos using text queries or similarity search.
- Recommendations: Generate product or advertisement recommendations based on multimodal data.
- Semantic Similarity: Compare the meaning of text fragments beyond keyword matching.
- Document Analysis: Identify documents with conceptually related content, regardless of formatting or layout.
- Retrieval Augmented Generation (RAG): Optimize retrieval for LLMs by using task-specific embeddings.
Getting Started
Install Google Gen AI SDK and other required packages
In [ ]python · cell 8
python
%pip install --upgrade --quiet google-genai numpy pandas seaborn scikit-learn pymupdfAuthenticate your notebook environment
If you are running this notebook in Google Colab, run the cell below to authenticate your account.
In [ ]python · cell 10
python
import sys
if "google.colab" in sys.modules:
from google.colab import auth
auth.authenticate_user()Set Google Cloud project information
To get started using Agent Platform, you must have an existing Google Cloud project and enable the Agent Platform API.
Learn more about setting up a project and a development environment.
In [ ]python · cell 12
python
import os
from google import genai
# fmt: off
PROJECT_ID = "[your-project-id]" # @param {type: "string", placeholder: "[your-project-id]", isTemplate: true}
LOCATION = "global" # @param {type: "string", placeholder: "[your-project-id]", isTemplate: true}
# fmt: on
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
PROJECT_ID = str(os.environ.get("GOOGLE_CLOUD_PROJECT"))
client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)Import libraries
In [ ]python · cell 14
python
import ast
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import pymupdf
import seaborn as sns
from IPython.display import Image, Markdown, Video, display
from google.genai.types import Content, EmbedContentConfig, Part
from sklearn.metrics.pairwise import cosine_similarity
pd.options.mode.chained_assignment = NoneLoad Embedding Model
In [ ]python · cell 16
python
# fmt: off
MODEL_ID = "gemini-embedding-2" # @param ["gemini-embedding-2"] {type: "string"}
# fmt: onGenerate Text Embeddings
Call the embed_content method with the model to generate text embeddings. You can embed a list of multiple prompts with one API call for efficiency.
In [ ]python · cell 19
python
response = client.models.embed_content(
model=MODEL_ID,
contents=[
"How do I get a driver's license?",
"What is the meaning of life?",
"How do I renew my driver's license?",
"How do I change my address on my driver's license?",
],
)
for i, embedding in enumerate(response.embeddings):
print(f"Embedding {i} length: {len(embedding.values)}")
print(f"First five values: {embedding.values[:5]}\n")Set embeddings dimensionality (Truncation)
The model is able to compress information into the earlier dimensions of the vector, allowing it to support lower embedding dimensions. Specify output_dimensionality to truncate the output.
- Default: 3072 dimensions.
- Reduced Dimensions: You can specify a lower value (e.g., 768 or even 10) to significantly reduce storage costs and increase search speed with minimal loss in accuracy.
In [ ]python · cell 21
python
text = ["Hello world"]
# Truncated to 768 dimensions
response = client.models.embed_content(
model=MODEL_ID,
contents=text,
config=EmbedContentConfig(output_dimensionality=768),
)
print(f"Truncated embedding length: {len(response.embeddings[0].values)}")Generate Multimodal Embeddings
The model supports the following formats:
- Text: Supports up to 8,192 tokens.
- Image: Maximum of 6 images per request. Supported formats: PNG, JPEG.
- PDF: Maximum of 6 pages.
- Audio: Maximum duration of 80 seconds. Supported formats: MP3, WAV.
- Video: Maximum duration of 128 seconds. Supported formats: MP4, MOV.
Embed Images
In [ ]python · cell 24
python
!wget -O cookies.png https://storage.googleapis.com/cloud-samples-data/generative-ai/image/cookies.png -q
with open("cookies.png", "rb") as f:
image_bytes = f.read()
response = client.models.embed_content(
model=MODEL_ID,
contents=[
Part.from_bytes(data=image_bytes, mime_type="image/png"),
],
)
print(f"Image embedding length: {len(response.embeddings[0].values)}")Embedding Aggregation
When working with multimodal content, how you structure your input affects the embedding output:
- Single content entry: Submitting multiple parts (e.g., text and an image) within a single
Contentobject produces one aggregated embedding for all modalities within that entry. - Multiple entries: Sending multiple entries in the
contentsarray returns separate embeddings for each entry. - Post-level representation: For complex objects like social media posts with multiple media items, we recommend aggregating separate embeddings (for example, by averaging) to create a coherent post-level representation.
Example of aggregated embedding (text + image):
In [ ]python · cell 26
python
response = client.models.embed_content(
model=MODEL_ID,
contents=[
Content(
parts=[
Part(text="An image of cookies"),
Part.from_bytes(data=image_bytes, mime_type="image/png"),
]
)
],
)
# This produces exactly ONE embedding
print(f"Aggregated embedding length: {len(response.embeddings[0].values)}")Embed Audio
In [ ]python · cell 28
python
AUDIO_URL = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/tell-a-story.wav"
response = client.models.embed_content(
model=MODEL_ID,
contents=[
Part.from_uri(file_uri=AUDIO_URL, mime_type="audio/wav"),
],
)
print(f"Audio embedding length: {len(response.embeddings[0].values)}")Embed Video
In [ ]python · cell 30
python
VIDEO_URL = (
"https://storage.googleapis.com/cloud-samples-data/generative-ai/video/animals.mp4"
)
response = client.models.embed_content(
model=MODEL_ID,
contents=[
Part.from_uri(file_uri=VIDEO_URL, mime_type="video/mp4"),
],
)
print(f"Video embedding length: {len(response.embeddings[0].values)}")Embed PDFs
PDF documents can be embedded directly. The model processes the visual and text content of each page.
[IMPORTANT] Page Limit Constraint: Currently, the model supports a maximum of 6 pages per PDF. If your document is longer, you must truncate the file before processing.
In [ ]python · cell 32
python
PDF_URL = "https://storage.googleapis.com/cloud-samples-data/generative-ai/pdf/w9.pdf"
!wget -q $PDF_URL -O sample.pdf
def truncate_pdf_bytes(file_path: str, max_pages: int = 6) -> bytes:
with pymupdf.open(file_path) as doc:
if doc.page_count <= max_pages:
return doc.tobytes()
doc.select(range(max_pages))
return doc.tobytes(garbage=3, deflate=True)
pdf_bytes = truncate_pdf_bytes("sample.pdf")
response = client.models.embed_content(
model=MODEL_ID,
contents=[
Part.from_bytes(data=pdf_bytes, mime_type="application/pdf"),
],
)
print(f"PDF embedding length: {len(response.embeddings[0].values)}")Use Cases
Find product images based on text search query
In [ ]python · cell 35
python
def load_and_prepare_data(url: str) -> pd.DataFrame:
df = pd.read_csv(url)
df["image_embeddings"] = df["image_embeddings"].apply(
lambda x: np.array(ast.literal_eval(x))
)
df["public_url"] = (
df["gcs_path"]
.str.replace("gs://", "https://storage.googleapis.com/", regex=False)
.str.replace(" ", "%20", regex=False)
)
return df
# get product list with pre-computed image embeddings
product_image_list = load_and_prepare_data(
"https://storage.googleapis.com/github-repo/embeddings/getting_started_embeddings/image_data_with_embeddings.csv"
)
def get_text_embedding(text: str, dimension: int = 768):
response = client.models.embed_content(
model=MODEL_ID,
contents=text,
config=EmbedContentConfig(output_dimensionality=dimension),
)
return np.array(response.embeddings[0].values)
def show_similar_images(query_emb: np.ndarray, df: pd.DataFrame, top_n: int = 5):
# Vectorized Dot Product (Matrix-Vector Multiplication)
# Stack all embeddings into one (N, Dimension) matrix
embeddings_matrix = np.stack(df["image_embeddings"].values)
# Calculate all scores at once using the @ operator (dot product)
df["score"] = embeddings_matrix @ query_emb
# Get Top N results efficiently
top_results = df.nlargest(top_n, "score")
# Display results
print(top_results[["score", "title"]])
for url in top_results["public_url"]:
display(Image(url=url, width=200))
query_emb = get_text_embedding("something related to dinosaurs theme")
show_similar_images(query_emb, product_image_list)Find videos based on text search query
In [ ]python · cell 37
python
video_list = pd.read_csv(
"https://storage.googleapis.com/github-repo/embeddings/getting_started_embeddings/video_data_with_embeddings.csv"
)
print(f"Items in the video list: {len(video_list)}")
video_list.head()In [ ]python · cell 38
python
def print_similar_videos(query_emb: list[float], data_frame: pd.DataFrame):
video_embs = data_frame["video_embeddings"]
scores = [np.dot(eval(video_emb), query_emb) for video_emb in video_embs]
data_frame["score"] = scores
data_frame = data_frame.sort_values(by="score", ascending=False)
# print results
print(data_frame.head()[["score", "file_name"]])
url = data_frame.iloc[0]["gcs_path"]
display(Video(url.replace("gs://", "https://storage.googleapis.com/")))
query_emb = get_text_embedding("A music concert")
print_similar_videos(query_emb, video_list)In [ ]python · cell 39
python
query_emb = get_text_embedding("A person doing TaiChi")
print_similar_videos(query_emb, video_list)Semantic Similarity Analysis
Semantic similarity analysis transforms data into numerical vectors that encapsulate their underlying meaning. In a high-dimensional vector space, semantically related items are represented by vectors that are mathematically close to one another.
This technique allows for comparison beyond keyword matching, capturing intent and context. For example:
- Thematic Clustering: Grouping content by topic regardless of format (e.g., a "programming" text and an "explainer video" on Python).
- Cosine Similarity: Measuring the angular distance between vectors to determine how "similar" two items are, where a score closer to 1.0 indicates higher semantic proximity.
In [ ]python · cell 41
python
text_examples = [
"I really enjoyed last night's movie",
"we watched a lot of acrobatic scenes yesterday",
"I had fun writing my first program in Python",
"huge sense of relief when my .py script finally ran without error",
"Oh Romeo, Romeo, wherefore art thou Romeo?",
]
df = pd.DataFrame(text_examples, columns=["text"])
df["embeddings"] = df.apply(lambda x: get_text_embedding(x.text), axis=1)
cos_sim_array = cosine_similarity(list(df.embeddings.values))
sim_df = pd.DataFrame(cos_sim_array, index=text_examples, columns=text_examples)
plt.figure(figsize=(8, 6))
ax = sns.heatmap(sim_df, annot=True, cmap="crest")
ax.xaxis.tick_top()
ax.set_xticklabels(text_examples, rotation=90)
plt.show()Multimodal Document Similarity Analysis
By measuring the angular distance (Cosine Similarity) between these vectors, you can identify documents with conceptually related content, regardless of differences in formatting or layout.
In [ ]python · cell 43
python
!wget -O google-q3-2025-report.pdf https://s206.q4cdn.com/479360582/files/doc_financials/2025/q3/2025q3-alphabet-earnings-release.pdf -q
!wget -O google-q4-2025-report.pdf https://s206.q4cdn.com/479360582/files/doc_financials/2025/q4/2025q4-alphabet-earnings-release.pdf -q
!wget -O meta-q4-2025-report.pdf https://s21.q4cdn.com/399680738/files/doc_financials/2025/q4/Earnings-Presentation-Q4-2025-FINAL.pdf -q
!wget -O gemini-2.5-report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf -q
docs = {
"Google Q3 2025": "google-q3-2025-report.pdf",
"Google Q4 2025": "google-q4-2025-report.pdf",
"Meta Q4 2025": "meta-q4-2025-report.pdf",
"Gemini 2.5 Tech Report": "gemini-2.5-report.pdf",
}
def get_pdf_embedding(file_path):
pdf_bytes = truncate_pdf_bytes(file_path, max_pages=6)
response = client.models.embed_content(
model=MODEL_ID,
contents=[Part.from_bytes(data=pdf_bytes, mime_type="application/pdf")],
)
return response.embeddings[0].values
doc_embeddings = {title: get_pdf_embedding(path) for title, path in docs.items()}
titles = list(doc_embeddings.keys())
embeddings = list(doc_embeddings.values())
sim_matrix = cosine_similarity(embeddings)
sim_docs_df = pd.DataFrame(sim_matrix, index=titles, columns=titles)
plt.figure(figsize=(8, 6))
sns.heatmap(sim_docs_df, annot=True, cmap="crest")
plt.show()Using task_type for different scenarios
When generating embeddings, especially for complex systems like Retrieval Augmented Generation (RAG), a one-size-fits-all approach can sometimes lead to lower-quality results. For instance, a question like "Why is the sky blue?" and its answer, "The scattering of sunlight causes the blue color," have distinct meanings as statements, and a general-purpose embedding model might not recognize their strong relationship.
To solve this, you can specify a task_type when creating embeddings. This instructs the model to produce vectors that are specifically optimized for your intended use case, which can significantly improve performance while also saving time and cost.
SEMANTIC_SIMILARITY: Compares how similar two pieces of text are in meaning.RETRIEVAL_QUERYandRETRIEVAL_DOCUMENT: Used for search systems. UseQUERYfor the user's question andDOCUMENTfor the content you are searching through. This is the foundation for building effective semantic search and RAG systems.CLASSIFICATION: Used when you need to sort text into specific, pre-defined categories (like labeling emails as "Spam" or "Not Spam").CLUSTERING: Groups similar texts together to find hidden patterns or topics when you don't have pre-set labels.
In [ ]python · cell 45
python
docs_df = pd.read_json(
"https://storage.googleapis.com/github-repo/embeddings/google-car.json"
)
# Generate retrieval embeddings for documents
docs_df["embeddings"] = docs_df.apply(
lambda x: (
client.models.embed_content(
model=MODEL_ID,
contents=x.contents,
config=EmbedContentConfig(task_type="RETRIEVAL_DOCUMENT"),
)
.embeddings[0]
.values
),
axis=1,
)
def query_docs(query, df):
query_emb = (
client.models.embed_content(
model=MODEL_ID,
contents=query,
config=EmbedContentConfig(task_type="RETRIEVAL_QUERY"),
)
.embeddings[0]
.values
)
scores = np.dot(np.stack(df.embeddings), query_emb)
best_idx = np.argmax(scores)
return df.contents.iloc[best_idx]
question = "how to make the fan speed stronger?"
context = query_docs(question, docs_df)
print(f"Best matching passage: {context}")Now you can do one augmented generation (the last step of the RAG process) using the best passage found by the first step, but still having custom answers for users instead of simply pasting large documents chunks directly:
In [ ]python · cell 47
python
final_answer_prompt = f"""
Your Role: You are a friendly AI assistant. Your purpose is to explain information to users who are not experts.
Your Task: Use the provided "Source Text" below to answer the user's question.
Guidelines for your Response:
Be Clear and Simple: Explain any complicated ideas in easy-to-understand terms. Avoid jargon.
Be Friendly: Write in a warm, conversational, and approachable tone.
Be Thorough: Construct a complete and detailed answer in full sentences, using all the relevant information from the source text.
Stay on Topic: If the source text does not contain the answer, you must state that the information is not available in the provided material. Do not use outside knowledge.
QUESTION: {question}
PASSAGE: {context}
"""
# fmt: off
GEMINI_MODEL_ID = "gemini-3.6-flash" # @param ["gemini-2.5-flash-lite", "gemini-2.5-flash", "gemini-3.5-flash", "gemini-3.1-flash-lite", "gemini-3.1-pro-preview"] {"allow-input":true, isTemplate: true}
# fmt: on
final_answer = client.models.generate_content(
model=GEMINI_MODEL_ID,
contents=final_answer_prompt,
)
display(Markdown(final_answer.text))What's next?
- Learn how to store the vectors (embeddings) into Vector Search: Notebook
- Learn how to tune the embeddings with your own data: Notebook
- Check out the Gemini Embedding 2 documentation for detailed reference.