Chapter 78
Intro to Gemini Agentic Vision
NotebookPython 321 cells
In [ ]python · cell 1
python
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.Intro to Gemini Agentic Vision
| Author |
|---|
| Eric Dong |
Overview
Gemini's code execution tool allows the model to generate and run Python code, enabling applications that benefit from code-based reasoning.
This capability unlocks Agentic Vision in models like Gemini 3 Flash. By enabling code execution, the model acts as an agent: instead of just analyzing a static image, it can write code to actively manipulate, crop, and inspect the image to find details that might otherwise be missed.
This feature is useful for building applications such as:
- Zoom and Inspect: Implicitly detecting when an object is too small and cropping the image to "zoom in" for a better look.
- Visual Math and Plotting: Performing precise multi-step calculations or re-plotting data accurately.
- Image Annotation: Identifying and bounding objects programmatically.
Getting Started
Install Google Gen AI SDK for Python
In [ ]python · cell 7
python
%pip install --upgrade --quiet google-genaiImport libraries
In [3]python · cell 9
python
import io
import os
import sys
import requests
from IPython.display import display
from PIL import Image
from google import genai
from google.genai import typesAuthenticate your notebook environment
If you are running this notebook on Google Colab, run the cell below to authenticate your environment.
In [4]python · cell 11
python
if "google.colab" in sys.modules:
from google.colab import auth
auth.authenticate_user()Authenticate your Google Cloud Project for Vertex AI
You can use a Google Cloud Project or an API Key for authentication. This tutorial uses a Google Cloud Project.
In [ ]python · cell 13
python
# fmt: off
PROJECT_ID = "[your-project-id]" # @param {type: "string", placeholder: "[your-project-id]", isTemplate: true}
# fmt: on
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
PROJECT_ID = str(os.environ.get("GOOGLE_CLOUD_PROJECT"))
LOCATION = "global"
client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)Choose a Gemini model
Use gemini-3.6-flash in this tutorial. Learn more about all Gemini models on Vertex AI.
In [6]python · cell 15
python
# fmt: off
MODEL_ID = "gemini-3.6-flash" # @param ["gemini-3.6-flash"] {type: "string"}
# fmt: onExample: Zoom and Inspect
In this example, when asked to count details that are small or hard to see (like expression pedals on an organ), the model recognizes that it cannot answer accurately from the full-resolution image.
Instead of guessing, it:
- Reasons that it needs a closer look.
- Writes Python code to crop the specific area of interest.
- Executes the code to generate a new, "zoomed-in" image.
- Inspects the new image to provide an accurate answer.
Example:
- Prompt: Locate the ESMT chip. What are the numbers on the chip?
- Image: https://storage.googleapis.com/cloud-samples-data/generative-ai/image/chips.jpeg
In [7]python · cell 17
python
# Download the input image
image_path = (
"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/chips.jpeg"
)
image_bytes = requests.get(image_path).content
image = types.Part.from_bytes(data=image_bytes, mime_type="image/jpeg")
response = client.models.generate_content(
model=MODEL_ID,
contents=[image, "Locate the ESMT chip. What are the numbers on the chip?"],
config=types.GenerateContentConfig(
tools=[types.Tool(code_execution=types.ToolCodeExecution)]
),
)
# Parse the multi-part response
for part in response.candidates[0].content.parts:
# 1. The model's reasoning text
if part.text is not None:
print(part.text)
# 2. The code the model generated to solve the problem
if part.executable_code is not None:
print("\n--- Generated Code ---")
print(part.executable_code.code)
# 3. The output of the code execution
if part.code_execution_result is not None:
print("\n--- Execution Output ---")
print(part.code_execution_result.output)
# 4. Any resulting images generated by the code (e.g., crops)
if part.as_image() is not None:
display(Image.open(io.BytesIO(part.as_image().image_bytes)))Output
--- Generated Code ---
import PIL.Image
import PIL.ImageDraw
# Load the image to get dimensions
img = PIL.Image.open('input_file_0.jpeg')
width, height = img.size
# ESMT chip is roughly around [280, 160, 410, 310] in normalized coordinates
# ESMT chip center is approximately [0.34 * height, 0.23 * width]
# Let's refine the bounding box for the ESMT chip.
# Looking at the original image:
# y is from roughly 275 to 408
# x is from roughly 165 to 308
esmt_chip_bbox = [270, 160, 415, 315] # [ymin, xmin, ymax, xmax]
# Convert normalized to pixel coordinates
def norm_to_pixel(norm_bbox, w, h):
ymin, xmin, ymax, xmax = norm_bbox
return [int(ymin * h / 1000), int(xmin * w / 1000), int(ymax * h / 1000), int(xmax * w / 1000)]
pixel_bbox = norm_to_pixel(esmt_chip_bbox, width, height)
crop_img = img.crop((pixel_bbox[1], pixel_bbox[0], pixel_bbox[3], pixel_bbox[2]))
crop_img.save('esmt_chip_zoom.png')
# Output detection for internal reference
print(f'# [{{box_2d: {esmt_chip_bbox}, label: "esmt chip"}}]')
--- Execution Output ---
# [{box_2d: [270, 160, 415, 315], label: "esmt chip"}]
<PIL.PngImagePlugin.PngImageFile image mode=RGB size=387x272>
[省略较大 image/png 输出]
--- Generated Code ---
# Rotate the zoomed image 180 degrees to read more easily
crop_img_rotated = crop_img.rotate(180)
crop_img_rotated.save('esmt_chip_zoom_rotated.png')
--- Execution Output ---
None
<PIL.PngImagePlugin.PngImageFile image mode=RGB size=387x272>
[省略较大 image/png 输出]
Based on the visual inspection of the ESMT chip, the numbers and codes printed on it are: * **ESMT** (Manufacturer: Elite Semiconductor Memory Technology) * **M12L64164A** (Part number) * **7T** (Speed grade or variant) * **SZB1C30C9** (Batch or internal code) * **0325** (Date code, likely 25th week of 2003)
Example: Visual Math and Plotting
While standard multimodal models can describe charts, they often struggle with precise multi-step calculations or re-plotting data accurately.
With Agentic Vision, the model acts as a data analyst:
- Extracts Data: It reads the raw values from the visual chart into a structured format (like a Python list or dictionary).
- Computes: It writes Python code to perform the requested math (normalization and averaging) with perfect arithmetic precision.
- Visualizes: It uses libraries like
matplotlibto generate a completely new, accurate chart based on the calculated data.
Example:
- Prompt: Make a bar chart of per-category performance, normalize prior SOTA as 1.0 for each task, then take average per-category. Plot using matplotlib with nice style.
- Image: https://storage.googleapis.com/cloud-samples-data/generative-ai/image/benchmark.jpeg
In [ ]python · cell 19
python
# Use to the benchmark image in Cloud Storage
image = types.Part.from_uri(
file_uri="https://storage.googleapis.com/cloud-samples-data/generative-ai/image/benchmark.jpeg",
mime_type="image/jpeg",
)
response = client.models.generate_content(
model=MODEL_ID,
contents=[
image,
"Make a bar chart of per-category performance, normalize prior SOTA as 1.0 for each task, then take average per-category. Plot using matplotlib with nice style.",
],
config=types.GenerateContentConfig(
tools=[types.Tool(code_execution=types.ToolCodeExecution)]
),
)
# Parse the multi-part response with comments
for part in response.candidates[0].content.parts:
# 1. The model's reasoning text (e.g., "I need to extract the data...")
if part.text is not None:
print(part.text)
# 2. The executable code generated by the model
if part.executable_code is not None:
print("\n--- Generated Code ---")
print(part.executable_code.code)
# 3. The text output of the code execution
if part.code_execution_result is not None:
print("\n--- Execution Output ---")
print(part.code_execution_result.output)
# 4. Any resulting charts/images generated by the code
if part.as_image() is not None:
display(Image.open(io.BytesIO(part.as_image().image_bytes)))Output
--- Generated Code ---
import matplotlib.pyplot as plt
import numpy as np
# Data dictionary: Category -> [ (Benchmark, G3Pro, G2.5Pro, Opus4.5, GPT5.1, lower_is_better) ]
data = {
'Visual Reasoning': [
('MMMU Pro', 81.0, 68.0, 72.0, 76.0, False),
('VLMsAreBiased', 50.6, 24.3, 32.7, 21.7, False)
],
'Document': [
('CharXiv Reasoning', 81.4, 69.6, 67.2, 69.5, False),
('OmniDocBench1.5*', 0.115, 0.145, 0.120, 0.147, True)
],
'Spatial': [
('ERQA', 70.5, 56.0, 51.3, 60.0, False),
('Point-Bench', 85.5, 62.7, 38.5, 41.8, False),
('RefSpatial', 65.5, 33.6, 19.5, 28.2, False),
('CV-Bench', 92.0, 85.9, 83.8, 84.6, False),
('MindCube', 77.7, 57.5, 58.5, 61.7, False)
],
'Screen': [
('ScreenSpot Pro', 72.7, 11.4, 49.9, 3.5, False),
('Gui-World QA', 68.0, 42.8, 44.9, 38.7, False)
],
'Video': [
('Video-MMMU', 87.6, 83.6, 84.4, 80.4, False),
('Video-MME', 88.4, 86.9, 84.1, 86.3, False),
('1H-VideoQA', 81.8, 79.4, 52.0, 61.5, False),
('Perception Test', 80.0, 78.4, 74.1, 77.8, False),
('YouCook2', 222.7, 188.3, 145.8, 132.4, False),
('Vatex', 77.4, 71.3, 60.1, 62.9, False),
('Motion Bench', 70.3, 66.3, 65.9, 61.1, False)
],
'Education': [
('Math Kangaroo', 84.4, 77.4, 68.9, 79.9, False)
],
'Biomedical': [
('MedXpertQA-MM', 77.8, 65.9, 62.2, 65.5, False),
('VQA-RAD', 81.9, 71.4, 76.0, 72.2, False),
('MicroVQA', 68.8, 63.5, 61.4, 61.5, False)
]
}
category_performance = {}
for category, benchmarks in data.items():
ratios = []
for name, g3p, g25p, opus, gpt, lower_better in benchmarks:
others = [g25p, opus, gpt]
if lower_better:
prior_sota = min(others)
ratio = prior_sota / g3p
else:
prior_sota = max(others)
ratio = g3p / prior_sota
ratios.append(ratio)
category_performance[category] = np.mean(ratios)
# Sort categories for consistent plotting if needed, but keeping original order from image might be better
categories = list(data.keys())
values = [category_performance[cat] for cat in categories]
# Plotting
plt.figure(figsize=(10, 6))
plt.style.use('seaborn-v0_8-muted') # Nice style
bars = plt.bar(categories, values, color='skyblue', edgecolor='navy', alpha=0.8)
# Add a horizontal line at 1.0 for prior SOTA
plt.axhline(y=1.0, color='red', linestyle='--', label='Prior SOTA (1.0)')
# Add labels and title
plt.ylabel('Average Performance Relative to Prior SOTA')
plt.title('Gemini 3 Pro Performance per Category (Normalized to Prior SOTA=1.0)', fontsize=14, fontweight='bold')
plt.ylim(0, max(values) * 1.2) # Give some space for labels
plt.xticks(rotation=15)
# Add value labels on top of bars
for bar in bars:
yval = bar.get_height()
plt.text(bar.get_x() + bar.get_width()/2, yval + 0.02, f'{yval:.2f}x', ha='center', va='bottom', fontweight='bold')
plt.legend()
plt.tight_layout()
plt.savefig('performance_chart.png')
plt.show()
--- Execution Output ---
None
<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=2000x1200>
[省略较大 image/png 输出]
<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=1000x600>
Based on the data provided in the benchmark table, I have calculated the performance of Gemini 3 Pro relative to the prior state-of-the-art (SOTA) for each task. The prior SOTA for each task was determined by taking the best performing model among Gemini 2.5 Pro, Claude Opus 4.5, and GPT-5.1. For the "OmniDocBench1.5*" benchmark, where a lower score is better, the relative performance was calculated as `Prior SOTA / Gemini 3 Pro`. After normalizing each task's performance to 1.0 (representing the best among other models), I calculated the average for each of the 7 categories. ### Gemini 3 Pro: Per-Category Average Performance vs. Prior SOTA - **Screen:** **1.49x** (49% improvement) - **Spatial:** **1.36x** (36% improvement) - **Visual Reasoning:** **1.31x** (31% improvement) - **Biomedical:** **1.11x** (11% improvement) - **Document:** **1.11x** (11% improvement) - **Education:** **1.06x** (6% improvement) - **Video:** **1.06x** (6% improvement) The resulting bar chart below visualizes these improvements across all categories: 
Example: Image Annotation & Spatial Reasoning
In this scenario, the model demonstrates spatial reasoning by modifying the pixel data directly. Instead of just describing where objects should go, it acts as an intelligent image editor.
The model:
- Identifies the objects (trash items) and targets (recycling bins) within the image.
- Reasons about the correct category for each item (e.g., paper vs. plastic).
- Calculates Coordinates: It determines the start and end points for the arrows.
- Draws: It uses the
PIL(Python Imaging Library) to draw colored arrows on the image, creating a visual guide that solves the user's problem.
Example:
- Prompt: Annotate on the image with arrows of different colors, which object should go into which bin.
- Image: https://storage.googleapis.com/cloud-samples-data/generative-ai/image/robotic.jpeg
In [ ]python · cell 21
python
# Use the robotic arm image in Cloud Storage
image = types.Part.from_uri(
file_uri="https://storage.googleapis.com/cloud-samples-data/generative-ai/image/robotic.jpeg",
mime_type="image/jpeg",
)
response = client.models.generate_content(
model=MODEL_ID,
contents=[
image,
"Annotate on the image with arrows of different colors, which object should go into which bin.",
],
config=types.GenerateContentConfig(
tools=[types.Tool(code_execution=types.ToolCodeExecution)]
),
)
# Parse the multi-part response with comments
for part in response.candidates[0].content.parts:
# 1. The model's reasoning text
if part.text is not None:
print(part.text)
# 2. The executable code generated by the model
if part.executable_code is not None:
print("\n--- Generated Code ---")
print(part.executable_code.code)
# 3. The text output of the code execution
if part.code_execution_result is not None:
print("\n--- Execution Output ---")
print(part.code_execution_result.output)
# 4. The final annotated image generated by the code
if part.as_image() is not None:
display(Image.open(io.BytesIO(part.as_image().image_bytes)))Output
--- Generated Code ---
import PIL.Image
import PIL.ImageDraw
# Load the image to get its dimensions
img = PIL.Image.open('f_https___storage.googleapis.com_cloud_samples_data_generative_ai_image_robotic.jpeg')
width, height = img.size
# Define objects and bins with normalized coordinates (ymin, xmin, ymax, xmax)
# Bins
light_blue_bin = [120, 308, 340, 437]
green_bin = [248, 677, 459, 830]
black_bin = [645, 407, 902, 578]
# Objects to go into Green bin (Organics)
green_pepper = [255, 482, 296, 545]
red_pepper = [316, 479, 348, 543]
grapes = [582, 554, 666, 594]
cherries = [463, 670, 513, 719]
# Objects to go into Light Blue bin (Recycling)
soda_can = [395, 523, 490, 606]
# Objects to go into Black bin (Waste/Landfill)
napkin = [177, 563, 252, 609]
cup = [270, 587, 346, 642]
choc_wrapper = [395, 421, 479, 504]
fruit_snack = [519, 464, 602, 544]
def denormalize(box, width, height):
return [box[0] * height / 1000, box[1] * width / 1000, box[2] * height / 1000, box[3] * width / 1000]
def get_center(box):
return [(box[1] + box[3]) / 2, (box[0] + box[2]) / 2]
# Map objects to bins
mapping = [
(green_pepper, green_bin, 'green'),
(red_pepper, green_bin, 'green'),
(grapes, green_bin, 'green'),
(cherries, green_bin, 'green'),
(soda_can, light_blue_bin, 'blue'),
(napkin, black_bin, 'gray'),
(cup, black_bin, 'gray'),
(choc_wrapper, black_bin, 'gray'),
(fruit_snack, black_bin, 'gray'),
]
draw = PIL.ImageDraw.Draw(img)
for obj_box, bin_box, color in mapping:
obj_denorm = denormalize(obj_box, width, height)
bin_denorm = denormalize(bin_box, width, height)
start_point = get_center(obj_denorm)
end_point = get_center(bin_denorm)
# Draw arrow (line + simple head)
draw.line([tuple(start_point), tuple(end_point)], fill=color, width=5)
# Simple arrowhead
import math
angle = math.atan2(end_point[1] - start_point[1], end_point[0] - start_point[0])
arrow_len = 20
arrow_point1 = (end_point[0] - arrow_len * math.cos(angle - math.pi / 6),
end_point[1] - arrow_len * math.sin(angle - math.pi / 6))
arrow_point2 = (end_point[0] - arrow_len * math.cos(angle + math.pi / 6),
end_point[1] - arrow_len * math.sin(angle + math.pi / 6))
draw.polygon([tuple(end_point), arrow_point1, arrow_point2], fill=color)
img.save('annotated_image.png')
# Output detections for reference
# [{box_2d: [255, 482, 296, 545], label: "green pepper"},
# {box_2d: [316, 479, 348, 543], label: "red pepper"},
# {box_2d: [582, 554, 666, 594], label: "grapes"},
# {box_2d: [463, 670, 513, 719], label: "cherries"},
# {box_2d: [395, 523, 490, 606], label: "soda can"},
# {box_2d: [177, 563, 252, 609], label: "napkin"},
# {box_2d: [270, 587, 346, 642], label: "cup"},
# {box_2d: [395, 421, 479, 504], label: "chocolate wrapper"},
# {box_2d: [519, 464, 602, 544], label: "fruit snack packet"},
# {box_2d: [120, 308, 340, 437], label: "light blue bin"},
# {box_2d: [248, 677, 459, 830], label: "green bin"},
# {box_2d: [645, 407, 902, 578], label: "black bin"}]
--- Execution Output ---
None
<PIL.PngImagePlugin.PngImageFile image mode=RGB size=1184x864>
[省略较大 image/png 输出]
[省略较大 image/jpeg 输出]
The image has been annotated with arrows showing the suggested disposal for each object: - **Green arrows** point food items (red pepper, green pepper, grapes, and cherries) toward the **green bin**, typically used for organic waste. - A **blue arrow** points the crushed soda can toward the **light blue bin**, generally intended for recyclables. - **Gray arrows** point non-recyclable or soiled waste (napkin, plastic cup, chocolate wrapper, and fruit snack packet) toward the **black bin**, which is commonly for general waste.