Generative AI on Google Cloud

31 / 120·notebook

Chapter 31

Gemini 3.5 Transcribe

NotebookPython 360 cells

In [ ]python · cell 1

python

# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Gemini 3.5 Transcribe

Author
Katie Nguyen

Overview

This notebook introduces Gemini 3.5 Transcribe, Google's model for converting speech to text in multiple languages, available through Agent Platform.

In this tutorial, you'll learn how to use Gemini 3.5 Transcribe with the Google Gen AI SDK, through both the synchronous generateContent API and the streaming BidiGenerateContent (Live) API, to:

Synchronous:

Transcribe an audio file with automatic language recognition
Generate word-level timestamps for a transcription
Transcribe an audio file using language codes
Transcribe an audio file with speaker diarization
Bias transcription output with custom vocabulary
Transcribe long audio files

Streaming:

Stream transcription with automatic language recognition
Stream transcription using language codes
Bias streaming transcription with custom vocabulary
Stream transcription for long audio files
Stream transcription from a microphone

Get started

Install Google Gen AI SDK for Python & other libraries

Install the Google Gen AI SDK along with a few supporting libraries used later in this notebook:

ipywebrtc: captures microphone audio directly in the notebook
soundfile: reads audio files as raw PCM frames for streaming
numpy: downmixes multi-channel audio to mono before streaming
pydub: converts and splits (chunks) audio files

In [ ]python · cell 7

python

%pip install --upgrade --quiet google-genai ipywebrtc soundfile numpy pydub> /dev/null 2>&1

Authenticate your notebook environment (Colab only)

If you are running this notebook on Google Colab, run the following cell to authenticate your environment.

In [ ]python · cell 9

python

import sys

if "google.colab" in sys.modules:
    from google.colab import auth

    auth.authenticate_user()

Import libraries

In [ ]python · cell 11

python

import warnings

warnings.filterwarnings("ignore", category=SyntaxWarning)

In [ ]python · cell 12

python

import asyncio
import os
import numpy as np
import soundfile as sf

from IPython.display import Audio, display, Markdown
from google import genai
from google.genai import types
from pydub import AudioSegment
from ipywebrtc import AudioRecorder, CameraStream

Set Google Cloud project information

To get started using Agent Platform, you must have an existing Google Cloud project and enable the Agent Platform API.

Learn more about setting up a project and a development environment.

In [ ]python · cell 14

python

# fmt: off
PROJECT_ID = "[your-project-id]"  # @param {type: "string", placeholder: "[your-project-id]", isTemplate: true}
# fmt: on
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    PROJECT_ID = str(os.environ.get("GOOGLE_CLOUD_PROJECT"))

LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "global")

client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)

Load the Gemini 3.5 Transcribe model

Define the model IDs used throughout this notebook:

MODEL_ID: used with client.models.generate_content() to transcribe a complete, pre-recorded audio file in a single request. (Synchronous)
MODEL_ID_LIVE: used with client.aio.live.connect() to stream audio and receive transcription results incrementally, in real time. (Streaming)

In [ ]python · cell 16

python

MODEL_ID = "gemini-3.5-transcribe-preview"
MODEL_ID_LIVE = "gemini-3.5-transcribe-live-preview"

Transcribe audio files (Synchronous)

Synchronous: These examples use the standard generate_content method to transcribe complete audio files that have already been recorded, returning the full transcript in a single response. Each example builds an AudioTranscriptionConfig inside GenerateContentConfig to demonstrate a different transcription parameter.

Transcription with automatic language recognition and timestamps

For this first request, run the following cell to download and play the audio you'll be transcribing. If you'd like to use a different audio clip, modify the audio_file_url variable below.

The request that follows sets word_timestamp=True to return word-level timing. The response's audio_transcription.words list contains each recognized word along with its start_offset and end_offset, while the full transcript is available as concatenated text across the response parts.

In [ ]python · cell 19

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/tell-a-story.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 20

python

with open(audio_file, "rb") as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type="audio/wav",
        ),
    ],
    config=types.GenerateContentConfig(
        audio_transcription_config=types.AudioTranscriptionConfig(
            word_timestamp=True,
        ),
    ),
)

parts = getattr(response, "parts", []) or []
if parts and (audio_tx := getattr(parts[0], "audio_transcription", None)):
    for w in getattr(audio_tx, "words", []) or []:
        display(Markdown(f"[{w.start_offset} - {w.end_offset}] {w.word}"))

if text := "".join(p.text for p in parts if getattr(p, "text", None)):
    display(Markdown(f"**{text}**"))

Transcription with language codes

Run the following cell to download and play a new audio clip.

In [ ]python · cell 22

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/spanish.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

Instead of relying on automatic detection, you can provide language_codes with one or more expected language codes. This request also demonstrates types.Part.from_uri(), which references audio already stored in Cloud Storage by its gs:// URI instead of uploading local bytes with types.Part.from_bytes(). The audio is still downloaded locally in the previous cell so you can play it back in the notebook, but the request itself streams directly from Cloud Storage.

In [ ]python · cell 24

python

with open(audio_file, "rb") as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_uri(
            file_uri="gs://cloud-samples-data/generative-ai/audio/spanish.wav",
            mime_type="audio/wav",
        ),
    ],
    config=types.GenerateContentConfig(
        audio_transcription_config=types.AudioTranscriptionConfig(
            language_codes=["es-ES"],
        ),
    ),
)

parts = getattr(response, "parts", []) or []
if text := "".join(p.text for p in parts if getattr(p, "text", None)):
    display(Markdown(text))

Transcription with speaker diarization

Run the following cell to download and play a new audio clip.

In [ ]python · cell 26

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/hobbies.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

Setting diarization=True asks the model to identify and label individual speakers. Each response part's audio_transcription.speaker_label indicates which speaker the accompanying text belongs to, so you can group and print the transcript by speaker.

In [ ]python · cell 28

python

with open(audio_file, "rb") as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type="audio/wav",
        ),
    ],
    config=types.GenerateContentConfig(
        audio_transcription_config=types.AudioTranscriptionConfig(
            diarization=True,
        ),
    ),
)

parts = getattr(response, "parts", []) or []
for p in parts:
    audio_tx = getattr(p, "audio_transcription", None)
    speaker = getattr(audio_tx, "speaker_label", "UNKNOWN") if audio_tx else "UNKNOWN"
    text = getattr(p, "text", "") or (getattr(audio_tx, "text", "") if audio_tx else "")

    if text:
        display(Markdown(f"**{speaker}**: {text}"))

Transcription with custom vocabulary

Run the following cell to download and play a new audio clip.

In [ ]python · cell 30

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/coffee_order.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

The custom_vocabulary field takes a list of phrases that bias Gemini 3.5 Transcribe toward recognizing specific terms, such as product names or brand-specific spellings (here, "oatmilk" and "oz").

Note: The model generally follows custom vocabulary instructions more reliably when a language is also specified.

In [ ]python · cell 32

python

with open(audio_file, "rb") as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type="audio/wav",
        ),
    ],
    config=types.GenerateContentConfig(
        audio_transcription_config=types.AudioTranscriptionConfig(
            language_codes=["en-US"],
            custom_vocabulary=["oatmilk", "oz"],
        ),
    ),
)
parts = getattr(response, "parts", []) or []

if text := "".join(p.text for p in parts if getattr(p, "text", None)):
    display(Markdown(text))

Long audio files for batch transcription

The generate_content transcription requests currently support audio clips up to 1 hour.

In [ ]python · cell 34

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/pixel.mp3"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 35

python

with open(audio_file, "rb") as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type="audio/wav",
        ),
    ],
    config=types.GenerateContentConfig(
        audio_transcription_config=types.AudioTranscriptionConfig(),
    ),
)

parts = getattr(response, "parts", []) or []
if text := "".join(p.text for p in parts if getattr(p, "text", None)):
    display(Markdown(text))

Transcribe streaming audio (Streaming)

Streaming: The remaining examples use the BidiGenerateContent (Live) API instead of generate_content. Rather than waiting for a complete audio file and returning one final transcript, a Live session stays open while you stream small chunks of audio to the model and receive transcription results incrementally, as they become available. This is the approach you'd use for near real-time captioning or transcribing microphone input.

Define helper functions for streaming transcription

The following helper functions manage a streaming transcription session and are reused throughout the rest of this notebook:

send_streaming_audio(session, audio_file): reads the audio file in small blocks with soundfile, downmixes multi-channel audio to mono, and sends each block to the Live session as raw PCM audio (audio/pcm;rate=<sample_rate>) via session.send_realtime_input(). A short delay between blocks paces the upload to roughly match real-time playback. Once the file has been fully sent, it signals audio_stream_end=True.
receive_streaming_messages(session, interim_display, transcript_buffer): listens for messages from the session. Partial results arrive as interim_input_transcription and update the live display in place; once a segment is finalized, it arrives as input_transcription and is appended to transcript_buffer.
streaming_main(audio_file, config, interim_display): opens a Live session with client.aio.live.connect(model=MODEL_ID_LIVE, config=config), confirms the server's setup_complete message before streaming any audio, then runs the send and receive helpers concurrently, giving the receiver a few extra seconds to process any trailing messages after the audio finishes sending.

In [ ]python · cell 38

python

async def send_streaming_audio(session, audio_file):
    try:
        with sf.SoundFile(audio_file, mode='r') as f:
            sample_rate = f.samplerate

            chunk_duration = 0.1
            chunk_frames = int(sample_rate * chunk_duration)
            pacing_delay = chunk_duration
            mime_type = f"audio/pcm;rate={sample_rate}"

            for block in f.blocks(blocksize=chunk_frames, dtype='int16'):
                if f.channels > 1:
                    # Average the channels together and cast back to 16-bit integers
                    block = np.mean(block, axis=1).astype(np.int16)
                data = block.tobytes()
                await session.send_realtime_input(audio=types.Blob(data=data, mime_type=mime_type))
                await asyncio.sleep(pacing_delay)

    except Exception as e:
        print(f"Error reading/streaming wav file: {e}")
    finally:
        await session.send_realtime_input(audio_stream_end=True)


async def receive_streaming_messages(session, interim_display, transcript_buffer):
    current_interim = ""
    try:
        async for message in session.receive():
            if message.server_content:
                server_content = message.server_content

                # Track the active interim segment
                interim = server_content.interim_input_transcription
                if interim and interim.text:
                    current_interim = interim.text

                    # Update the temporary live-streaming view
                    active_view = " ".join(transcript_buffer + [current_interim])
                    interim_display.update(Markdown(f"{active_view}"))

                # Save final transcript and clear the interim
                final = server_content.input_transcription
                if final and final.text:
                    transcript_buffer.append(final.text)
                    current_interim = ""

                    active_view = " ".join(transcript_buffer)
                    interim_display.update(Markdown(f"{active_view}"))

    except Exception as e:
        print(f"Error transcribing file: {e}")


async def streaming_main(audio_file, config, interim_display):
    transcript_buffer = []

    async with client.aio.live.connect(model=MODEL_ID_LIVE, config=config) as session:
        if session.setup_complete is None:
            print("No setup_complete received from server.")
            return
        send_task = asyncio.create_task(send_streaming_audio(session, audio_file))
        recv_task = asyncio.create_task(receive_streaming_messages(session, interim_display, transcript_buffer))

        await send_task
        try:
            await asyncio.wait_for(recv_task, timeout=5.0)
        except asyncio.TimeoutError:
            recv_task.cancel()

Transcription with automatic language recognition

To transcribe streaming audio, build a LiveConnectConfig with the following fields:

response_modalities: Since you want transcribed text back, set this to ["TEXT"].
input_audio_transcription: An AudioTranscriptionConfig describing how to transcribe the incoming audio.

Run the following cell to download and play the audio you'll be streaming, then run the cell after it to start the streaming session.

In [ ]python · cell 40

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/tell-a-story.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 41

python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(),
)
interim_display = display(Markdown(""), display_id=True)
await streaming_main(audio_file, config, interim_display)

Transcription with a specific language

Once again, run the following cell to download and play the audio you'll be transcribing.

In [ ]python · cell 43

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/korean-english.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

This time, build a new config and add a language_codes parameter inside AudioTranscriptionConfig containing one or more expected language codes. The audio in this example mixes Korean and English, so both ko-KR and en-US are provided as codes.

In [ ]python · cell 45

python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(
        language_codes=["ko-KR", "en-US"],
    ),
)
interim_display = display(Markdown(""), display_id=True)
await streaming_main(audio_file, config, interim_display)

Transcription with custom vocabulary

Run the following cell to download and play the audio file.

In [ ]python · cell 47

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/coffee_order.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In this config, you'll set custom_vocabulary inside AudioTranscriptionConfig. This is a list of phrases that bias Gemini 3.5 Transcribe toward recognizing specific terms, which can be helpful for product names, brand terms, or other vocabulary you want to appear accurately in the transcript. The model generally follows custom vocabulary instructions more reliably when a language is also specified.

In [ ]python · cell 49

python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(
        language_codes=["en-US"],
        custom_vocabulary=["oatmilk", "oz"],

    ),
)
interim_display = display(Markdown(""), display_id=True)
await streaming_main(audio_file, config, interim_display)

Transcribing long audio with session chunking

Streaming sessions have a practical length limit. For longer recordings, split the audio into overlapping chunks and open a new Live session for each one. Run the following cell to download a longer audio file.

In [ ]python · cell 51

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/Accessible_writing_tip_Informative_semantic_titles_and_headings.mp3"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

transcribe_long_audio splits the audio into chunk_length_ms-long segments (2.5 minutes) using pydub, re-including the last overlap_ms of the previous chunk so words aren't cut off at a chunk boundary. Each chunk is exported to a temporary WAV file and streamed in its own Live session with streaming_main, updating a separate display per chunk.

In [ ]python · cell 53

python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(),
)


async def transcribe_long_audio(audio_file, config):
    print(f"Loading '{audio_file}' into memory...")
    audio = AudioSegment.from_file(audio_file)

    chunk_length_ms = 150000
    overlap_ms = 500

    temp_file_path = "temp_chunk.wav"
    full_transcript = []

    try:
        for i, chunk_start in enumerate(range(0, len(audio), chunk_length_ms)):
            # Handle boundary overlap
            actual_start = max(0, chunk_start - overlap_ms) if i > 0 else chunk_start
            chunk_end = min(chunk_start + chunk_length_ms, len(audio))

            # Slices and exports the chunk to wav
            chunk = audio[actual_start:chunk_end]
            chunk.export(temp_file_path, format="wav")

            interim_display = display(
                Markdown(f"**Chunk {i+1}:** *Initializing...*"),
                display_id=True
            )

            await streaming_main(
                temp_file_path,
                config,
                interim_display,
            )

    finally:
        if os.path.exists(temp_file_path):
            os.remove(temp_file_path)
            print("\nCleaned up temporary chunk file.")
    return "\n\n".join(full_transcript)


await transcribe_long_audio(audio_file, config)

Speech recognition from microphone input

In the following cells, you'll simulate transcribing text from an audio stream. To start, you'll record an audio clip with your microphone by running the following cell. You might need to run it twice after granting microphone permission.

In [ ]python · cell 55

python

if "google.colab" in sys.modules:
    from google.colab import output

    output.enable_custom_widget_manager()

camera = CameraStream(constraints={"audio": True, "video": False})
recorder = AudioRecorder(stream=camera)
recorder

Once the audio is captured and you've stopped recording, you'll use FFmpeg to convert and save the clip to an MP3 file for processing. (FFmpeg comes preinstalled in Colab; install it locally if you're running this notebook elsewhere.)

In [ ]python · cell 57

python

with open("recording.webm", "wb") as f:
    f.write(recorder.audio.value)

audio_file = "recording.mp3"
!ffmpeg -i recording.webm -vn -ar 44100 -ac 2 -f mp3 recording.mp3

Now, you'll read the audio file and generate audio chunks to simulate streaming using the previously defined helper functions.

In [ ]python · cell 59

python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(),
)
interim_display = display(Markdown(""), display_id=True)
await streaming_main(audio_file, config, interim_display)

If you'd like to run an example with true live translation, view this example. Note, the lifetime of a connection is limited to around 10 minutes due to WebSocket connection constraints.

← 返回课程

Generative AI on Google Cloud

31 / 120·notebook

Chapter 31

Gemini 3.5 Transcribe

NotebookPython 360 cells

In [ ]python · cell 1

python

# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Gemini 3.5 Transcribe

Author
Katie Nguyen

Overview

This notebook introduces Gemini 3.5 Transcribe, Google's model for converting speech to text in multiple languages, available through Agent Platform.

In this tutorial, you'll learn how to use Gemini 3.5 Transcribe with the Google Gen AI SDK, through both the synchronous generateContent API and the streaming BidiGenerateContent (Live) API, to:

Synchronous:

Transcribe an audio file with automatic language recognition
Generate word-level timestamps for a transcription
Transcribe an audio file using language codes
Transcribe an audio file with speaker diarization
Bias transcription output with custom vocabulary
Transcribe long audio files

Streaming:

Stream transcription with automatic language recognition
Stream transcription using language codes
Bias streaming transcription with custom vocabulary
Stream transcription for long audio files
Stream transcription from a microphone

Get started

Install Google Gen AI SDK for Python & other libraries

Install the Google Gen AI SDK along with a few supporting libraries used later in this notebook:

ipywebrtc: captures microphone audio directly in the notebook
soundfile: reads audio files as raw PCM frames for streaming
numpy: downmixes multi-channel audio to mono before streaming
pydub: converts and splits (chunks) audio files

In [ ]python · cell 7

python

%pip install --upgrade --quiet google-genai ipywebrtc soundfile numpy pydub> /dev/null 2>&1

Authenticate your notebook environment (Colab only)

If you are running this notebook on Google Colab, run the following cell to authenticate your environment.

In [ ]python · cell 9

python

import sys

if "google.colab" in sys.modules:
    from google.colab import auth

    auth.authenticate_user()

Import libraries

In [ ]python · cell 11

python

import warnings

warnings.filterwarnings("ignore", category=SyntaxWarning)

In [ ]python · cell 12

python

import asyncio
import os
import numpy as np
import soundfile as sf

from IPython.display import Audio, display, Markdown
from google import genai
from google.genai import types
from pydub import AudioSegment
from ipywebrtc import AudioRecorder, CameraStream

Set Google Cloud project information

To get started using Agent Platform, you must have an existing Google Cloud project and enable the Agent Platform API.

Learn more about setting up a project and a development environment.

In [ ]python · cell 14

python

# fmt: off
PROJECT_ID = "[your-project-id]"  # @param {type: "string", placeholder: "[your-project-id]", isTemplate: true}
# fmt: on
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    PROJECT_ID = str(os.environ.get("GOOGLE_CLOUD_PROJECT"))

LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "global")

client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)

Load the Gemini 3.5 Transcribe model

Define the model IDs used throughout this notebook:

MODEL_ID: used with client.models.generate_content() to transcribe a complete, pre-recorded audio file in a single request. (Synchronous)
MODEL_ID_LIVE: used with client.aio.live.connect() to stream audio and receive transcription results incrementally, in real time. (Streaming)

In [ ]python · cell 16

python

MODEL_ID = "gemini-3.5-transcribe-preview"
MODEL_ID_LIVE = "gemini-3.5-transcribe-live-preview"

Transcribe audio files (Synchronous)

Transcription with automatic language recognition and timestamps

For this first request, run the following cell to download and play the audio you'll be transcribing. If you'd like to use a different audio clip, modify the audio_file_url variable below.

In [ ]python · cell 19

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/tell-a-story.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 20

python

with open(audio_file, "rb") as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type="audio/wav",
        ),
    ],
    config=types.GenerateContentConfig(
        audio_transcription_config=types.AudioTranscriptionConfig(
            word_timestamp=True,
        ),
    ),
)

parts = getattr(response, "parts", []) or []
if parts and (audio_tx := getattr(parts[0], "audio_transcription", None)):
    for w in getattr(audio_tx, "words", []) or []:
        display(Markdown(f"[{w.start_offset} - {w.end_offset}] {w.word}"))

if text := "".join(p.text for p in parts if getattr(p, "text", None)):
    display(Markdown(f"**{text}**"))

Transcription with language codes

Run the following cell to download and play a new audio clip.

In [ ]python · cell 22

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/spanish.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 24

python

with open(audio_file, "rb") as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_uri(
            file_uri="gs://cloud-samples-data/generative-ai/audio/spanish.wav",
            mime_type="audio/wav",
        ),
    ],
    config=types.GenerateContentConfig(
        audio_transcription_config=types.AudioTranscriptionConfig(
            language_codes=["es-ES"],
        ),
    ),
)

parts = getattr(response, "parts", []) or []
if text := "".join(p.text for p in parts if getattr(p, "text", None)):
    display(Markdown(text))

Transcription with speaker diarization

Run the following cell to download and play a new audio clip.

In [ ]python · cell 26

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/hobbies.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 28

python

with open(audio_file, "rb") as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type="audio/wav",
        ),
    ],
    config=types.GenerateContentConfig(
        audio_transcription_config=types.AudioTranscriptionConfig(
            diarization=True,
        ),
    ),
)

parts = getattr(response, "parts", []) or []
for p in parts:
    audio_tx = getattr(p, "audio_transcription", None)
    speaker = getattr(audio_tx, "speaker_label", "UNKNOWN") if audio_tx else "UNKNOWN"
    text = getattr(p, "text", "") or (getattr(audio_tx, "text", "") if audio_tx else "")

    if text:
        display(Markdown(f"**{speaker}**: {text}"))

Transcription with custom vocabulary

Run the following cell to download and play a new audio clip.

In [ ]python · cell 30

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/coffee_order.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

Note: The model generally follows custom vocabulary instructions more reliably when a language is also specified.

In [ ]python · cell 32

python

with open(audio_file, "rb") as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type="audio/wav",
        ),
    ],
    config=types.GenerateContentConfig(
        audio_transcription_config=types.AudioTranscriptionConfig(
            language_codes=["en-US"],
            custom_vocabulary=["oatmilk", "oz"],
        ),
    ),
)
parts = getattr(response, "parts", []) or []

if text := "".join(p.text for p in parts if getattr(p, "text", None)):
    display(Markdown(text))

Long audio files for batch transcription

The generate_content transcription requests currently support audio clips up to 1 hour.

In [ ]python · cell 34

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/pixel.mp3"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 35

python

with open(audio_file, "rb") as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type="audio/wav",
        ),
    ],
    config=types.GenerateContentConfig(
        audio_transcription_config=types.AudioTranscriptionConfig(),
    ),
)

parts = getattr(response, "parts", []) or []
if text := "".join(p.text for p in parts if getattr(p, "text", None)):
    display(Markdown(text))

Transcribe streaming audio (Streaming)

Define helper functions for streaming transcription

The following helper functions manage a streaming transcription session and are reused throughout the rest of this notebook:

send_streaming_audio(session, audio_file): reads the audio file in small blocks with soundfile, downmixes multi-channel audio to mono, and sends each block to the Live session as raw PCM audio (audio/pcm;rate=<sample_rate>) via session.send_realtime_input(). A short delay between blocks paces the upload to roughly match real-time playback. Once the file has been fully sent, it signals audio_stream_end=True.
receive_streaming_messages(session, interim_display, transcript_buffer): listens for messages from the session. Partial results arrive as interim_input_transcription and update the live display in place; once a segment is finalized, it arrives as input_transcription and is appended to transcript_buffer.
streaming_main(audio_file, config, interim_display): opens a Live session with client.aio.live.connect(model=MODEL_ID_LIVE, config=config), confirms the server's setup_complete message before streaming any audio, then runs the send and receive helpers concurrently, giving the receiver a few extra seconds to process any trailing messages after the audio finishes sending.

In [ ]python · cell 38

python

async def send_streaming_audio(session, audio_file):
    try:
        with sf.SoundFile(audio_file, mode='r') as f:
            sample_rate = f.samplerate

            chunk_duration = 0.1
            chunk_frames = int(sample_rate * chunk_duration)
            pacing_delay = chunk_duration
            mime_type = f"audio/pcm;rate={sample_rate}"

            for block in f.blocks(blocksize=chunk_frames, dtype='int16'):
                if f.channels > 1:
                    # Average the channels together and cast back to 16-bit integers
                    block = np.mean(block, axis=1).astype(np.int16)
                data = block.tobytes()
                await session.send_realtime_input(audio=types.Blob(data=data, mime_type=mime_type))
                await asyncio.sleep(pacing_delay)

    except Exception as e:
        print(f"Error reading/streaming wav file: {e}")
    finally:
        await session.send_realtime_input(audio_stream_end=True)


async def receive_streaming_messages(session, interim_display, transcript_buffer):
    current_interim = ""
    try:
        async for message in session.receive():
            if message.server_content:
                server_content = message.server_content

                # Track the active interim segment
                interim = server_content.interim_input_transcription
                if interim and interim.text:
                    current_interim = interim.text

                    # Update the temporary live-streaming view
                    active_view = " ".join(transcript_buffer + [current_interim])
                    interim_display.update(Markdown(f"{active_view}"))

                # Save final transcript and clear the interim
                final = server_content.input_transcription
                if final and final.text:
                    transcript_buffer.append(final.text)
                    current_interim = ""

                    active_view = " ".join(transcript_buffer)
                    interim_display.update(Markdown(f"{active_view}"))

    except Exception as e:
        print(f"Error transcribing file: {e}")


async def streaming_main(audio_file, config, interim_display):
    transcript_buffer = []

    async with client.aio.live.connect(model=MODEL_ID_LIVE, config=config) as session:
        if session.setup_complete is None:
            print("No setup_complete received from server.")
            return
        send_task = asyncio.create_task(send_streaming_audio(session, audio_file))
        recv_task = asyncio.create_task(receive_streaming_messages(session, interim_display, transcript_buffer))

        await send_task
        try:
            await asyncio.wait_for(recv_task, timeout=5.0)
        except asyncio.TimeoutError:
            recv_task.cancel()

Transcription with automatic language recognition

To transcribe streaming audio, build a LiveConnectConfig with the following fields:

response_modalities: Since you want transcribed text back, set this to ["TEXT"].
input_audio_transcription: An AudioTranscriptionConfig describing how to transcribe the incoming audio.

Run the following cell to download and play the audio you'll be streaming, then run the cell after it to start the streaming session.

In [ ]python · cell 40

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/tell-a-story.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 41

python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(),
)
interim_display = display(Markdown(""), display_id=True)
await streaming_main(audio_file, config, interim_display)

Transcription with a specific language

Once again, run the following cell to download and play the audio you'll be transcribing.

In [ ]python · cell 43

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/korean-english.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 45

python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(
        language_codes=["ko-KR", "en-US"],
    ),
)
interim_display = display(Markdown(""), display_id=True)
await streaming_main(audio_file, config, interim_display)

Transcription with custom vocabulary

Run the following cell to download and play the audio file.

In [ ]python · cell 47

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/coffee_order.wav"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 49

python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(
        language_codes=["en-US"],
        custom_vocabulary=["oatmilk", "oz"],

    ),
)
interim_display = display(Markdown(""), display_id=True)
await streaming_main(audio_file, config, interim_display)

Transcribing long audio with session chunking

In [ ]python · cell 51

python

audio_file_url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/Accessible_writing_tip_Informative_semantic_titles_and_headings.mp3"
audio_file = "input.wav"

!wget -q $audio_file_url -O "input.wav"
display(Audio(filename=audio_file, autoplay=False))

In [ ]python · cell 53

python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(),
)


async def transcribe_long_audio(audio_file, config):
    print(f"Loading '{audio_file}' into memory...")
    audio = AudioSegment.from_file(audio_file)

    chunk_length_ms = 150000
    overlap_ms = 500

    temp_file_path = "temp_chunk.wav"
    full_transcript = []

    try:
        for i, chunk_start in enumerate(range(0, len(audio), chunk_length_ms)):
            # Handle boundary overlap
            actual_start = max(0, chunk_start - overlap_ms) if i > 0 else chunk_start
            chunk_end = min(chunk_start + chunk_length_ms, len(audio))

            # Slices and exports the chunk to wav
            chunk = audio[actual_start:chunk_end]
            chunk.export(temp_file_path, format="wav")

            interim_display = display(
                Markdown(f"**Chunk {i+1}:** *Initializing...*"),
                display_id=True
            )

            await streaming_main(
                temp_file_path,
                config,
                interim_display,
            )

    finally:
        if os.path.exists(temp_file_path):
            os.remove(temp_file_path)
            print("\nCleaned up temporary chunk file.")
    return "\n\n".join(full_transcript)


await transcribe_long_audio(audio_file, config)

Speech recognition from microphone input

In [ ]python · cell 55

python

if "google.colab" in sys.modules:
    from google.colab import output

    output.enable_custom_widget_manager()

camera = CameraStream(constraints={"audio": True, "video": False})
recorder = AudioRecorder(stream=camera)
recorder

In [ ]python · cell 57

python

with open("recording.webm", "wb") as f:
    f.write(recorder.audio.value)

audio_file = "recording.mp3"
!ffmpeg -i recording.webm -vn -ar 44100 -ac 2 -f mp3 recording.mp3

Now, you'll read the audio file and generate audio chunks to simulate streaming using the previously defined helper functions.

In [ ]python · cell 59

python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(),
)
interim_display = display(Markdown(""), display_id=True)
await streaming_main(audio_file, config, interim_display)

If you'd like to run an example with true live translation, view this example. Note, the lifetime of a connection is limited to around 10 minutes due to WebSocket connection constraints.