Inconsistent Audio Support in Gemma-4-31B-IT: Video Audio Works, Standalone Audio Fails
I observed an inconsistency in the multimodal behavior of google/gemma-4-31B-it when served through the OpenAI-compatible API.
The model successfully understands and summarizes spoken content when the audio is embedded inside an MP4 video. However, when I provide the exact same audio as a standalone MP3 file, the request fails with an error indicating that the model does not support audio input.
Since both inputs contain identical speech, the only difference is the container format (MP4 vs. MP3), making the behavior appear inconsistent.
Environment
Model: google/gemma-4-31B-it
Serving Framework: vLLM 0.24.0
API: OpenAI-compatible Chat Completions API
Case 1: Video Input (Works)
Request
curl http://10.10.5.54:8701/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "google/gemma-4-31B-it",
"max_tokens": 2048,
"temperature": 0,
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Summarize the audio"
},
{
"type": "video_url",
"video_url": {
"url": "file:///home/wesee/Downloads/output.mp4"
}
}
]
}
]
}'
Result
The model successfully summarizes the spoken content from the video's audio track.
Example response:
"The provided audio is a comprehensive guide on how to use the 'Search' feature in a specific software application..."
This demonstrates that the model is capable of understanding and reasoning over the audio contained within the video.
Case 2: Standalone Audio (Fails)
The MP3 file below is extracted directly from the same MP4 file used in Case 1.
Request
curl http://10.10.5.54:8701/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "google/gemma-4-31B-it",
"max_tokens": 2048,
"temperature": 0,
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Summarize the audio"
},
{
"type": "audio_url",
"audio_url": {
"url": "file:///home/wesee/Downloads/output.mp3"
}
}
]
}
]
}'
Result
The request fails with the following error:
Audio input was provided but the model '/home/wesee/.cache/huggingface/hub/models--google--gemma-4-31B-it/...' does not have an audio tower. Audio inference is only supported for Gemma4 models that include an audio_config.
HTTP Status: 400 Bad Request
Expected Behavior
If the model is capable of understanding speech contained within a video, it should also be capable of processing the same speech when provided as a standalone audio file (e.g., MP3 or WAV). Alternatively, if standalone audio is intentionally unsupported, the documentation should clearly explain why audio embedded in video is accepted while direct audio inputs are rejected.
Actual Behavior
Input Result
MP4 containing speech ✅ Successfully processes and summarizes the spoken audio
MP3 extracted from the same MP4 ❌ Rejected with "model does not have an audio tower"
Why This Appears Inconsistent
The MP4 and MP3 contain identical audio content. The only difference is the container format.
The successful processing of the MP4 demonstrates that the system is capable of extracting and reasoning over the audio track. However, providing the identical audio directly as an MP3 is rejected before inference with an error indicating that audio input is unsupported.
From a developer's perspective, this is confusing because:
The model successfully answers questions about spoken content in videos.
The same spoken content cannot be processed when supplied as a standalone audio file.
Users are forced to convert audio into a video container as a workaround, even though the underlying audio content is unchanged.
Questions
Could you clarify whether this behavior is:
An intended limitation of Gemma-4-31B-IT?
A limitation of the current vLLM implementation?
A limitation of the OpenAI-compatible API?
Or an unintended inconsistency/bug?