giangndm/Qwen3-ASR-1.7B-encoder

Qwen3-ASR-1.7B Audio Encoder & Projector (bfloat16 Transformers SafeTensors)

This repository provides the standalone bfloat16 safetensors Audio Encoder & Multi-Modal Projector extracted from Qwen/Qwen3-ASR-1.7B-hf.

It can be loaded directly with transformers.AutoModel via trust_remote_code=True without requiring the full 1.7B language model backbone.

---

📐 Architecture & Specifications

Parameter	Value	Description
Audio Tower Architecture	24 Transformer Layers (16 heads, FFN 4096)	Whisper/Conformer-style Transformer Encoder
Audio Hidden Dimension	1024-dim	Raw output of the Audio Tower (hidden_states)
Projected Dimension	2048-dim	Output after Multi-Modal Projector matching Qwen3 LLM (last_hidden_state)
Frame Rate	12.5 frames/s (80ms / frame)	$8\times$ temporal downsampling from 10ms log-mel frames
Audio FrontEnd	128 Mel Bins, 16 kHz	3 Conv2D downsampling stages ($2\times 2\times 2 = 8\times$)
Weight Format & Precision	model.safetensors (bfloat16)	Lightweight standalone package: ~605 MB

---

💻 Quick Start with Hugging Face transformers

import torch
import torchaudio
from transformers import AutoModel

# 1. Load Audio Encoder from Hugging Face
model_id = "giangndm/Qwen3-ASR-1.7B-encoder"
model = AutoModel.from_pretrained(
    model_id, 
    trust_remote_code=True, 
    torch_dtype=torch.bfloat16
).to("cuda")
model.eval()

# 2. Load Any 16kHz Audio File
waveform, sr = torchaudio.load("path/to/audio.wav") # [1, T_samples]

# 3. Extract 128-dim Log Mel-Fbank Features (Using the Built-in Helper)
fbank = model.extract_fbank(waveform.to("cuda"), sample_rate=sr).to(torch.bfloat16) # [1, 1, T_mel, 128]

# 4. Forward Pass through Audio Tower + Projector
with torch.no_grad():
    output = model(fbank, return_projected=True)
    audio_embeds = output.last_hidden_state # [1, T_frames, 2048] (12.5 Hz / 80ms rate)

print("Audio Embeddings Shape:", audio_embeds.shape)
# Example: 1-second audio -> torch.Size([1, 12, 2048])

---

📦 Internal Layer Structure:

• conv2d1 $\to$ conv2d2 $\to$ conv2d3: 3 Conv2D blocks ($3\times 3$, stride 2) performing $8\times$ downsampling on mel bins and time.
• conv_out: Linear projection layer mapping $7680 \to 1024$.
• layers.0 ... layers.23: 24 Transformer self-attention layers with Pre-LayerNorm.
• ln_post: Final LayerNorm of the Audio Tower.
• projector_linear1 $\to$ GELU $\to$ projector_linear2: 2-layer MLP projector mapping $1024 \to 2048$ into the Text LLM embedding space.