giangndm/nemotron-3.5-asr-streaming-encoder

Nemotron-3.5 ASR Streaming Pure Audio Encoder

This repository contains the standalone Pure Audio Encoder extracted from NVIDIA's nvidia/nemotron-3.5-asr-streaming-0.6b.

Pure Speech Backbone: This encoder contains ONLY the FastConformer acoustic backbone (609.1M parameters). It has NO task-specific projection layers, language prompts, or decoders. All downstream language fusion and ASR projection heads reside in the companion decoder (giangndm/nemotron-3.5-asr-streaming-decoder).

It provides pure, high-fidelity 1024-dimensional continuous speech representations at 12.5 Hz (80ms per frame), optimized as a general-purpose acoustic backbone for multiple downstream speech tasks.

---

Why Use Nemotron-3.5 Pure Encoder as your Speech Backbone?

1. Optimal 12.5 Hz Frame Rate (80ms per frame):
   • Traditional Whisper encoders operate at 50 Hz (20ms/frame), producing 1,500 frames for 30s of audio.
   • Nemotron's FastConformer $8\times$ depthwise-separable subsampling produces 12.5 Hz (only 375 frames for 30s audio).
   • Saves up to 75% sequence length when projected into Large Language Models (LLMs), drastically cutting LLM KV cache memory and speeding up prefill/inference.
2. Versatile Streaming Lookahead:
   • Supports lookahead tokens: 0 (pure causal, 0ms latency), 3 (240ms), 6 (480ms), and 13 (1120ms max quality).
3. General-Purpose Speech Representations:
   • Continuous 1024-dim features directly suitable for:
     • Audio-LLM Acoustic Backbones (e.g., Qwen, Gemma, LLaMA linear/Q-Former projectors)
     • Discrete Acoustic Tokenization (K-Means clustering into 1,024 – 16,384 discrete acoustic codes)
     • Voice Activity Detection (VAD) & Diarization
     • Speech-to-Unit / Speech-to-Phoneme transcription heads

---

Architecture Specifications

Parameter	Value	Description
Model Type	FastConformer	Depthwise separable convolution + Multi-Head Self-Attention
Parameters	609.1M	Pure encoder backbone without task-specific heads
Input Audio	16 kHz Mono	Mel-Spectrogram (128 mel bins, 25ms window, 10ms hop)
Subsampling Rate	$8\times$	Subsamples 10ms mel frames into 80ms acoustic frames (12.5 Hz)
Output Dimension	1024	Raw continuous speech representations (last_hidden_state)
Conformer Layers	24	Hidden size: 1024, Intermediate size: 4096, 8 Attention Heads
Streaming Lookahead	[0, 3, 6, 13]	Configurable from 0ms latency up to 1120ms lookahead

---

Installation

pip install torch torchaudio soundfile "transformers>=4.48.0" huggingface_hub safetensors

---

Quickstart: Feature Extraction for Downstream Tasks

import soundfile as sf
import torch
from transformers import AutoFeatureExtractor, AutoModel

repo_id = "giangndm/nemotron-3.5-asr-streaming-encoder"

# 1. Load feature extractor and pure encoder
feat_extractor = AutoFeatureExtractor.from_pretrained(repo_id)
encoder = AutoModel.from_pretrained(
    repo_id, trust_remote_code=True, torch_dtype=torch.bfloat16
).cuda().eval()

# 2. Preprocess 16 kHz audio
audio, sr = sf.read("voice_sample.wav")
inputs = feat_extractor(audio, sampling_rate=16000, return_tensors="pt")
input_features = inputs.input_features.to("cuda", dtype=torch.bfloat16)

# 3. Extract continuous 1024-dim representations at 12.5 Hz (80ms/frame)
with torch.no_grad():
    # num_lookahead_tokens: 0 (pure causal 0ms), 3 (240ms), 6 (480ms), 13 (1120ms)
    outputs = encoder(input_features, num_lookahead_tokens=13)
    acoustic_features = outputs.last_hidden_state

# Shape: (batch_size, time_subsampled, 1024)
print("Continuous Acoustic Representations shape:", acoustic_features.shape)

---

Downstream Application Patterns

1. Audio-LLM Linear Projector (e.g. Qwen / Gemma)

import torch.nn as nn

class AudioLLMProjector(nn.Module):
    def __init__(self, encoder_dim=1024, llm_dim=4096):
        super().__init__()
        self.linear = nn.Linear(encoder_dim, llm_dim)

    def forward(self, acoustic_features):
        # Maps (B, T, 1024) -> (B, T, 4096) to concatenate into LLM prompt
        return self.linear(acoustic_features)

2. Discrete Acoustic Tokenization (K-Means)

# Flatten time and batch dimensions for vector quantization
flat_feats = acoustic_features.squeeze(0).float().cpu().numpy()
# cluster_ids = kmeans.predict(flat_feats)

---

Citation & Acknowledgements

• Original model: NVIDIA Nemotron-3.5 ASR Streaming 0.6B
• License: CC-BY-4.0