giangndm/nemotron-3.5-asr-streaming-decoder

Nemotron-3.5 ASR Streaming Decoder & Projector

This repository contains the standalone Language Projector + RNN-T Predictor + Joint Network extracted from NVIDIA's nvidia/nemotron-3.5-asr-streaming-0.6b.

It accepts raw continuous 1024-dimensional acoustic representations from the Pure Audio Encoder (giangndm/nemotron-3.5-asr-streaming-encoder), conditions them on the language prompt, and autoregressively decodes them into text tokens.

---

Architecture Specifications

Parameter	Value	Description
Model Type	Language Projector + RNN-T	2-layer MLP Projector + 2-layer LSTM Decoder + Joint Head
Total Parameters	28.9M	Projectors: 5.1M, LSTM Decoder: 15.4M, Joint: 8.4M
Input Acoustic Dim	1024	Accepts raw continuous speech features directly from Pure Encoder
Projected Dim	640	Fuses language ID one-hot (prompt_ids) and projects 1024 -> 640
Predictor LSTM	2 layers	Hidden dimension: 640, Embedding: 13,088 $ imes$ 640
Vocabulary Size	13,088	Multilingual BPE tokens + language prompts
Blank Token ID	13087	Emission indicating progression along acoustic time axis

---

Quickstart: End-to-End Speech Transcription

import soundfile as sf
import torch
from transformers import AutoFeatureExtractor, AutoModel, AutoTokenizer

enc_repo = "giangndm/nemotron-3.5-asr-streaming-encoder"
dec_repo = "giangndm/nemotron-3.5-asr-streaming-decoder"

# 1. Load Pure Encoder and Decoder
feat_extractor = AutoFeatureExtractor.from_pretrained(enc_repo)
encoder = AutoModel.from_pretrained(enc_repo, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda().eval()

decoder = AutoModel.from_pretrained(dec_repo, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained(dec_repo, trust_remote_code=True)

# 2. Process audio
audio, sr = sf.read("voice_sample.wav")
inputs = feat_extractor(audio, sampling_rate=16000, return_tensors="pt")
input_features = inputs.input_features.to("cuda", dtype=torch.bfloat16)

with torch.no_grad():
    # Step 1: Pure acoustic encoding -> (B, T, 1024)
    enc_out = encoder(input_features, num_lookahead_tokens=13)
    acoustic_1024d = enc_out.last_hidden_state

    # Step 2: Decoder projects 1024d -> 640d with language prompt & greedy decodes
    # prompt_ids: 33 for vi-VN, 0 for en-US, 101 for auto
    token_ids = decoder.greedy_decode(acoustic_1024d)[0]

text = tokenizer.decode(token_ids, skip_special_tokens=True)
print("Transcript:", text)

---

Citation & Acknowledgements

• Original model: NVIDIA Nemotron-3.5 ASR Streaming 0.6B
• License: CC-BY-4.0