dd3434/Streamalign-R32

StreamAlign R=32 RVQ Speech Tokenizer

Streaming speech tokenizer reproducing the StreamAlign paper recipe: a char-level RNN-T aligner (frozen encoder) with a word-aligned acoustic head quantized by a plain ResidualVQ (R=32 layers, codebook 512, dim 256), decoding to CosyVoice3 speech tokens for streaming reconstruction (chunk_size=4, left_context=32).

Checkpoint

final.pt is the released tokenizer, taken from the end of the cosine phase. Dict keys: {epoch, hubert_state_dict, optimizer_state_dict, train_loss}. Load hubert_state_dict into models/model_tokenizer.py::Data2VecSemanticAcousticModel (streamASR, branch refactor/tokenizer-r32) with env RVQ_R=32 RVQ_CODEBOOK_SIZE=512.

Training (3-phase, global batch 16)

phase	trainer	LR	epochs
A continuous (RVQ_BYPASS=1)	train_tokenizer.py	1e-4	15
B RVQ on (subalign-init from A)	train_tokenizer.py	1e-4	14
C cosine finetune (from B)	train_tokenizer_cosine.py	1e-5 -> 0	13

Data: LibriSpeech 960h + Emilia-EN 400h subset (precomputed CosyVoice3 features). Pipeline: scripts/train_tokenizer_r32_pipeline.sh.

Results (LibriSpeech test-clean 2620, streaming reconstruction)

metric	value	note
WER	4.43%	whisper-large-v3 (paper: 4.41%)
CER	1.92%
UTMOS	4.23	versa pseudo_mos
SECS	0.585	versa speaker, RawNet3

Dependencies at inference

Frozen char RNN-T encoder ckpt (stage 1), word/BPE streaming ASR + tokenizer, boundary classifier, CosyVoice3-0.5B vocoder. See train_tokenizer_r32_pipeline.sh eval for the exact wiring.

Citation

Accepted to Findings of EMNLP 2026.

@inproceedings{kim2026streamalign,
  title     = {{StreamAlign: Streaming Text-Aligned Speech Tokenization}},
  author    = {Kim, Kang-wook and Park, Jinyoung and Kim, Jinsoo and
               Lee, Sehun and Woo, Tony and Kim, Gunhee},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}