dd3434/Streamalign-R16

Streamalign (R16)

Complete R16 speech tokenizer stack. Paired SLM: Streamalign-SLM-R16 (reported metrics: SALMon 69.1, StoryCloze 72.1).

Contents & roles

Path	Role
rvq_teacher/	The acoustic tokenizer: the streaming encoder plus the R16 residual-VQ quantizer. Produces the R16 speech units. R16-specific.
alignment_model/	Char-level streaming Conformer-Transducer. Base scaffold the tokenizer is built on; supplies the RNN-T predictor/joiner for char-level alignment. R-independent.
streaming_asr/	Word-level streaming Conformer-Transducer. Generates the chunk TextGrids (word alignment) and drives the boundary classifier. R-independent.
boundary_classifier/	Word-boundary detector for streaming chunking; consumes streaming_asr. R-independent.
alignment.yaml	Extractor / alignment_model hparams.

Pipeline

audio -> streaming_asr (word chunks + TextGrids) + boundary_classifier -> alignment_model scaffold + rvq_teacher encoder -> R16 RVQ units.

Only rvq_teacher is R16-specific; the ASR / alignment / boundary components are shared across R8/R16/R32.

Tokenizer checkpoint

The checkpoint under rvq_teacher/ is the final R16 tokenizer, trained on LibriSpeech plus the full Emilia set.

Load it with RVQ_R=16 and RVQ_CODEBOOK_SIZE=512, and leave RVQ_CODEBOOK_DIM unset: codebook_dim defaults to feat_dim=256, which is what these weights expect.

Citation

Accepted to Findings of EMNLP 2026.

@inproceedings{kim2026streamalign,
  title     = {{StreamAlign: Streaming Text-Aligned Speech Tokenization}},
  author    = {Kim, Kang-wook and Park, Jinyoung and Kim, Jinsoo and
               Lee, Sehun and Woo, Tony and Kim, Gunhee},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}