thelarryproject larry-speech-encoder v1

The first model trained under thelarryproject in phase 1, larry-speech-encoder is primarily a speech unit ID encoder with a CTC proxy task head for training speech feature alignment.

As you can see from the above diagram, this model was trained with only PD, CC0, or CC-BY (no ND, NC, or SA) data. This data was additionally never processed, filtered, or evaluated using a model that was trained on anything other than PD, CC0, or CC-BY data. Essentially, this model is verifiably both by legal requirements and intent requirements to not be a derivative of ANY NC, ND, SA, or similarly licensed content.

Note: explicit consent for AI training specifically was not obtained from the original license holders, and PD/CC0/CC-BY is assumed consent.

This model features in order:

• a log mel frontend
• conv subsampling frontend to downsample input 100hz mel features to 25hz (4x downsampling)
• an encoder1 stack of 6 non-causally attending SAN-M style transformer encoder layers
  • contains another subsampling merge down to 12.5hz after encoder layer 3
  • SAN-M: uses FSMN attention layers instead of traditional MHA; uses a convolutional memory path in addition to MHA
    • FSMN uses kernel size of 31 prior to the merge and 15 after the merge
• an FSQ layer for quantizing into a fixed set of codes
  • codebook size for this model is 15360
• an encoder2 stack of 6 more non-causally attending traditional (MHA) transformer encoder layers
• a CTC head over the vocabulary of the packaged SentencePiece tokenizer (plus a blank token) trained specifically on the same weighted distribution of data that the model trains on

For a total of 163,618,694 parameters.

The codes output by the FSQ layer are the real reason for this model to exist, and the CTC head is not intended to be used for real inference for CTC alignment. Post-quantization CTC alignment is not the intended task for this model, with it only achieving ~5.52% WER (using greedy decoding and the whisper standard for transcript normalization on the model output and ground-truth labels) on the validation set. CTC is only a proxy task, and using this model for plain ASR or CTC alignment is not supported explicitly, as the quantization layer likely hampers performance measurably. As yet, no study has been done on this specific model for those effects.

The unit ID vocabulary is meant to encode speech features required for speech reconstruction by a downstream decoder that takes unit IDs, speaker representation, F0/harmonics conditioning, and optionally prepended sequence conditioning to produce plausible speaker-cloned speech.

This model's code is an approximation of the code used in CosyVoice2 to train a similar speech tokenizer, but is not directly copied from them. Their code is licensed as apache-2.0 at https://github.com/QwenAudio/CosyVoice.

Construction

This model used the following training and validation splits, which is configured in more detail in the model's own config yamls in the repo for thelarryproject at https://github.com/Mekadrom/larry:

Dataset	Training Hours	Validation Hours
libriheavy-large	37,000	166.8
Common Voice 27.0	1,800	25.4
thelarryproject/scotus	1,100	1.8
VCTK	39.3	2
LJSpeech	23.7	N/a
HiFi-TTS v1	289.8	0.6
librispeech-asr	N/a	10.1
Total	40,252.8	206.7

Where CV27, scotus, vctk, and librispeech are all speaker-disjoint from their train splits (where a train split exists). The Libriheavy and HiFi-TTS validation splits share speakers with their train splits.

Train Statistics

• Tokens seen: 3,658,563,934 (~3.7B) CTC SentencePiece vocab tokens

Eval Statistics

• wer_normalized (whisper-style transcript normalization on ground-truth labels and model greedy CTC decoding outputs)
  • Weighted average of all datasets: ~5.52%
  • wer_normalized/cv27: ~13.4%
  • wer_normalized/hifi-tts-all: ~0.510%
  • wer_normalized/libriheavy-test-clean-large: ~3.39%
  • wer_normalized/libriheavy-test-other-large: ~6.56%
  • wer_normalized/librispeech-asr: ~5.00%
  • wer_normalized/scotus: ~4.33%
  • wer_normalized/vctk: ~1.27%
  • measures CTC proxy task performance only
• Codebook (FSQ) perplexity: 5827
  • intended to represent how rich the vocabulary/codebook space is; a measure of how diffuse the codebook space is by approximate frequency of occurrence. For example, if every code was used the exact same amount, the perplexity would be 15360. If only one single code was ever used, the perplexity would be 1. This is different from LLM perplexity, though they are both proxy metrics for model health and represent exp(entropy) of the output space.
• Codebook usage: 0.999
  • 99.9% of FSQ codes are seen in a full pass of the validation set; 99.9% of 15360 is 15345, leaving ~15 codes that happened to not appear in the FSQ layer's output at eval.

Citation

If you use this model, please cite the model that it is based on as well:

@article{du2024cosyvoice,
  title={Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens},
  author={Du, Zhihao and Chen, Qian and Zhang, Shiliang and Hu, Kai and Lu, Heng and Yang, Yexin and Hu, Hangrui and Zheng, Siqi and Gu, Yue and Ma, Ziyang and others},
  journal={arXiv preprint arXiv:2407.05407},
  year={2024}
}

@article{du2024cosyvoice2,
  title={Cosyvoice 2: Scalable streaming speech synthesis with large language models},
  author={Du, Zhihao and Wang, Yuxuan and Chen, Qian and Shi, Xian and Lv, Xiang and Zhao, Tianyu and Gao, Zhifu and Yang, Yexin and Gao, Changfeng and Wang, Hui and others},
  journal={arXiv preprint arXiv:2412.10117},
  year={2024}
}

@article{du2025cosyvoice,
  title={CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training},
  author={Du, Zhihao and Gao, Changfeng and Wang, Yuxuan and Yu, Fan and Zhao, Tianyu and Wang, Hao and Lv, Xiang and Wang, Hui and Shi, Xian and An, Keyu and others},
  journal={arXiv preprint arXiv:2505.17589},
  year={2025}
}

@inproceedings{lyu2025build,
  title={Build LLM-Based Zero-Shot Streaming TTS System with Cosyvoice},
  author={Lyu, Xiang and Wang, Yuxuan and Zhao, Tianyu and Wang, Hao and Liu, Huadai and Du, Zhihao},
  booktitle={ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={1--2},
  year={2025},
  organization={IEEE}
}

As well as thelarryproject:

@misc{larry-speech-encoder-2026,
  author       = {{thelarryproject}},
  title        = {larry-speech-encoder: a 12.5 Hz FSQ speech encoder},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/thelarryproject/larry-speech-encoder}},
  note         = {Phase 1 release}
}

Training pipeline code is at https://github.com/Mekadrom/larry, and this model is fully reproducible (RNG state permitting) with a single command in that repo given the following presumptions:

• A preprocessed normalized Common Voice 27.0 distribution stored locally
  • See thelarryproject/common-voice-27-index-asr for the command to create this and optionally the index locally
    • Configure the training script to point at these datasets
• The provided larry-ctc-sp512.model as tokenizer_model_path
• a speaker mapping from uv run speaker-mapping as speaker_mapping_path

uv run training -f configs/training/voice/ctc/larry-speech-encoder.yaml -f configs/training/voice/mixtures/asr.yaml

This works best on commits later than the one on provenance.json, as the default config values as of that commit have been replaced with the values that produced the model distributed here. The commit that changed the default values is 3b12f6b.

Usage

Input is mono 16khz audio waveforms, and the output is one unit ID in the range 0-15,359 per 80 ms of input audio.

Install the larry package locally: pip install git+https://github.com/Mekadrom/larry (requires Python 3.14+)

from larry.voice.model.speech_encoder import SpeechEncoderModel

model = SpeechEncoderModel.from_pretrained("thelarryproject/larry-speech-encoder")

# assuming waveforms is a tensor of [batch, samples] and waveform_lengths is a tensor of [batch]
unit_ids, unit_lengths = model.tokenize(waveforms, waveform_lengths)

Other

• Speaker information in the codes: linear and non-linear probes trained to extract speaker identity on the static set of speakers in the training corpus only achieve 14.6% and 12.7% validation accuracy respectively when receiving the codes as input.
  • When receiving the raw features prior to the quantization, the same linear and non-linear probes achieve 93.3% and 89.1% speaker identification accuracy respectively, and this drop is taken to mean that quantization removes most speaker information.
  • Trained on 233 speakers, chance is 0.43%, so some speaker leakage still occurs.
• Per-dataset codebook usage is more nuanced than 99.9% of codes being seen at eval:
  • ~98% for CV27
  • ~73% for VCTK
  • ~50% for hifi-tts
  • this seems to mostly be an artifact of the size of these validation datasets