DM -Codec: Distilling Multimodal Representations for Speech Tokenization (2025)

Abstract: Recent advancements in speech-language models have yielded significant improvements in speech tokenization and synthesis. However, effectively mapping the complex, multidimensional attributes of speech into discrete tokens remains challenging. This process demands acoustic, semantic, and contextual information for precise speech representations. Existing speech representations generally fall into two categories: acoustic tokens from audio codecs and semantic tokens from speech self-supervised learning models. Although recent efforts have unified acoustic and semantic tokens for improved performance, they overlook the crucial role of contextual representation in comprehensive speech modeling. Our empirical investigations reveal that the absence of contextual representations results in elevated Word Error Rate (WER) and Word Information Lost (WIL) scores in speech transcriptions. To address these limitations, we propose two novel distillation approaches: (1) a language model (LM)-guided distillation method that incorporates contextual information, and (2) a combined LM and self-supervised speech model (SM)-guided distillation technique that effectively distills multimodal representations (acoustic, semantic, and contextual) into a comprehensive speech tokenizer, termed DM-Codec. The DM-Codec architecture adopts a streamlined encoder-decoder framework with a Residual Vector Quantizer (RVQ) and incorporates the LM and SM during the training process. Experiments show DM-Codec significantly outperforms state-of-the-art speech tokenization models, reducing WER by up to 13.46%, WIL by 9.82%, and improving speech quality by 5.84% and intelligibility by 1.85% on the LibriSpeech benchmark dataset.

## introduction in recent years, the advent of large language models (llms) has revolutionized various domains, offering unprecedented advancements across a wide array of tasks @xcite . a critical component of this success has been the tokenization of input data, enabling vast amounts of information processing @xcite . inspired by these breakthroughs, significant attention has shifted towards replicating similar successes in the realm of speech understanding and generation @xcite . however, tokenizing speech into discrete units presents unique challenges compared to text, as speech is inherently continuous and multidimensional, requiring various speech attributes such as acoustic properties, semantic meaning, and contextual clues @xcite . traditional approaches using feature representations such as mel-spectrograms @xcite , mel-frequency cepstral coefficients (mfccs) @xcite , and waveforms @xcite have proven inadequate in capturing this full spectrum of information, resulting in suboptimal performance in downstream tasks such as speech synthesis @xcite . these limitations led researchers to explore various approaches, and one prominent direction leading to audio codecs @xcite . notable examples include soundstream @xcite and encodec @xcite , which utilize residual vector quantizers (rvq) within a neural codec framework, iteratively refining quantized vectors to discretize speech into acoustic tokens. concurrently, self-supervised speech representation learning models such as hubert @xcite and wav2vec 2.0 @xcite facilitated extracting speech representations as semantic tokens @xcite . efforts to unify acoustic and semantic representations have led to two notable approaches: speechtokenizer figure 1 : overview of speech tokenization using discrete acoustic, semantic, and contextual tokens. dm-codec integrates these features for robust and comprehensive speech representation. @xcite , which utilizes semantic distillation from hubert, and facodec @xcite , which proposes a factorized vector quantizer to disentangle speech representation into different subspaces using separate rvqs with supervision. however, these approaches often overlook a key aspect: the integration of contextual information. language models (lms) have demonstrated an ability to learn contextual representations that capture the meaning of tokens from broader context @xcite . these contextual representations can provide essential insights into speech representation, allowing for a more nuanced understanding of words in varying linguistic contexts. our empirical investigations also reveal that existing discrete speech representation models struggle to align reconstructed speech with accurate textual form, resulting in elevated word error rates (wer) and word information lost (wil) scores in speech transcription. this observation underscores the need for a more comprehensive approach to speech tokenization that incorporates contextual information. to address these challenges, we propose dm-codec, a novel speech tokenizer that unifies multimodal language and speech representations. central to our innovation is the introduction of an lmguided distillation method that incorporates contextual representations into the speech tokenization process. this allows dm-codec capturing the nuances of linguistic context often missed by existing models. building upon the lm-guided approach, we propose a distillation method combining both lm and speech model (sm)-guided techniques. moreover, we introduce a [cls]-token-based distillation strategy that leverages sequence-level holistic representations from the lm, effectively cap-turing global contextual information. our distillation method only utilizes the lm and sm during training, without increasing model complexity and parameters, and are not required during inference. to the best of our knowledge, we are the first to attempt to integrate all three essential aspects of speech representation-acoustic, semantic, and contextual-within a single codec. see figure 1 for a depiction. in addition, to demonstrate the impact of multimodal representation and generalizability of dm-codec in downstream tasks, we introduce dm-codec-tts, a novel multimodal representation distilled neural codec language model. through extensive experiments on librispeech @xcite , we show dm-codec's superiority. dm-codec achieves a wer of 4.05 and wil of 6.61, outperforming speechtokenizer (4.49, 7.10), facodec (4.68, 7.33), and encodec (4.53, 7.17). it also improves speech quality, with a visqol score of 3.26 and mos of 3.72, surpassing encodec (3.08, 3.09), speechtokenizer (3.09, 3.67), and facodec (3.13, 3.70). dm-codec-tts excels on librispeech and vctk, outperforming uslm and vall-e. on librispeech, it achieves a wer of 5.08, wil of 7.32, mos of 3.70, and smos of 3.89, while on vctk, it achieves a wer of 3.58, wil of 5.65, mos of 3.78, and smos of 3.85. notably, dm-codec-tts-small achieves a wer of 10.26, wil of 13.79, mos of 3.24, and smos of 3.20, outperforming uslm (libri) across all metrics despite using a smaller dataset. we summarize our contributions below: ## proposed method this section introduces dm-codec, a novel speech tokenizer that distills multimodal (acoustic, semantic, and contextual) representations. as shown in figure 2 , we propose two training approaches: (i) lm-guided distillation, integrating contextual and acoustic representations, and (ii) combined lm and sm-guided distillation, incorporating semantic, contextual, and acoustic representations. additionally, a [cls] token-guided distillation leverages sequence-level holistic contextual representations. we also present dm-codec-tts, integrating dm-codec into a neural codec language model. the following subsections detail the distillation methods ( §3.1), dm-codec model and training objectives ( §3.2, §3.3), and dm-codec-tts ( §3.4). ## experimental setup dataset. we trained dm-codec using the lib-rispeech 100-hour clean speech training set @xcite , widely used for speech tokenizer and modeling @xcite @xcite . data were standardized by cropping samples to three seconds and ensuring a 16 hz sample rate. for dm-codec-tts, we used libri-heavy @xcite , a 50k-hour read english speech dataset derived from librivox, selecting samples between 0.5 and 15 seconds at 16 khz. additionally, a smaller dm-codec-tts version was trained on libritts @xcite , a 585-hour dataset with 2,456 speakers sampled at 24 khz. for evaluation, we tested dm-codec on 300 randomly selected audio samples from the lib-rispeech test-clean subset, aligning with baseline speechtokenizer's @xcite setup. a random seed of 42 ensured consistent sampling, and baseline models were evaluated on the same setup. additionally, we tested dm-codec-tts on librispeech test-clean and vctk @xcite . for librispeech, we followed vall-e's @xcite setup by constructing a 2.2hour subset from the test-clean data, selecting sam-ples of 4-10 seconds. for each synthesis, a different utterance from the same speaker was randomly selected as input text, and a 3-second segment was cropped as the enrollment speech prompts. each experiment was repeated three times, and the average score was reported. for vctk, we followed speechtokenizer's @xcite ) setup, using a 3s utterance as the prompts, while the input text was derived from a different utterance. training. we trained dm-codec utilizing two a100 gpus for 100 epochs with batch size of 6. we applied a learning rate of 1 × 10 -4 using the adam optimizer with a 0.98 learning rate decay. the embedding size was set to 1024 for rvq and 768 for the lm and sm. for all experiments, we used a random seed of 42 to ensure reproducibility. for the overall generator loss, we select the weight coefficients proportionally as follows: λ l/@xmath0, λ @xmath1.375x, λ @xmath2.15x, λ @xmath3.085x, and set λ @xmath4. for dm-codec-tts, we trained the autoregressive (ar) and nonautoregressive (nar) models separately, each for 4 epochs, while dm-codec-tts-small was trained with 100 epochs for the ar model and 196 epochs for the nar model. the batch size was determined dynamically based on the maximum number of audio seconds, set to 280 for ar and 200 for nar. we employed a base learning rate of 0.05 using the scaledadam optimizer with warmup steps of 200. baselines. we compared dm-codec with baselines: encodec (24 khz) @xcite , speechtokenizer @xcite , facodec (naturalspeech3) @xcite , dac (16 khz) @xcite , bigcodec @xcite , and moshi @xcite . we reproduced speechtokenizer using its official training code and used official model checkpoints for the rest. additionally, we compared dm-codec-tts with neural codec language models: uslm (from speechtokenizer @xcite ) and vall-e @xcite . since vall-e and uslm's training codes and models are not open-source, we relied on reported results. we also used uslm (libri), an official model checkpoint trained on lib-ritts and shared on github. all baseline model artifacts are open source and were used according to their licenses and intended research use. evaluation metrics. to measure context preservation, we used word error rate (wer) and word information lost (wil), with whisper (whispermedium) @xcite generating the transcriptions. wer quantifies transcription errors, ## experimental results and discussion in this section, we present experimental results evaluating speech reconstruction with dm-codec variants: dm-codec (lm) with lm-guided distillation, dm-codec (lm+sm) with combined lm and sm distillation, and dm-codec (cls) with [cls] token guided distillation against baseline speech tokenizers ( §5.1). we then assess zero-shot speech synthesis with dm-codec-tts against neural codec language models ( §5.2). furthermore, we conducted rigorous significance analysis (appendix c) and comprehensive ablation studies of dm-codec design choices (appendix d). ## conclusion we introduced dm-codec, a speech tokenizer with novel distillation methods that leverage multimodal (acoustic, semantic, and contextual) representations from language and speech self-supervised models. experimental results demonstrate that distilling multimodal representations enables dm-codec to encode salient speech information in discrete tokens. this approach showcases the potential of multimodal representations to improve speech tokenization across domains, including multilingual and code-switched speech processing.