Towards Codec- LM Co-design for Neural Codec Language Models (2025)

Abstract: Neural codec language models (or codec LMs) are emerging as a powerful framework for audio generation tasks like text-to-speech (TTS). These models leverage advancements in language modeling and residual vector quantization (RVQ)-based audio codecs, which compress audios into discrete codes for LMs to process. Despite the close interdependence of codecs and LMs in these systems, research on codecs and LMs has largely remained siloed. In this work, we propose three techniques for better codec-LM co-design: (i) a frame-wise codec encoder that improves both LM log-likelihood and end-to-end TTS metrics, (ii) LM codebook level dropout, a method to efficiently navigate a portion of the codec-LM design space by training a single LM, and (iii) increased codec frame duration, which we show can accelerate inference while maintaining end-to-end performance. Our experiments demonstrate that combining all three co-design techniques results in doubled inference speed, and improvements in intelligibility, audio quality, and speaker control in TTS relative to a siloed baseline.

## introduction neural codec language models (or codec lms) @xcite have recently emerged as a prominent framework for text-to-speech (tts) @xcite @xcite and general audio generation tasks @xcite @xcite , replacing autoregressive methods that model continuous raw waveforms @xcite @xcite . the success of codec lms can be attributed to improvements in the architecture, scaling, and efficiency of language models (lms) @xcite @xcite , as well as increasingly high-fidelity convolutional audio codecs that employ the residual vector quantization (rvq) technique @xcite @xcite , bridging continuous-domain audio generation tasks with lm methods that model discrete tokens. although the codec and the lm are closely coupled, they represent relatively isolated research areas. research on codecs @xcite @xcite primarily focuses on achieving higher compression rates (i.e., lower bandwidths) while maintaining reconstruction quality. conversely, research on codec-based lms @xcite @xcite typically treats the codec as a fixed module and explores how to best model the codec tokens. (we defer more detailed related work to appendix a.) while the design space of codecs and lms combined is too large to explore exhaustively, considering each in isolation may be suboptimal when the goal is to improve the end-to-end performance. in this work, we aim to break the isolation and uncover co-design principles between the codec and the lm. we identify several aspects that play a key role in the interactions between the two, and substantially impact the end-to-end generation quality and/or efficiency. leveraging these co-design insights, we propose actionable interventions which can improve the performance and efficiency (both at training and inference) of end-to-end audio generation systems. our technical contributions are: a schematic diagram of our end-to-end audio generation system and proposed techniques can be found in fig. 1 . our experiments are based on a streamable (i.e., causal) variant of the dac codec @xcite , and we implement our changes (i.e., framewise encoder, and longer frame duration) without altering its architecture. we then train delay-pattern lms @xcite for tts, where lm codebook level dropout is applied, on the rvq codes from our codecs. we finally demonstrate that combining all three co-design techniques doubles the end-to-end tts inference speeds while improving all end-to-end tts metrics ( we open source our implementation of the framewise and causal dac @xcite codecs at https://github.com/slseanwu/ descript-audio-codec/tree/main . ## technical background residual vector quantization (rvq)-based audio codecs. an rvq-based audio codec compresses a continuous waveform @xmath0, where t is the duration (in seconds) and f s is the sampling rate (in hz) of the waveform, into discrete codes @xmath1. here, v := {1, 2, . . . , |v|} represents the codebook, f x (typically much smaller than f s ) is the frame rate (in hz) of the codec, and q is the number of codebook levels used to represent each frame. we also call downsampling rate of the codec, i.e., f s /f x , the frame size (an integer number of audio samples) and 1/f x the frame duration (in seconds). the term residual refers to how the q codebook levels are structured to progressively refine the quantization @xcite . let the unquantized representation (i.e., the codec encoder output) for the i-th frame be denoted by h (1) @xmath2, where d is the codec encoder's output dimension. the rvq process works iteratively for each level @xmath3, . . . , q} on a frame-by-frame basis, quantizing the residual information from preceding levels using a level-wise learned codebook c (q) : @xmath4. the operations at each level are: typically, during rvq codec training, quantizer dropout @xcite is applied, which sometimes performs eqn. (1) and (2) for q @xmath5. this enables the codec to encode and reconstruct audio waveforms at all q possible rvq level counts. language modeling with delay pattern of rvq codes. we can construct an end-to-end audio generative model by training an lm on the rvq codes @xmath6, where q ′ ∈ {1, . . . , q} is a subset of the rvq levels to model. to model such 2d-structured codes, we adopt the delay pattern proposed in @xcite , which makes a good tradeoff between the efficiency and efficacy of modeling the rvq codes x. instead of naively flattening x to a sequence of tf x × q ′ elements, it shifts the q-th level of x to the right by q positions, creating a shifted code sequence x (delay) ∈ v (tfx+q ′ - foot_0 )×q ′ , where each frame is x (delay) t := [x t-q+1, q ] q ′ @xmath7. then, the lm models: ## experiments and discussion we first conduct experiments specifically for each proposed technique (i.e., sec 3.1, 3.2, and 3.3) to elucidate their individual effects, and finally combine them to show their collective benefits. we use word error rate (wer), nisqa @xcite , and cosine similarity of speaker embeddings @xcite to evaluate the intelligibility, audio quality, and speaker control of endto-end tts generations. for music generation, we use fréchet audio distance (fad) @xcite to capture overall quality. more experimental setups are deferred to appendix c. 1 2 3 4 5 6 7 8 9 10 11 lms w/o cl drop at each q ′ ∈ {1, . . . , 12}. note that practitioners can then train a second lm at the found optimal level count w/o cl drop for best possible performance. shaded bands represent stdev over 3 runs. codec config codec recons. text-to-speech efficiency frame dur. log 2 (|v|) q ′ rel. bitrate mel-l1↓ wer↓ nisqa↑ spk. sim.↑ inf. speedup↑ 11ms 10 9 1.00× .873 3.71 ±.19 4.37 ±.02 80.7 ±.2 1.00× 11ms 15 6 1.00× .874 3.73 ±.28 4.33 ±.01 80.4 ±.1 1.01× 22ms 10 16 0.89× .888 4.21 ±.33 4.42 ±.01 81.0 ±.1 1.94× 22ms 15 11 0.92× .876 3.55 ±.36 4.33 ±.01 79.3 ±.1 2.00× 44ms 10 32 0.89× .875 6.73 4.14 76.7 3.20× 44ms 15 20 0.83× .871 4.53 3.65 73.2 3.77× the codec. 5 we experiment with various dropout distributions p(q) (see app. d for details) and conclude that it is best to train at the full level count (i.e., 12 in this case) for 90% of the steps and uniformly distribute the remaining 10% to all lower level counts. the curves in fig. 2 show that training a single lm with cl drop produces a performance profile closely aligned with training 12 separate lms without cl drop. this demonstrates that cl drop is a reliable method for practitioners to efficiently optimize for the level count q ′ with significantly reduced training compute. besides, the curves also show that wer, which focuses on (coarser) word-level information, reaches the best early at 3∼4 levels, while nisqa and speaker similarity, which are tied more closely to the finegrained details, peak at around 9 levels. though different metrics behave differently w.r.t. level count, we find that choosing the best level count based on fad (shown in fig. 3 , which uses the same codec as here and would suggest using 9 levels) achieves a balanced performance between all the tts metrics we consider. longer frame duration. combining all techniques. future work. our work may be extended to: