R ep C odec: A Speech Representation Codec for Speech Tokenization (2024)

Abstract: With recent rapid growth of large language models (LLMs), discrete speech tokenization has played an important role for injecting speech into LLMs. However, this discretization gives rise to a loss of information, consequently impairing overall performance. To improve the performance of these discrete speech tokens, we present RepCodec, a novel speech representation codec for semantic speech tokenization. In contrast to audio codecs which reconstruct the raw audio, RepCodec learns a vector quantization codebook through reconstructing speech representations from speech encoders like HuBERT or data2vec. Together, the speech encoder, the codec encoder and the vector quantization codebook form a pipeline for converting speech waveforms into semantic tokens. The extensive experiments illustrate that RepCodec, by virtue of its enhanced information retention capacity, significantly outperforms the widely used k-means clustering approach in both speech understanding and generation. Furthermore, this superiority extends across various speech encoders and languages, affirming the robustness of RepCodec.We believe our method can facilitate large language modeling research on speech processing.

## introduction the significant achievements of large language models (llms) within the field of natural language processing have attracted considerable attention, as evidenced by notable works such as openai (2023); @xcite ; @xcite ; @xcite ; @xcite . bridging the realms of continuous speech and tokenbased language models necessitates a key technique known as speech tokenization, which discretizes an audio signal into a finite set of tokens. by converting speech into discrete tokens, language models can predict the future semantic content and generate realistic speech with long-term consistency @xcite . as a result, a growing body of research has begun to incorporate speech tokenization into the realm of llm. noteworthy examples include audiolm @xcite , au-diopalm @xcite , vall-e @xcite , polyvoice @xcite , and speechgpt @xcite . discrete speech tokens can be divided into two categories: semantic tokens and acoustic tokens. acoustic tokens are produced by audio codecs @xcite @xcite , which aim to reconstruct the original audio so that it can be perceptually identical to listeners. however, attempting to preserve all information of the audios leads to high bitrates of acoustic tokens. the process not only imposes significant computational demands on the llms, but sometimes makes training infeasible with such lengthy sequences. for example, converting a 30-second audio segment into acoustic tokens needs 18,000 tokens. therefore, current language modeling approaches often require substantial architectural adjustments to accommodate such long sequences @xcite . semantic tokens, on the other hand, aim at preserving only the semantic information of the audio, which allows much lower bitrates. if the task relies only on the content of the speech (e.g. speech recognition / translation), using semantic tokens should be a better choice. at present, k-means clustering on speech representations @xcite is the most prevalent technique of extracting semantic tokens. however, this method has two drawbacks. firstly, it suffers from a loss of semantic information compared to the original speech representations @xcite . secondly, not all sets of speech representations are suitable for clustering. @xcite reported that the choice of speech encoder significantly affects the downstream task performance of speech tokenization. thus, we are motivated to improve the method of extracting semantic tokens by addressing the above problems. in this paper, we propose repcodec, a general tokenization approach for representations that can be applied to speech to extract its semantic tokens. repcodec leverages an end-to-end neural codec to preserve more information of the speech representations. repcodec is composed of an encoder, a vector quantizer (vq) and a decoder, and it is trained to reconstruct the input speech representations as much as possible. the speech encoder, codec encoder and the vq codebook together constitute the speech tokenization pipeline, which can produce high-quality semantic tokens for downstream tasks with a low bitrate. we evaluate the repcodec tokens in downstream tasks of speech understanding and generation. specifically, we use a decoder-only asr modelling task to evaluate the speech understanding of repcodec tokens, and a unit-to-speech generation task to measure the quality of repcodec tokens for speech generation. our comprehensive experiments demonstrate that rep-codec significantly outperforms the dominant kmeans clustering approach. it is worth noting that supervised approach such as asr and phoneme recognition can also be considered as forms of speech tokenization that convert speech into word tokens or phoneme tokens. however, it is essential to highlight the large amount of parallel data required for supervised training only exists for highresource languages. repcodec, on the contrary, is an unsupervised speech tokenization method that can be applied to any languages. the contribution of our work can be summarized as follows: ## related work several lines of works are related to repcodec, including self-supervised speech representation learning, speech tokenization and vector quantization. self-supervised speech representation learning. this line of research has recently gained huge success in the area of speech processing. prevalent methods usually requires the model to predict the content of unseen regions @xcite @xcite or to contrast the target unseen frame with randomly sampled ones @xcite . specifically, hubert @xcite is a pioneering work to employ k-means for speech tokenization. the generated tokens serve as the training targets for the speech encoders. and they find that these tokens exhibit a strong correlation with the phonemes. later on, @xcite ; @xcite show that these semantic tokens can be used directly to perform downstream tasks like asr or voice conversion. in addition to the self-supervised approaches that only use unlabeled speech data, many works train the speech encoder in an end-to-end manner @xcite with labeled speech data. among them, whisper @xcite is currently the largest open-source model for speech recognition, which achieves the lowest overall wer across various domains. speech tokenization. discrete speech tokens can be divided into two categories: semantic tokens @xcite @xcite and acoustic tokens @xcite @xcite . semantic tokens maintain the linguistic information of the speech and have high correlation with phonemes @xcite . they are commonly generated by applying k-means clustering to pretrained speech encoders like hubert @xcite or data2vec @xcite . semantic tokens are widely used for downstream tasks. for example, @xcite ; @xcite ; @xcite use them to train a unit-vocoder to generate raw speech. audiolm @xcite inputs the semantic tokens of w2v-bert to represent semantic information of the audio. zhang et al. (2023a) jointly trains a language model with semantic tokens to inject speech recognition ability to gpt-like models. dong et al. ( 2023 ) also incorporates semantic tokens for speech to speech translation. however, the discretization step of kmeans discards plenty of information of the speech, resulting in degraded performance in downstream tasks @xcite . in contrast to semantic tokens, acoustic tokens aim to preserve all the information of the audio. soundstream @xcite and encodec @xcite use a neural audio codec with residual vector quantizers (rvq) to learn acoustic tokens that can be directly reconstructed into raw audios. as these acoustic tokens contain acoustic information of the audio, they can be used to perform more complicated tasks than semantic tokens. for example, vall-e @xcite , uses them for zero-shot text to speech (tts), and audiolm @xcite employs these tokens not only to produce realistic speech but also music. however, as acoustic tokens need to preserve a lot of information unrelated to semantics, their bitrates surpass the capacity of conventional language models.consequently, handling these tokens requires specialized techniques @xcite , making their practical utility challenging. vector quantization. learning vector quantization is important for efficient coding of information. vq-vae (van den oord et al., 2017) introduces vector quantization into vae (kingma and welling, 2014) to reconstruct images by learning discrete codebooks. they propose a straight-through gradient method to allow gradient back-propagation through a non-differentiable quantization operation so that optimization of the network is feasible. furthermore, recent advancements include softmax quantization (kankanahalli, 2018), exponential moving average (ema) @xcite , and gumble-softmax @xcite also work well for optimization of the vq module. in addition to multiple vq codebooks in @xcite , soundstream @xcite introduces a new rvq method that is able to compress the raw audio with different bitrates. ## method despite the wide applications of semantic tokens in speech modeling @xcite , the discretization of the representations results in severe information loss. consequently, downstream tasks, such as asr or speech translation, suffer from a significant downgrade in performance. in audiolm @xcite , the wer is dramatically increased from 2.5% to 6.0% by using the discrete tokens of k-means from w2v-bert xl @xcite . the unit-vocoder in mhubert @xcite also relatively increases the wer of the generated audio by about 70%. these results all demonstrate that the severe information loss actually prevents the discretization of speech from obtaining sota performance. the information loss motivates us to preserve more information during the discretization of the representations. to this end, we propose a novel method, repcodec, to perform more efficient compression on the representations so that the semantic tokens can preserve more information and achieve better performance in downstream tasks. ## conclusion interacting with llms through speech leads to an increased demand for speech tokenization. to this end, we propose repcodec, a novel speech representation codec to convert continuous speech waveforms into discretized tokens. in contrast to previous methods, repcodec employs a parametric network to preserve more semantic information of the speech representations. the extensive experiments demonstrate that semantic tokens extracted by repcodec outperform the prevalent k-means algorithm in downstream tasks of both speech understanding and generation. moreover, the experiments also demonstrate that repcodec is a universal algorithm that can be applied to any speech encoders and to multiple languages.