Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer (2024)

Abstract: While recent advancements in speech language models have achieved significant progress, they face remarkable challenges in modeling the long acoustic sequences of neural audio codecs. In this paper, we introduce G enerative P re-trained S peech T ransformer (GPST), a hierarchical transformer designed for efficient speech language modeling. GPST quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a hierarchical transformer architecture, allowing for a unified one-stage generation process and enhancing Hi-Res audio generation capabilities. By training on large corpora of speeches in an end-to-end unsupervised manner, GPST can generate syntactically consistent speech with diverse speaker identities. Given a brief 3-second prompt, GPST can produce natural and coherent personalized speech, demonstrating in-context learning abilities. Moreover, our approach can be easily extended to spoken cross-lingual speech generation by incorporating multi-lingual semantic tokens and universal acoustic tokens. Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity. See https://youngsheen.github.io/GPST/demo for demo samples.

## introduction speech quantization has emerged as a crucial technique for speech language models to generate controllable, high-quality speech waveforms @xcite @xcite @xcite . specifically, a speech waveform can be quantized into two distinct types of discrete representations: semantic tokens @xcite and acoustic tokens @xcite . semantic tokens are typically obtained by applying the k-means clustering algorithm to the continuous activation space of self-supervised speech models @xcite . notably, gslm @xcite finds that auto-regressive models trained on semantic tokens can capture high-level linguistic content, supporting language modeling and resynthesis @xcite . however, semantic tokens fail to retain acoustic details such as speaker identity, resulting in suboptimal reconstruction. in contrast, acoustic tokens generated by neural codec models @xcite effectively compress speech at low bitrates while capturing the nuances of speech waveforms. consequently, a speech language model can maintain long-term consistency with semantic tokens and produce high-quality synthesis with acoustic tokens. however, neural codec models require an excessive number of codes for high-quality speech synthesis. for example, encodec @xcite generates codec embeddings at 75 hz for audio waveforms at 24 khz. subsequently, these codec embeddings are modeled using residual vector quantization (rvq), wherein high-quality synthesis typically requires eight or more hierarchical quantizers with 1024 entries. therefore, a mere 10second waveform results in at least 75 × 8 × 10 = 6000 codes, which constitutes an excessively long sequence for language models due to the quadratic complexity with respect to the sequence length for calculating self-attention @xcite . consequently, addressing the trade-off between the perceptual quality and computational complexity remains a core challenge for speech language models. recently, some methods have been proposed to address the issue of lengthy acoustic sequences. acoustic tokens inherently possess a hierarchical structure because of residual vector quantiza-tion: tokens from the preceding quantizers restore acoustic properties such as speaker identity, while the subsequent quantizers capture finer acoustic details. each quantizer is trained to model the residuals from the previous quantizers. recent approaches @xcite @xcite treat the acoustic token generation process as a multi-stage framework to avoid learning excessively long sequences simultaneously. in this work, we present generative pre-trained speech transformer (gpst), a model that facilitates controllable, high-quality speech generation in single stage. our approach combines speech quantization with the architecture of a hierarchical transformer @xcite . gpst initially models the semantic sequence with a next token prediction task, followed by modeling the acoustic sequence with the task of predicting the next d stack codes. the semantic sequence serves as a prompt for the acoustic token as a condition. we design a specialized hierarchical architecture to model the underlying hierarchical structure of the acoustic sequence, which comprises of a large global transformer and a small local transformer. the global transformer learns the highlevel relationships between the semantic tokens and the stacked acoustic tokens, while the local transformer models the hierarchical details in the stacked acoustic codes. by incorporating semantic and acoustic tokens within one hierarchical transformer, gpst can significantly reduce computational costs and effortlessly learn the long-term interactions of semantic tokens and local dependencies among residual codes. furthermore, we propose a training technique called "local-drop" to further improve the training efficiency of hi-res speech generation, which is typically impractical in current speech language models because of a large number of residual quantizers. consequently, our model can generate high-quality and semantically coherent speeches in one stage efficiently. our main contributions are summarized as follows. ## generative pre-trained speech language model (gpst) in this section, we start with the formulation of speech language modeling, along with the modeling challenges in speech generation. next, we elaborate on our proposed model gpst in detail, followed by an efficient training technique for gpst to train hi-res speech model. additionally, we discuss various inference modes with in-context learning. ## conclusion we introduce gpst, a generative pre-trained speech language model that integrates semantic tokens and acoustic tokens within a hierarchical transformer architecture, allowing for a unified onestage generation process. gpst demonstrates its capability to generate coherent speech and speaker identity transfer with in-context learning. furthermore, we show that gpst can generate hi-res speech and spoken multi-lingual speech as well.