Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition (2024)

Abstract: Advances in machine learning have made it possible to perform various text and speech processing tasks, such as automatic speech recognition (ASR), in an end-to-end (E2E) manner. E2E approaches utilizing pre-trained models are gaining attention for conserving training data and resources. However, most of their applications in ASR involve only one of either a pre-trained speech or a language model. This paper proposes integrating a pre-trained speech representation model and a large language model (LLM) for E2E ASR. The proposed model enables the optimization of the entire ASR process, including acoustic feature extraction and acoustic and language modeling, by combining pre-trained models with a bridge network and also enables the application of remarkable developments in LLM utilization, such as parameter-efficient domain adaptation and inference optimization. Experimental results demonstrate that the proposed model achieves a performance comparable to that of modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach.

## introduction mainstream of automatic speech recognition (asr) has shifted from traditional pipeline methods to end-to-end (e2e) ones @xcite . some of the key techniques in e2e asr include a connectionist temporal classification (ctc) @xcite , and an attention-based encoder-decoder model @xcite . alternatively, speech representation models trained in a selfsupervised manner using large amounts of unlabeled speech data have also attracted considerable attention @xcite . fine-tuning these models achieves a better performance in downstream tasks, including asr, particularly in low-data settings. however, these methods typically employ external language modelfused decoding, which complicates the decoding process. although a more recent study @xcite showed remarkable success with a single encoder-decoder model using 680k hours of weakly supervised labeled data without such pre-training models and decoding fusion, it is challenging to collect data on this scale. in natural language processing (nlp) research, large language models (llms) pre-trained on massive amounts of text data have significantly impacted nlp benchmarks, such as question answering, knowledge retrieval, and natural language understanding @xcite @xcite . in recent years, the success of llms has sparked research interest in integrating llms with other modalities such as images, audio, and video @xcite @xcite @xcite @xcite @xcite @xcite @xcite . pioneering studies @xcite utilized llm as a control hub with several well-trained speech expert models. several studies have investigated feeding the audio information to the llm as a prompt so that the decoder-only transformer @xcite can adapt to speech processing tasks @xcite @xcite @xcite @xcite . these methods demonstrate superior performance in various speech-text processing tasks such as asr and automatic speech translation, benefiting from the powerful linguistic knowledge of llms. in addition, some work on asr and speech synthesis have also shown the potential of a pure decoder-only transformer that has yet to be pre-trained @xcite . inspired by recent successes in speech representation learning and llms, this paper proposes a novel e2e asr model by integrating two largescale pre-trained models, hubert @xcite and gpt @xcite . our proposed method benefits from the powerful language modeling and decoding capabilities of autoregressive llms, eliminating the need for complex decoding processes, such as external language modelfused decoding. even with simple greedy decoding, our approach achieves a performance comparable to that of recent well-trained and carefully designed asr models. furthermore, our model can take advantage of recent rapid developments in llm research fields, such as inference optimization tools and rich domain adaptation knowledge, which suggest promising avenues for future progress. the primary contributions of this paper are as follows: we call the proposed model nue-asr foot_0 and release an inference code with the model checkpoint for japanese e2e asr foot_1 . ## related work there has been a recent surge in the use of llms for speech processing tasks, including asr. this emerging trend reflects a growing awareness of the potential of llms to apply their ability to perform a variety of tasks. the primary approach to directly exploiting llms for speech processing tasks is to feed the continuous representation obtained from the speech processing model @xcite @xcite or discretized features from its representation @xcite @xcite @xcite to the llm, empowering the decoder-only transformer to recognize and generate multimodal content, including audio. in practice, continuous speech representations or discrete audio tokens are injected into a linguistic embedding space. in the continuous representation-based approach, transformer-based speech encoders are often introduced to obtain speech representations from acoustic features such as mel filterbank outputs. the sequence length of these features is longer than that of text tokens since these kinds of features are often 50 and 100 frames per second. therefore, downsampling using a convolution layer @xcite or ctc compression @xcite @xcite is often employed. other approaches employ a querying transformer (q-former) @xcite to convert variablelength speech representations into fixed-length output query representations, or an enhancement of the q-former to extract segment-level query representations @xcite . these approaches involve pre-training the speech encoder module or ctc compression network with paired speech and text data using ctc loss, or leveraging the encoder from a well-trained encoder-decoder asr model. this implies a substantial requirement for training the speech processing model with asr tasks. in the discrete representation-based approach, the outputs of speech representation models such as hubert @xcite and w2v-bert @xcite are discretized via k-means, or neural audio codec models such as encodec @xcite are utilized to extract discrete token from speech waveforms. to address the gap between the sequence lengths of speech and text tokens, a technique has also been proposed to remove adjacent duplicate indices. discrete speech tokens can be treated similarly to text tokens by extending the llm vocabulary. however, detailed information in speech may not be fully exploited, and optimization is inherently a two-step process. this paper integrates a pre-trained speech representation model with the llm to utilize the speech representations extracted from speech waveforms as continuous features directly. our method, categorized as a continuous representation-based approach, diverges from prior studies by not relying on signal processing-based acoustic features, thus completely enjoying the advantages of a speech representation model with self-supervised learning. the integrated model with a convolution-based bridge network can be trained with only a single causal language modeling objective. consequently, it achieves fully e2e asr by directly processing speech waveforms, exploiting the combined capabilities of speech models and llms. this paper focuses on the asr task, which is appropriate as a first consideration for integrating pre-trained speech and language models for e2e speech processing tasks. it can assess whether speech information is accurately passed to the llm without missing any content. ## methodologies an overview of the proposed model is presented in fig 1 . this model consists of three main components: speech encoder, bridge network, and llm. ## conclusions this paper proposed a fully e2e asr model that integrates a pre-trained speech representation model and an llm. in the proposed model, the pre-trained hubert and gpt are connected by a convolutionbased bridge network and are fully fine-tuned, where the bridge network passes meaningful continuous latent representations extracted from the speech waveform sample to the llm as speech prompts. experiments demonstrated that the proposed model achieved a performance comparable to that of publicly available modern asr models. we also investigated the domain adaptation capabilities of different text and speech domains using peft. the strength of the proposed model lies in its potential to be easily integrated with various speech and language pre-trained models, including multilingual llms such as @xcite . future work includes further comparison of existing e2e baseline models, use of different pre-training models, and extension to multilingual asr. while this paper focused on the asr task as an extension of pre-trained models, we believe that it is essential to investigate ways to improve performance in specific domains to advance multi-task speech-text processing capabilities by leveraging these models. extending pre-trained models to encompass a wider range of multimodal speech and text tasks, such as automatic speech translation, speech question answering, and other related areas, is a promising future direction.