LL a ST : Improved End-to-end Speech Translation System Leveraged by Large Language Models (2024)

Abstract: We introduces ***LLaST***, a framework for building high-performance Large Language model based Speech-to-text Translation systems. We address the limitations of end-to-end speech translation (E2E ST) models by exploring model architecture design and optimization techniques tailored for LLMs. Our approach includes LLM-based speech translation architecture design, ASR-augmented training, multilingual data augmentation, and dual-LoRA optimization. Our approach demonstrates superior performance on the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs.We believe this effective method will serve as a strong baseline for speech translation and provide insights for futureimprovements of the LLM-based speech translation framework.

## introduction the speech-to-text translation (st) task, which transcribes spoken language into written text in a different language, is pivotal for bridging communication barriers. this capability has a wide array of applications, including facilitating global communication, enabling automatic subtitling, and aiding in language learning. conventional st systems are typically composed of two distinct components: an automatic speech recognition (asr) module that transcribes spoken speech into written text in the source language, and a machine translation (mt) module that subsequently translates this text into the target language. these modules can be trained using paired asr and text-to-text translation data, significantly enhancing the overall performance of st systems. despite their modular design, cascade systems are prone to error accumulation, where inaccuracies from the asr stage are compounded in the mt phase, often leading to sub-optimal translations. recently, the focus has shifted towards the development of end-to-end speech translation (e2e st) models that bypass the need for separate automatic speech recognition (asr) and machine translation (mt) modules by directly converting spoken input into text in the target language. nonetheless, these approaches often necessitate extensive training datasets and are contingent upon sophisticated model architectures to achieve strong performance. speech translation is intrinsically linked to natural language processing (nlp), as it involves the conversion of spoken language into written text in a target language, necessitating a deep understanding of both the source and target languages' linguistic structures and semantics. the unprecedented capabilities that large language models (llms) have demonstrated across a variety of nlp tasks @xcite have opened up new possibilities to construct potent speech translation systems by leveraging these llms as a foundation. recent research has seen some preliminary attempts exploring this direction @xcite @xcite . despite these advancements, the question remains on how to most effectively harness the vast potential of llms to develop a high-performance st system in an efficient manner, without compromising on quality or scalability. in this study, we focus on the exploration of best practices for constructing an effective speech translation system powered by large language models (llms), which we term llast. the paper delves into the core aspects of the development process, specifically the model architecture design and optimization techniques. our exploration begins with the creation of a minimalist model architecture, examining the selection of key modules such as the speech encoder and llms. subsequently, we investigate training strategies, including asr-augmented training and dual-lora optimization. moreover, to deepen our understanding of scaling laws in llmbased st, we also scrutinize the impact of model size variations. through these concerted efforts, we aim to uncover insights that can significantly enhance the performance and training efficiency of llast. our contributions are listed as follows. ## method this section presents our method in detail. we begin by introducing the problem setting of the speech-to-text translation task in sec. 3.1. then, we explain the structure of the proposed model in sec. 3.2, followed by the description of the training and inference processes in sec. 3.3. ## experiments in this section, we conduct a series of experiments to validate the effectiveness of our method. we start by detailing experimental configurations in sec. 4.1, followed by an overview of quantitative results in sec. 4.2. ## ablation analysis in this section, we delve into a meticulous ablation study and analysis of llast. we begin by examining the impact of model architecture in sec. 5.1, followed by an exploration of optimization strategies in sec. 5.2. finally, we investigate the relationship between model scale and performance in sec. 5.3. ## limitation while our study has yielded significant findings, it is crucial to recognize the limitations that may impact the interpretation and broad applicability of our results. although we delved into the architecture design and optimization strategies, our reliance on a relatively narrow data source and the use of short voice samples could potentially affect the generalizability of our outcomes. to address this, future research will expand to encompass a more diverse array of data. moreover, due to the constraints of our current resources, we have not ventured into exploring larger language models or a broader range of language pairs in this study. in speech translation, llast's use of llms raises concerns in actual application as teh following: (a)probabilistic inaccuracy, mistranslations may occur due to nuances or dialects, impairing accuracy and cultural relevance. (b)data imbalances, insufficient representation in training data can lead to biased translations or reduced effectiveness for underrepresented groups. (c)deployment challenges, large model sizes and complexity may cause latency, high energy usage, and device compatibility issues. (d)harmful content generation, despite post-processing, risks persist; ongoing monitoring, filter refinement, and expert collaboration are needed. ## conclusion we presents the development and analysis of llast, a novel speech translation model that harnesses llm in this work. the study demonstrates that integrating well-tuned speech encoders like whisper with different sizes of llms significantly improves speech-to-text translation performance. through meticulous ablation studies, it is shown that applying dual lora optimization to both speech encoders and llms leads to substantial gains in bleu scores. additionally, experiments confirm that increasing the scale of either the speech encoder or the llm positively impacts performance, though the rate of improvement decreases as size increases. furthermore, incorporating asr augmentation and multilingual training further enhances the model's performance on specific language pairs. overall, llast underscores the potential of large language models for advancing speech translation tasks and offers valuable insights into their effective integration.