Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing? (2024)

Abstract: The field of natural language processing (NLP) has recently witnessed a transformative shift with the emergence of foundation models, particularly Large Language Models (LLMs) that have revolutionized text-based NLP. This paradigm has extended to other modalities, including speech, where researchers are actively exploring the combination of Speech Foundation Models (SFMs) and LLMs into single, unified models capable of addressing multimodal tasks. Among such tasks, this paper focuses on speech-to-text translation (ST). By examining the published papers on the topic, we propose a unified view of the architectural solutions and training strategies presented so far, highlighting similarities and differences among them. Based on this examination, we not only organize the lessons learned but also show how diverse settings and evaluation approaches hinder the identification of the best-performing solution for each architectural building block and training choice. Lastly, we outline recommendations for future works on the topic aimed at better understanding the strengths and weaknesses of the SFM+LLM solutions for ST.

## introduction the natural language processing (nlp) landscape has recently undergone a paradigm shift with the emergence of foundation models @xcite . among them, large language models (llms) have revolutionized text-based nlp, showcasing remarkable capabilities across a wide range of nlp tasks @xcite . this unprecedented success has spurred research into creating foundation models for other modalities, including speech processing @xcite . building on the translation abilities of llms @xcite @xcite @xcite and the remarkable speech recognition and understanding capabilities achieved by speech foundation models (sfms) @xcite @xcite , researchers are now actively exploring their combination. the resulting large multimodal models leverage, on the one hand, the sfm ability to encode speech content into rich and high-level representations and, on the other, the extensive linguistic knowledge of the llm to generate fluent outputs and address a wide range of tasks @xcite @xcite @xcite . focusing on the speech-to-text translation (st) task -the scope of this paper -the rapid pace of the advancements has led to multiple parallel endeavors, resulting in a variety of solutions. while all these efforts have the merit of demonstrating the viability and effectiveness of this line of work, their contemporaneity, along with methodological inconsistencies, hinders a fair comparison. for this reason, we provide a systematic analysis of the proposed sfm+llm solutions for st with the multiple goals of identifying their similarities and differences, organizing the lessons learned, and suggesting future research directions, along with best practices for insightful evaluations. at its core, this paper addresses two key questions: what is there? we survey the publicly available works that propose an sfm+llm solution for st, resulting in 9 papers (henceforth referred to as ,..., ), and analyze them ( §2) focusing on two orthogonal aspects: architectural building blocks ( §2.1): we delve into the sfm+llm architectures, identifying a common abstraction made of 5 building blocks and underscoring similarities and differences in the sfm and llm choices, along with the strategies adopted for combining them; training and evaluation ( §2.2): we inspect the training data, tasks, and strategies employed in the studies, as well as evaluation data and supported language pairs, gathering insights about promising solutions, and highlighting the sparsity of the current landscape; what is missing? we conclude by underscoring the importance of establishing a standard training setting based on open data to ease direct comparability across works, and by identifying aspects that need further investigation to better understand the potential of sfm+llm combination for st ( §3). in this section, we explore two key aspects of sfm+llm research in st: first, we delve into the architectural components of sfm+llm models ( §2.1); second, we examine the training and evaluation settings utilized in these studies ( §2.2). ## what is missing? alongside the need for focused and thorough analyses devoted to identifying the best-performing option for each architectural building block highlighted in §2.1 and the effects of the training choices discussed in §2.2, in the following we identify blind spots that need to be addressed for a more grounded and insightful progress in research on sfm+llm solutions for st. open standard training settings. as highlighted throughout the previous section, the lack of common experimental settings prevents the fair and direct comparison of different works. the adoption of public and standard training settings holds paramount importance in advancing research and fostering progress within the scientific community @xcite . on the one hand, it enables the comparison among various works, thus providing actionable insights on the most promising architectural choices. on the other, it fosters inclusivity and accessibility, allowing researchers without access to large proprietary corpora to contribute to the field @xcite , and thus supporting ai democratiza-tion in the development process @xcite . therefore, we advocate for future research to adhere to established data-setting standards, paving the way for cumulative progress and shared understanding in the field. however, as experimenting with different data sizes is also an interesting topic and findings may vary depending on the datasets and the tasks used in the training stage (see §2.2), it is debatable which would be the most appropriate training set. in the english-to-many scenario, researchers commonly adhere to the iwslt offline constrained data condition, 5 comprising ∼4.5k hours of english audio, while, for smaller-scale experiments, must-c (∼500 hours) is a widespread option. for many-to-english settings, covost2, mtedx @xcite , and europarl-st @xcite are open datasets with st references and can be complemented with larger asr resources such as commonvoice @xcite and voxpopuli. notice that, by advocating for standardized and public training data, we do not imply that researchers should not investigate the effects of training in different data conditions. rather, we suggest that, for works primarily focused on defining new architectural solutions, reporting results for (at least) a standard setting would ease comparisons with other alternatives and reduce the overall computational costs. standard and reliable evaluation. the comparison between different methods is currently hindered not only by different training conditions but also by the fact that practitioners do not systematically present results on a common open benchmark. the reliability of the reported evaluation is also questioned by the fact that most sfm and llm are trained on proprietary data and their training data is not disclosed. this can hinder their proper evaluation because of the possible data contamination with the test set @xcite . furthermore, all works rely on the bleu metric @xcite , except for , which additionally reports comet @xcite . although we acknowledge that bleu is still widespread @xcite despite the wide consensus on its limited dependability and correlation with human judgments @xcite , we argue that this specific scenario exacerbates the need for adopting alternative metrics to assess translation quality. the main reason behind this argument is the well-known tendency of n-gram-based metrics to penalize translations generated by llms that are, in general, less literal @xcite . as a suggestion for future works, we recommend reporting at least one semantic metric (e.g., comet), and, preferably, multiple metrics. we also believe that reporting scores on open and multilingual benchmarks, such as covost2, would improve comparability across studies without the need for re-running costly experiments, thereby promoting faster, cost-effective progress within the research community. comparison with standard st approaches. in analogy to studies @xcite and initiatives @xcite dedicated to assessing the strengths and weaknesses of the two established end-to-end and cascade st paradigms, the emergence of the sfm+llm solution calls for thorough and finegrained evaluations to investigate its peculiarities compared to other, more traditional methods. this need is also motivated by a recent analysis in the context of text-to-text translation @xcite , which showed that llms are partly affected by long-standing problems of neural approaches (e.g., the translation of rare entities and out-ofdomain settings), while they do not face others (e.g., the translation of long sentences) and suffer from new ones (e.g., pre-training data imbalance across domains and languages). among the new problems, a noteworthy element is the inference efficiency: the comparison with the standard methods -which typically rely on models of limited size (100-300m parameters) -should account for this aspect, which is critical for social, economic, and environmental reasons @xcite and tasks such as simultaneous translation @xcite @xcite . along this line, important research directions include i) pruning the llm (and possibly the sfm) in a task-aware manner @xcite @xcite , ii) dynamic layer selection during decoding @xcite @xcite , and iii) efficient decoding strategies @xcite @xcite . in addition, the speech source contains a wide range of information that can be exploited depending on the paradigm used (e.g., prosody is not handled by cascade systems - @xcite . as such, the ability of sfm+llm models to leverage this information has to be investi-gated. the fine-grained evaluation of these aspects calls for the comparison of sfm+llm models with other paradigms on tailored test suites @xcite , similar to those used in mt @xcite . in-context learning assessment. one of the most interesting emergent abilities of llms @xcite is their ability to exploit a few demonstrations or examples to perform a task or enhance their performance on it @xcite . this ability -referred to as in-context learning (icl) @xcite -is one of the main motivations for integrating an sfm and an llm into a single st model. however, the transfer of icl capabilities of llms to the speech modality, and, even more so, to the sfm+llm approach to st, cannot be taken for granted. in fact, while the icl ability of sfm+llm has been successfully assessed in asr and slu @xcite @xcite ) -also within the retrievalaugmented framework @xcite , where the relevant context is retrieved from a knowledge base @xcite -the only attempt in st has not been similarly successful @xcite . moreover, sfms like whisper feature similar (yet limited) icl capabilities @xcite , which might make the sfm+llm integration not even necessary. for this reason, investigating whether and to what extent the integration of sfms with llms transfers the icl ability of the latter to the st task is an important and interesting avenue for future studies. ## conclusions the st landscape has recently witnessed the emergence of a new paradigm, which is the combination of sfms and llms into single st models. to summarize the lessons learned and establish a unified framework, we surveyed the existing works on the topic, analyzing their architectural and training choices. as a result, we identified a common abstraction of the surveyed sfm+llm architectures, which consists of five building blocks: i) the sfm extracting high-level speech representations, ii) the length adapter compressing such sequence of features, iii) the modality adapter mapping them to an embedding space more suitable for the llm, iv) the prompt-speech merger combining the speech information with an adequate prompt for the llm, and v) the llm generating the output translation. subsequently, we highlighted how the current lack of standardized training recipes and evaluations hinders the direct comparison of the proposed approaches, limiting the possibility of extracting precise and unified indications. lastly, we pointed out the need for thorough comparisons with standard st approaches and in-depth investigations of the inherent capabilities of sfm+llm solutions in order to shed light on its real potential for st.