Blending LLM s into Cascaded Speech Translation: KIT ’s Offline Speech Translation System for IWSLT 2024 (2024)

Abstract: Large Language Models (LLMs) are currently under exploration for various tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT), and even End-to-End Speech Translation (ST). In this paper, we present KIT’s offline submission in the constrained + LLM track by incorporating recently proposed techniques that can be added to any cascaded speech translation. Specifically, we integrate Mistral-7B into our system to enhance it in two ways. Firstly, we refine the ASR outputs by utilizing the N-best lists generated by our system and fine-tuning the LLM to predict the transcript accurately. Secondly, we refine the MT outputs at the document level by fine-tuning the LLM, leveraging both ASR and MT predictions to improve translation quality. We find that integrating the LLM into the ASR and MT systems results in an absolute improvement of 0.3% in Word Error Rate and 0.65% in COMET for tst2019 test set. In challenging test sets with overlapping speakers and background noise, we find that integrating LLM is not beneficial due to poor ASR performance. Here, we use ASR with chunked long-form decoding to improve context usage that may be unavailable when transcribing with Voice Activity Detection segmentation alone.

## introduction this paper provides an overview of karlsruhe institute of technology's speech translation (st) system developed for the offline track of iwslt 2024. we participated in the constrained plus large language models (llms) condition, focusing on the translation direction from english to german. under this condition, llms with parameters of around 7 billion are allowed, and they have proven effective in many nlp tasks. one of the interesting aspects of this condition is how one can effectively integrate them into st systems. in recent years, there has been a significant interest in developing several open-sourced and medium-scale llms @xcite . the adaptability of llms to diverse tasks, using techniques such as in-context-learning @xcite or parameterefficient fine-tuning with 4-bit quantization @xcite , enables their exploitation even with limited resources. with these recent advancements, exploiting llms for st shows great promise and offers several potential benefits. for instance, one common challenge in automatic speech recognition (asr) is dealing with input noise, which can often render it difficult to comprehend the speaker's words. however, llms, trained on vast amounts of data, may excel at predicting words compared to decoders trained solely during asr. moreover, llms possess a richer vocabulary and understanding of complex terminology that task-specific asr systems may lack. motivated by these advantages, various studies have explored the integration of llms into asr @xcite , machine translation (mt) @xcite , and st @xcite . @xcite employ the llm to generate a new hypothesis based on the n-best list of the asr model. this strategy relies on the observation that n-best lists tend to exhibit enough diversity, especially during uncertain conditions, allowing accurate transcript prediction by examining the list. on the other hand, for mt, @xcite proposes leveraging the llm to automatically postedit translations by analyzing the source and hypothesis documents to rectify contextual errors. both approaches are system-agnostic and have demonstrated successful enhancement of system quality. furthermore, it is also the case that cascaded systems are shown to be superior than end to end systems in previous iwslt findings and submissions making the leveraging of llms et al. easily compatible. @xcite . our system builds on these two approaches to effectively use the llms to improve the cascaded st pipeline by refining the intermediate outputs at both asr and mt while maintaining its modular structure. we utilize pre-trained models to create the individual components and fine-tune them with the allowed data. specifically, we employ wavlm @xcite and mbart50 @xcite we present our main findings below: ## data this section describes the evaluation and training data we use in our experiments. for evaluation, we report results on the tst2019 and acldev @xcite test sets to compare with findings from previous works @xcite . we also use the eptv (european parlament activities), itv (tv series), and peloton (fitness tv) dev sets from the subtitling track consisting of overlapping speakers with different accents to evaluate the asr performance in challenging scenarios. as the data conditions did not change from iwslt23 to this year, we rely on the data processed from last year's submission (kit'23) @xcite . for the training data of asr, we use the same system that used common voice @xcite , librispeech @xcite , @xcite , ted-lium v3 @xcite , and voxpopuli @xcite . while for mt fine-tuning, we use the cleaned training data from last year created from the available parallel data. this includes europarl v7 and v10 @xcite , newscommentary v16, opensubtitles v2018 (lison and tiedemann, 2016), tatoeba @xcite , elrc-cordis_news and ted2020 (reimers and gurevych, 2020) and consists in total of 23 million sentence pairs. for the rest of the paper, we refer to the full parallel data as seed and ted2020 as in-domain. ## conclusion this system paper presented kit's submission for the offline track in the constrained + llms condition, focusing on the english-to-german translation direction. using modular techniques, we successfully integrated llms into any cascaded st pipeline. additionally, we highlighted the benefits of long-form decoding in scenarios involving noisy and overlapping speech. for future work, we aim to explore robust techniques for integrating llms that can effectively handle challenging scenarios where asr quality is sub-par. furthermore, the translation's latency is quite high as it needs to call the llm twice. however, integrating quality estimation techniques to decide when we need the llm can limit the effects of the high latency problem. set after every epoch. we stopped the training after the dev loss did not increase after 10 epochs. for fine-tuning the llm with qlora we use the peft @xcite along with the transformers library. we add lora adapters to the target modules @xcite . we set the adapter rank to 16, alpha to 32 and lora dropout to 0.1. we use a batch size of 8, learning rate of 5e -5 with other parameters set to default. after every 200 steps, we validate and terminate the training if it does not improve 10 consecutive times. during inference, we use beam search for all asr, mt and llm components. the asr and mt decode with beam size of 5, whereas the llm does it with beam size of 3.