Improving Low Resource Speech Translation with Data Augmentation and Ensemble Strategies (2023)

Abstract: This paper describes the speech translation system submitted as part of the IWSLT 2023 shared task on low resource speech translation. The low resource task aids in building models for language pairs where the training corpus is limited. In this paper, we focus on two language pairs, namely, Tamasheq-French (Tmh→Fra) and Marathi-Hindi (Mr→Hi) and implement a speech translation system that is unconstrained. We evaluate three strategies in our system: (a) Data augmentation where we perform different operations on audio as well as text samples, (b) an ensemble model that integrates a set of models trained using a combination of augmentation strategies, and (c) post-processing techniques where we explore the use of large language models (LLMs) to improve the quality of sentences that are generated. Experiments show how data augmentation can relatively improve the BLEU score by 5.2% over the baseline system for Tmh→Fra while an ensemble model further improves performance by 17% for Tmh→Fra and 23% for Mr→Hi task.

## introduction speech translation (st) systems have multiple applications. they can be utilized in a wide range of scenarios such as closed captioning in different languages while watching videos or even as a realtime assistant that translates speeches to live audiences. one persistent challenge for speech translation systems continues to be performing translations for low resource language pairs. 1 the iwslt 2023 @xcite shared task for low resource speech translation targets 8 language pairs that include tunisian arabic (aeg) to english (en), irish (ga) to english (en), marathi (mr) to hindi (hi), maltese (mlt) to english (en), pashto (pus) to french (fr), tamasheq (tmh) to french (fr), and quechua (que) to spanish (es). this paper outlines a low resource speech translation system (from the amazon alexa ai team) for 2 language pairs, namely, tamasheq-french (@xmath0. depending on the type of output that is generated the end-to-end speech translation task has two formats: (a) speech-to-text (s2t), and (b) speechto-speech (s2s). there are two types of st systems. the first is a cascaded system where speech recognition and language translation are decoupled. 2 the second is an end-to-end (e2e) model that combines both audio processing and language translation. we design and evaluate an e2e model in this paper. in the past, various approaches have been proposed to build e2e low resource speech translation models. @xcite designs an initial system that is an encoder-decoder architecture that integrates a convolutional neural network (cnn) and recurrent neural network (rnn). @xcite try to improve st models for low resource languages by pretraininig the model on automated speech recognition (asr) task. @xcite propose a new learning framework called allost that trains a transformer architecture with languageindependent phonemes. @xcite al. (2022) in this paper, we extend the previous work with the following contributions: apart from these contributions, we also explore post-processing techniques with large language models (llms), focusing on re-ranking generated translations @xcite , correcting the grammar of translations and masking tokens so that the llm can complete the translate sentence. these methods though, did not yield any noticeable improvement. the paper is organized as follows: section 2 describes our speech translation system, 3.1 has details about the datasets for various language pairs, 3.2 contains analysis of our experimental results and we finally conclude in 4. ## conclusion in this paper, we explore multiple types of strategies to improve speech translation for two language pairs: tamasheq-french (@xmath1: test 2023 results for @xmath2. marathi-hindi (@xmath3. we show expanding the training dataset with paraphrases of translated sentences as well as an ensemble model (of trained st models with different seeds and data augmentation methods), improves performance over the baseline model for (@xmath4. similarly, an ensemble model for marathi-hindi (@xmath5. we also explore the use of large language models and find that post-processing using them did not show any noticeable improvement.