UM - DFKI M altese Speech Translation (2023)

Abstract: For the 2023 IWSLT Maltese Speech Translation Task, UM-DFKI jointly presents a cascade solution which achieves 0.6 BLEU. While this is the first time that a Maltese speech translation task has been released by IWSLT, this paper explores previous solutions for other speech translation tasks, focusing primarily on low-resource scenarios. Moreover, we present our method of fine-tuning XLS-R models for Maltese ASR using a collection of multi-lingual speech corpora as well as the fine-tuning of the mBART model for Maltese to English machine translation.

## introduction speech translation (st), or speech-to-text translation, involves converting speech in a source language into written text in a target language. with the rise of deep learning, steep progress has been made in this field and many other areas that fall under the natural language processing (nlp) umbrella @xcite . however, development for low-resource languages has continued to present difficulties and obstacles due to a variety of factors, including the lack of sufficient training data, language experts and other resources @xcite . the international workshop on spoken language translation (iwslt) shared task is an annual competition that aims to foster research in the field of speech translation. with its low-resource track, it also contributes to advanced research for speech translation in low-resource scenarios. in this paper, we present our submission to the low resource track: a pipeline system for english-maltese speech-to-text translation. we begin by discussing the state of the art in speech translation and describe the two main approaches, cascade and end-to-end. afterwards, we briefly summarise the challenges posed by lowresource languages and possible mitigation strategies. we then describe our system, a pipeline approach containing an internal automatic speech recognition (asr) component and the outward facing machine translation (mt) component. the asr component can use one of five fine-tuned xls-r @xcite models, whereas the mt stage always uses an mbart-50 model. ## literature review the following literature review aims to provide an overview of previous iwslt st submissions, with a particular focus on low-resource scenarios. the review is divided into two sections; where the first explores the general approaches and challenges associated with low-resource st, and the second section discusses previous approaches to low-resource st as applied to iwslt. ## methodology for this task, we decided to use a cascade system where the asr and mt components were trained separately but evaluated jointly. in this section, a detailed description of both components is given. first, the training data is described, followed by the pre-processing steps applied to said data. next, the models are introduced, and lastly training, the training procedure is outlined. ## evaluation and results the test set proved to be difficult for both the asr and mt systems to get right due to the type of language used as well as the speed of the speech in general. the mt-50 output is relatively similar to the reference sentence, except for a few minor errors, including the misspelling of the name "mark". however, this should still be a good sentence to input into the machine translation system. in stark contrast to the mt+all system outputs. the main issue here is that this system does not output maltese characters and completely omits them, which presents an issue for the downstream translation task since the meaning of the word is lost in these cases. machine translation also had similar issues. the training set contained data coming from legal texts, so the data is very formal, making it very difficult to evaluate since the input text is very informal and unlike the legal text data seen. unfortunately, most of this is unrelated to what was actually said. looking into the translations deeper, one can see the reasoning behind certain translations. for example, the dataset does not contain a lot of conversational data, so general greetings like "merhba" may not be present. this case is represented by the translation of the token "merba", which was translated to "four". here the token "merba" (welcome) was mistaken for "erba" (four). other mistakes include those that are phonetically plausible but grammatically incorrect output, such as the transcription for "podcast" which was transcribed as "pot kast". certain expressions like "din id-darba" were correctly translated to "this time", however rarer words such as "polemikuż" and "kontroversjali", both of which have the same meaning as "controversial", seemed to not appear in the translation. continuing the trend observed in @xcite , the use of additional languages when finetuning an xls-r model proved to be detrimental towards the final output. as observed in section 4, some models trained with additional data lost the ability to transcribe maltese-specific alphabetic characters. so far, the character-to-sound pair was always made with the source language in mind. for example, the french 'ç' is transformed into the 'c' character, which itself is only present in the maltese alphabet when english words are loaned and used directly. it's important to note that code-switching to english is very common in maltese speech. future work should explore these character-to-sound pairs. ## conclusion and future work this paper showcased the results of a speech-totext translation system in the direction of maltese to english. a cascade system is chosen, where asr and mt models are pipelined together. the automatic speech recognition system chosen is based on xls-r and is fine-tuned on data from different languages. the best-performing model was the xls-r 300m model fine-tuned on 50 hours of maltese speech. the machine translation system chosen is based on mbart-50, and it was finetuned on parallel maltese -english data. aside from fine-tuning, no modifications were made to the pre-trained models. for future work, we have various potential avenues for improvement. for machine translation, since mbart-50 was not pre-trained on maltese data, extending the vocabulary to include maltesespecific tokens would improve the representation and potentially the downstream performance as well. moreover, our approach solely relied on parallel data and did not investigate techniques which leverage monolingual data, such as backtranslation. monolingual corpora, such as korpus malti v4 @xcite , not only provide significantly more data but also have more diversity in terms of domains. apart from this, it might be beneficial to perform more quality checks on the parallel dataset since some portions of the publicly available datasets are automatically crawled and, in some cases, contain noise. regarding asr improvement, other systems, such as whisper and, most recently meta's massively multilingual speech (mms) project should be tried and evaluated. the research made in multi-lingual fine-tuning needs to be more focused. one idea we can explore is the transliteration of foreign alphabetic characters into maltese characters, e.g. 'h' in english would be transliterated as 'h'. it is also the case that no language model is used to correct the asr output mistakes; this is currently our next milestone.