UOM -Constrained IWSLT 2024 Shared Task Submission - M altese Speech Translation (2024)

Abstract: This paper presents our IWSLT-2024 shared task submission on the low-resource track. This submission forms part of the constrained setup; implying limited data for training. Following the introduction, this paper consists of a literature review defining previous approaches to speech translation, as well as their application to Maltese, followed by the defined methodology, evaluation and results, and the conclusion. A cascaded submission on the Maltese to English language pair is presented; consisting of a pipeline containing: a DeepSpeech 1 Automatic Speech Recognition (ASR) system, a KenLM model to optimise the transcriptions, and finally an LSTM machine translation model. The submission achieves a 0.5 BLEU score on the overall test set, and the ASR system achieves a word error rate of 97.15%. Our code is made publicly available.

## introduction speech translation (st) may be defined as the task of transforming audio in a source language to its transcription in a target language. st is generally tackled through two main approaches: the first being an end-to-end approach; with the source language audio serving as input to the model, which in turn produces a transcription in the target language as output, the second being a pipeline or cascading approach; suggesting multiple systems with varying responsibilities, primarily generating asr transcription and machine translation. a meta-net white paper series confirms the maltese language as low-resourced; meaning it has little support for speech technology, including translation tasks @xcite . this paper introduces a cascading system that utilises an asr system to generate transcriptions in the source language, a language model to improve ## literature review the literature review focuses on previous attempts at automated speech recognition (asr) and machine translation (mt), in particular, when applied to the maltese language. furthermore, the main models attempted for this task are defined, these being: hmm, deepspeech 1 for asr, lstm and transformers for mt. ## evaluation and results this section presents and discusses the models' results. the official results for our constrained task submission are presented in tables 2 and 3 . the final pipeline result was significantly influenced by the asr performance. it can be extrapolated that the high word error rate (wer) of the asr is a result of the limited training data, which was not adequate to train a capable asr system. incoherent speech recognition outputs were considered 'out of domain' by the machine translation system since it was trained on meaningful data. to further evaluate and understand the results, specific outputs of both the asr as well as the mt system were analysed. it may be noted that the asr output is relatively poor; with most outputs consisting of invalid maltese words. in addition to using deepspeech 1, we made use of a kenlm trained specifically for this task, but whilst some improvements were seen, as illustrated in to further illustrate the asr issues, the first audio file of the cv test set was transcribed by the model as: "dan ma sarqat". the first two words were predicted correctly, however, the last word was invalid. the correct transcription should have been: "dan ma sar qatt", meaning "this was never done". since this is a pipeline setup, the resulting transcription was passed to the mt system. the translated output was "this doesn't happen to him". the output here was not surprising, since the two words that the asr got correct (dan ma) roughly mean he has never [...]. since the word that the asr got incorrect does not exist in the maltese language (sarqat), it is likely that the mt system treated it as an unknown token. admittedly, this was one of the few examples that the asr system performed well in. results were exceptionally poor when a named entity was included. for example, the name simon busuttil was outputted as sajminbużutiel. this is expected due to the small size of the training data. apart from this, the asr model struggled to understand when a word starts and ends. in most cases, the output sounds phonetically similar to what the actual transcription should be, however, the spelling is incorrect. for example, the word mhux was tran-4 159 scribed as mux, which is understandable as the 'h' is silent. overall, the asr model performed poorly, with most resulting sentences not resembling the actual transcription, highlighted by the 97.15% wer. these errors naturally propogated to the mt system. since the dataset of the mt system is also constrained and very limited in nature, it did not have the implicit understanding of the language to identify the typos written by the asr system (such as mux instead of mhux). this is even harder with phonetic misspellings. generally, the mt system output a (seemingly) random response since the input given by the asr system is equally poor. overall, it is evident that an increase in training data would have yielded better results. the asr set-up makes it difficult to evaluate the mt system alone, given the model pipelining and overall poor performance. ## conclusion and future work this paper presents the different approaches to st for low-resource languages under constrained settings. a short overview of previous research into challenges associated with speech translation was presented, as well as specific attempts and pipelines used for the task. the final pipeline consisted of a deepspeech 1 model, kenlm model and lstm model, each fine-tuned for the task at hand. the final results show that the constrained setting has an extreme impact on the models performance, with a final wer of 97.15%. the very poor asr performance highlights the challenges present in lowresource settings. future work on asr includes the use of higher-quality training data, as well as dealing with named entities in the data itself. it is also suspected that pre-trained models would likely yield better results in low-resourced environments, helping to compensate for data scarcity.