Interplay of Machine Translation, Diacritics, and Diacritization (2024)

Abstract: We investigate two research questions: (1) how do machine translation (MT) and diacritization influence the performance of each other in a multi-task learning setting (2) the effect of keeping (vs. removing) diacritics on MT performance. We examine these two questions in both high-resource (HR) and low-resource (LR) settings across 55 different languages (36 African languages and 19 European languages). For (1), results show that diacritization significantly benefits MT in the LR scenario, doubling or even tripling performance for some languages, but harms MT in the HR scenario. We find that MT harms diacritization in LR but benefits significantly in HR for some languages. For (2), MT performance is similar regardless of diacritics being kept or removed. In addition, we propose two classes of metrics to measure the complexity of a diacritical system, finding these metrics to correlate positively with the performance of our diacritization models. Overall, our work provides insights for developing MT and diacritization systems under different data size conditions and may have implications that generalize beyond the 55 languages we investigate.

## introduction diacritics are symbols added to a letter to modify its meaning, pronunciation, or phonetic value in an orthographic system @xcite @xcite . these symbols can have a lexical or grammatical function @xcite . in their lexical function, diacritics distinguish one word from another. for instance in yorùbá, diacritics differentiate meanings in words such as: ògún (a deity), ogun (battle), ògùn (a river), ogún (number 20 / inheritance). on the other hand, diacritics also serve a grammatical function by distinguishing one grammatical category from another. for example in iau, diacritics differentiate past and perfect verbs as in: bá figure 1 : illustration of our experimental setup, taking a swedish datapoint 'tack så mycket.' (thank you very much.) as an example. to answer our (rqs), we develop four types of models: three single-task models onlymt dia (trained to translate with diacritized source), onlymt undia (trained to translate with undiacritized source), and onlydia (trained to diacritize); and one multi-task model diamt (trained to translate and diacritize simultaneously). despite the important role of diacritics, we are not aware of work that investigates their effect on mt across languages. in this paper, we attempt to fill this knowledge gap by studying the interaction between machine translation (mt), diacritics and diacritization. diacritization is the task of correctly attaching diacritics to characters. for the interplay between mt and diacritics, we test the effect of keeping and removing diacritics on mt. for the interplay of mt and diacritization, we design a multi-task setting that involves both mt and diacritization. the multi-task models learn to translate and attach diacritics to characters simultaneously. specifically, we raise two main research questions: in a multi-task setting, whether or not, and if so to what extent does diacritization benefit mt (rq1a.), and mt benefit diacritization (rq1b.); and in a single-task setting, whether or not, and if so to what extent, does keeping and removing diacritics affect performance of mt systems (rq2.). an overview of our experimental setup is shown in figure 1 . we also examine how varying training data sizes, hereafter referred to as 'train sizes', impact the model's performance across various languages. our contributions can be summarized as follows: (1) we propose a novel approach to enhance the performance of low-resource machine translation by incorporating diacritization as a multi-task training. (2) we illustrate that, in a single-task setting, the choice of either retaining or omitting diacritics generally has minimal impact on machine translation performance. (3) we propose two categories of language-agnostic metrics designed to assess the complexity of the diacritical system in a language and examine their implications on diacritization performance. to the best of our knowledge, this study represents the most comprehensive analysis of the interplay between diacritics and machine translation. drawing insights from our experimental findings, we offer practical guidelines for researchers and practitioners involved in developing machine translation or diacritization systems. this paper is organized as follows. section 2 is a literature review. experimental settings are provided in section 3. section 4 presents information of the data and our proposed language-agnostic complexity metrics. in section 5, we present and discuss our results and key findings. we conclude in section 6. ## related work we first review existing literature on mt and diacritics, followed by work on diacritization as a standalone task, and finally we discuss the interplay between diacritization and mt. mt and diacritics. there are three primary approaches to handling diacritics in mt: diacritics removal, retention, and restoration. the decision to adopt any of these approaches is motivated by various factors. for example, the inconsistent use of diacritics in a dataset has been identified as a key reason to remove them @xcite . removing diacritics may also be useful for addressing data sparsity and/or out-of-vocabulary issues @xcite . in certain instances, the removal of diacritics has been found to improve bleu score @xcite . while the reasons for diacritics removal are explicit in some cases, other studies have not explicitly stated their motivations @xcite . meanwhile, retaining diacritics can enhance performance for certain languages but may have a detrimental effect on others @xcite . when to retain or remove diacritics remains an open question that this paper also hopes to address. finally, restoration of diacritics has positive impact on mt systems in languages like @xcite . diacritization. a number of works focus on the task of diacritization. for example, belinkov and glass (2015) employ a bi-lstm-based model to create a many to many recurrent neural network to perform diacritization. @xcite build a transformer-based sequence-to-sequence framework to train a diacritization model for arabic. laki and yang (2020) create diacritization models with transformer architecture for 14 east european languages. improving diacritization with mt. @xcite propose an approach for arabic diacritization that uses mt as an auxialiary task in a multi-task setting. their findings reveal that incorporating translation improves performance of diacritization. they hypothesize that this improvement stems from the implicit acquisition of semantic knowledge during the training of the mt process. while their experiments focus solely on arabic, our study expands the scope to cover a broader range of languages, specifically 55 languages across african and european regions. ## conclusion in this study, we empirically explore the interactions between machine translation (mt), diacritics, and diacritization. we conduct comprehensive experiments involving numerous african and european languages across different dataset sizes. in the multi-task learning setting, we observe that introducing diacritization is advantageous for mt in low-resource scenarios but detrimental otherwise. additionally, we find that while mt generally has a negative impact on diacritization, it can facilitate substantial performance improvements for specific languages in high-resource settings. in the context of single-task learning, we determine that the removal or retention of diacritics has minimal influence on mt performance. to assess the complexity of diacritical systems, we propose six languageagnostic metrics, establishing a strong positive correlation with our model's performance.