Together We Can: Multilingual Automatic Post-Editing for Low-Resource Languages (2024)

Abstract: This exploratory study investigates the potential of multilingual Automatic Post-Editing (APE) systems to enhance the quality of machine translations for low-resource Indo-Aryan languages. Focusing on two closely related language pairs, English-Marathi and English-Hindi, we exploit the linguistic similarities to develop a robust multilingual APE model. To facilitate cross-linguistic transfer, we generate synthetic Hindi-Marathi and Marathi-Hindi APE triplets. Additionally, we incorporate a Quality Estimation (QE)-APE multi-task learning framework. While the experimental results underline the complementary nature of APE and QE, we also observe that QE-APE multitask learning facilitates effective domain adaptation. Our experiments demonstrate that the multilingual APE models outperform their corresponding English-Hindi and English-Marathi single-pair models by 2.5 and 2.39 TER points, respectively, with further notable improvements over the multilingual APE model observed through multi-task learning ( +1.29 and +1.44 TER points), data augmentation ( +0.53 and +0.45 TER points) and domain adaptation ( +0.35 and +0.45 TER points). We release the synthetic data, code, and models accrued during this study publicly for further research.

## introduction automatic post-editing (ape) aims to address the limitations of an unknown machine translation (mt) system by automatically correcting recurrent translation errors, thereby enhancing the overall translation output @xcite . wmt has been driving the research in this field through apeshared tasks over the years @xcite @xcite . however, as the generation of authentic (human-annotated) ape data is difficult, publicly 1 multilingual-ape available ape data is limited in terms of language pairs and sizes. the unavailability of an adequate amount of ape data for closely related languages has resulted in research focusing on different ape models for different language pairs. therefore, while we see considerable research devoted to advancing neural machine translation (nmt) systems through multilingual mt @xcite @xcite @xcite , to the best of our knowledge, multilingual ape (mape) has not studied yet. akin to how using multiple languages together helps improve translation capabilities, extending this approach to the ape task should enhance the translation correction capabilities for closely related languages despite limited resources. this study focuses on english-hindi (en-hi) and english-marathi (en-mr) ape. hindi and marathi are two prominent indo-aryan languages with notable linguistic similarities because of their common linguistic ancestry @xcite . for example, both languages utilize the devanagari script for writing and exhibit substantial vocabulary overlap @xcite , largely due to their shared roots in sanskrit. grammatically, they both follow the subject-object-verb (sov) sentence structure, providing a familiar syntactic framework for speakers of either language @xcite . such linguistic similarities enable crosslinguistic transfer and suggest that advancements in multilingual ape (mape) could be particularly beneficial for improving translation quality across both these language pairs. our contributions are: ## related work wmt has been driving the ape research over the years through the ape shared tasks @xcite @xcite . we see a paradigm shift from the wmt18 ape shared task onwards, where supervised transformed-based ape approaches became prominent for the task of improving high-quality translations obtained from neural mt systems @xcite . most neural ape approaches leverage transfer learning through the use of pre-trained language or translation models in ape model training @xcite @xcite . to obtain latent representations of source and target sentences, @xcite employ multilingual or crosslingual models. while these approaches followed a two-step training approach where the models are initially trained on synthetic ape data and then on authentic ape data, @xcite demonstrated an increasingly complex multi-step training approach, referred to as the curriculum training strategy (cts), under which a model is gradually trained on more and more advanced tasks enhances ape performance. yang et al. (2020); deoghare and bhattacharyya (2022) 3 methodology neural ape approaches model the ape problem as a multi-source translation task. for each pair of the input source sentence and its machine-generated translation as @xmath0, @xmath1, with lengths t x , and t z , respectively, the ape is trained in a supervised fashion to generate the post-edited translation ŷ = {ŷ i } t ŷ @xmath2. for training the model, the cross-entropy loss is used as shown in equation 1. 10801 l ap @xmath3,e log (ŷ w,e ) where |s| and |v | denote the length of the sequence and the vocabulary size, respectively. ŷw,e , and y w,e represent the ape output and its reference, respectively. while ape aims to generate corrected translations, qe systems are focused on assessing the extent of correction required (sentence-level qe) and precise identification of locations where those corrections are required (word-level qe). the taskspecific head is added to the representations obtained from the final encoder layer to perform these tasks. the sentence-level qe task is modeled as a regression task with the goal of predicting the direct assessment (da) score for given source sentences and their translation. equation 2 shows the mean squared error (mse) loss used for this task. where ŷda and y da are the predicted da score and the ground truth, respectively. the word-level quality estimation task is treated as a token-level classification task in which each token is classified into ok/bad tags to denote its correctness. the cross-entropy loss utilized for this task is shown in equation 3. where y word and ŷword denote ground truths and predicted tags. this complementary nature of ape and qe tasks has been exploited in the qe-ape multitask learning-based framework proposed by @xcite in which a two-encoder singledecoder model is jointly trained on ape, sentencelevel qe, and word-level qe tasks. they observed higher ape performance when a sophisticated multitask learning algorithm, referred to as nash-mtl @xcite , is used instead of linear scalarization where the task-specific losses are simply added together as shown in equation 4. we use the nash-mtl approach as described in @xcite . model architecture in our work, we follow the approach proposed by @xcite for training the ape models and also use the same model architecture as shown in figure 1 . the architecture without the components shown in the dashed rectangle illustrates the architecture of the non-multitask learning-based ape model before the fine-tuning stage. when fine-tuning, no changes are made to the architecture in the case of single-task experiments. however, the qe taskspecific heads are added to the shared representation layer that receives inputs from the final encoder layers for the multitask learning-based experiments. except for the transfer learning-based experiments, the encoder weights are initialized using indicbert @xcite , and the decoder weights are initialized randomly. we train all models using the curriculum training strategy (cts), same as @xcite . we consider the following three baselines in our work. ## multilingual automatic post-editing in the hope of getting help from the shared linguistic features of hindi and marathi, we perform various mape experiments hierarchically. we leverage synthetic and authentic ape triplets of both en-hi and en-mr language pairs for all mape experiments. we combine and shuffle the synthetic ape datasets of both language pairs to create multilingual synthetic ape dataset. similarly, we combine and shuffle the authentic ape datasets of both pairs to create multilingual authentic ape dataset. these combined sets of multilingual synthetic and authentic ape datasets are then used to train a mape model. as an initial approach, we do not provide the model any indication about the language id on the target side and represent it as w/o-langid in further discussion. extending authentic triplets w/-langid (only authentic w/-langid) this experiment uses multilingual synthetic ape data as used in the w/o-langid experiment. however, we prepend a respective language id to a source sentence in each multilingual authentic ape dataset triplet. extending synthetic and authentic triplets w/-langid (w/-langid) in this experiment, we prepend a respective language id to a source sentence in each ape triplet of the multilingual synthetic and multilingual authentic ape dataset. based on the experimental results, we perform further mape exploration considering w/-langid as the base experiment. ## datasets the mt and ape datasets of english-hindi and english-marathi are smaller than other language pairs like english-german, english-russian, and english-chinese. considering the amount of mt parallel corpus, synthetic ape triplets, and authentic ape triplets for these language pairs, we consider english-hindi and english-marathi to be lowresource language pairs. for english-marathi ape, we use datasets released through the wmt22 @xcite english-marathi ape shared task. the ape data consists of 18k authentic (human-generated post-edits) and 2.5m synthetic @xcite ape triplets. the authentic ape triplets come from the news (6.5k), tourism (6.5k), and health (5k) domains, and the synthetic ape data triplets from multiple domains. we use an in-house created ape corpus consisting of 9k training triplets for the english-hindi pair, as an open-source ape corpus for this language pair is unavailable. source sentences used in the authentic ape corpus are taken from publicly available data. 2.25k authentic triplets come from each of the news, tourism, law, and education domains. also, the mt system used for translating the source sentences is open-source. apart from the training set, additional 1k-1k authentic triplets are set aside to be used as development and test sets. the source sentences in these datasets cover multiple domains. additionally, we generate multidomain 2.5m synthetic ape triplets from a subset of the anuvaad foot_0 parallel corpus following the approach proposed by @xcite . along with this corpora, its details, annotation process, and guidelines are planned to be released through the upcoming wmt ape shared task. further, we also use the bpcc @xcite corpus for both pairs to train an mt model during the first stage of the cts. we use the english-marathi and english-hindi sentence-level qe data released through the wmt22 @xcite and wmt23 @xcite ) qe da shared tasks, respectively, to get the da scores for the ape triplets. we generate the word-level qe tags for both the pairs through the qe-corpus-builder tool foot_1 . ## results and discussion this section discusses the results of different experiments. for the quantitative evaluation, we consider ter @xcite as the primary metric. we also report bleu @xcite scores. we perform a statistical significance test considering primary metric (ter) using william's significance test @xcite . the model training approach and the training details used are discussed in appendix a. ## conclusion and future work in this work, we explore multilingual ape, which has not yet been attempted to the best of our knowledge. we focus on two low-resource english-indic language pairs where the target languages, hindi and marathi, are closely related indo-aryan languages. the results of our initial experiments show the effectiveness of multilingual training for improving ape performance of both pairs. relatively higher improvements for english-hindi than english-marathi suggest that the mape can benefit the lower resource language pair in the ape setting, to improve upon their ape performance. we observe further performance gains using the proposed simple yet effective data augmentation approach in which we generate synthetic hindi-marathi and marathi-hindi ape triplets, to enable cross-lingual transfer. our investigation into the use of the qe-ape multitask learning framework in the multilingual ape setting reveals that the qe helps mape improve its capability to assess translation quality, and mitigate the problem of over-correction. the exploration of using only the domain-specific qe information to perform domain adaptation shows that modest performance gains can also be obtained, even in resource-constrained settings. with 17.06 and 16.75 ter on english-hindi and english-marathi evaluation sets, our multilin-gual ape model achieves state-of-the-art performance for the english-marathi pair, and establishes a strong baseline for english-hindi. also, our qualitative analysis hints at the possible future work-using mape for consistent post-edits across languages. the investigation performed in this work suggests that multilingual ape is a promising research direction for the advancement of ape systems for low-resource language pairs. even though the gains of more than 3 ter points highlight the robustness of mape, in the future, this work can be extended to multiple low-resource language pairs to analyze the generalizability. an in-depth study could be conducted to analyze the extent of consistency of the mape model outputs across target languages when fed with the same source and similar quality translations. further experiments could be conducted to gauge the optimal number of augmented triplets of specific additional language pairs to improve the mape performance on specific or all language pairs.