IIT B ombay’s WMT 22 Automatic Post-Editing Shared Task Submission (2022)

Abstract: This paper describes IIT Bombay’s submission to the WMT22 Automatic Post-Editing (APE) shared task for the English-Marathi (En-Mr) language pair. We follow the curriculum training strategy to train our APE system. First, we train an encoder-decoder model to perform translation from English to Marathi. Next, we add another encoder to the model and train the resulting dual-encoder single-decoder model for the APE task. This involves training the model using the synthetic APE data in multiple training stages and then fine-tuning it using the real APE data. We use the LaBSE technique to ensure the quality of the synthetic APE data. For data augmentation, along with using candidates obtained from an external machine translation (MT) system, we also use the phrase-level APE triplets generated using phrase table injection. As APE systems are prone to the problem of ‘over-correction’, we use a sentence-level quality estimation (QE) system to select the final output between an original translation and the corresponding output generated by the APE model. Our approach improves the TER and BLEU scores on the development set by -3.92 and +4.36 points, respectively. Also, the final results on the test set show that our APE system outperforms the baseline system by -3.49 TER points and +5.37 BLEU points.

## introduction automatic post-editing (ape) is a post-processing task in a machine translation (mt) workflow. it aims to automatically identify and correct errors in mt outputs @xcite . @xcite and @xcite show that ape systems have the potential to reduce human effort by automatically correcting repetitive translation errors. the initial years of the wmt ape shared task focused on correcting errors in statistical machine translation (smt) translations, where participants explored various statistical and neural ape approaches @xcite . although neural ape approaches showed high potential for significantly improving the quality of smt translations, these approaches faced challenges in improving translations obtained from relatively-more-robust neural machine translation (nmt) systems @xcite . a possible reason for this could be that correcting a high-quality translation requires fewer edits, and therefore ape approaches need to be precise in identifying and in correcting the errors. also, the neural ape approaches use large neural networks that require significant training data. ape training data consists of 'triplets' in the form of source sentence (src), its translation generated using an mt system (mt), and a human postedited version of the translation (pe). obtaining pe is an expensive task in terms of time and money; therefore, there is a lack of large ape datasets. to deal with this problem, various data augmentation techniques have been proposed @xcite @xcite . @xcite used imitation learning to filter the ape data for tackling the distributional difference between real and synthetic ape data. @xcite augmented the ape training data with translations generated using a different mt system. inspiring from the work of @xcite , we augment the ape data by generating phrase-level ape triplets using smt phrase tables. to ensure the quality of the synthetic data, we use the labse technique @xcite and filter low-quality triplets. another effective approach for dealing with the problem of data sparsity is transfer learning in which pre-trained models are adapted to the ape task @xcite ). an ape system needs to understand both the source and target languages to obtain joint encoding of src and mt. therefore, @xcite uses a cross-lingual language model instead of a monolingual one. unlike these approaches, @xcite al. (2020) although recent ape systems use a single encoder to encode both the source sentence and its translation @xcite , we use separate encoders for encoding src and mt as english and marathi do not share much vocabulary; and belong to different language families. we use indicbert @xcite to initialize weights of our the src encoder and mt encoder. we train and fine-tune our models using the cts over the good-quality ape data. the training data is also augmented with external mt candidates and phrase-level ape triplets. it is known that ape systems are prone to making unnecessary edits to translation output @xcite . to mitigate this issue of over-correction, we use a sentence-level qe system to select the final output. when evaluated on the development set, our approach improves the ter @xcite ) by -3.92 points and the bleu @xcite by +4.98 points. similarly, the final results on the test set show that our ape system outperforms the baseline system by -3.49 ter points and +5.37 bleu points. we summarize the main features of our approach as follows: ## results in we compare the results of our experiments against a 'do nothing' ape baseline that simply outputs mt without any modification. when we trained our model using cts and external mt candidates to increase feature diversity, the ter and bleu scores improved to 20.08 ter points and 67.39 bleu points from the baseline ter and bleu scores of 22.93 and 64.51, respectively. the third row in the 2 shows the results of an experiment where we use a good-quality synthetic dataset for ape training obtained by filtering lowquality triplets using labse-based filtering. the experiment also involves training the ape model in two phases: first, the model is trained on outof-domain synthetic data and then on in-domain synthetic data. this setting brings -3.2 and +3.35 ter and bleu score improvements over the baseline, and underlines the importance of using goodquality in-domain ape data. the only change we make for performing the next experiment is augmenting the synthetic and real ape data using phrase-level ape triplets. results of this experiment show that performance improves over the baseline by -3.54 ter points and +3.84 bleu points. towards the end, we also used a sentence-qe system to rate the original translation and the ape output. we then select one of them with a higher rating as the final output of our ape system. with the combination of the ape model and sentence-qe system, we see that the ter score improves to 19.01 points, and bleu score increases to 68.87 points; which shows that using the sentence-level qe system is an effective approach to discard ape output, in cases of overcorrection. as per the information received by the shared task organizers, our ape system achieves a ter score of 16.79 points and a bleu score of 72.92 when evaluated on the official test set, which is -3.49 ter points and +5.37 bleu points improvement over the baseline. ## conclusions and future work this paper presents our ape system submitted to the wmt-22 ape english-marathi shared task. we use a dual-encoder single-decoder model where both encoders are initialized using indicbert. we propose a new way to generate artificial phraselevel ape triplets by extending the phrase-pair injection method used in mt for ape. we show that augmenting ape training data with these phraselevel triplets and training the model with the cts on good-quality in-domain ape data improves the performance of the ape system. furthermore, we also explore using the sentence-level qe system to discard low-quality ape outputs. evaluation of our ape system shows that our approach achieves significant gains on the wmt-22 ape development and test sets. in future, we would like to extend this approach for automatic post-editing with the help of wordlevel quality estimation and come up with a single architecture for performing both the qe tasks along with ape. we would also like to attempt a multilingual ape system with a shared decoder across multiple languages.