HW - TSC ’s Participation at WMT 2020 Automatic Post Editing Shared Task (2020)

Abstract: The paper presents the submission by HW-TSC in the WMT 2020 Automatic Post Editing Shared Task. We participate in the English-German and English-Chinese language pairs. Our system is built based on the Transformer pre-trained on WMT 2019 and WMT 2020 News Translation corpora, and fine-tuned on the APE corpus. Bottleneck Adapter Layers are integrated into the model to prevent over-fitting. We further collect external translations as the augmented MT candidates to improve the performance. The experiment demonstrates that pre-trained NMT models are effective when fine-tuning with the APE corpus of a limited size, and the performance can be further improved with external MT augmentation. Our system achieves competitive results on both directions in the final evaluation.

## introduction automatic post editing (ape) has been used in many scenarios where the performance of a blackbox machine translation (mt) system is unknown, or, domain specific corrections are required @xcite @xcite . the continuous improvements of nmt systems's performances along with deep learning advancements insert great challenges on developing sound ape systems, as simple translation errors are rarely seen in machine translation outputs nowadays while the remaining errors are still tough to solve. transfer learning and data augmentation techniques have demonstrated their efficiency in recent years when models are trained on datasets with limited size @xcite . therefore, such techniques are also adopted in ape tasks @xcite . participants in the ape tasks are required develop systems to automatically post edit the trans-lation outputs from an unknown mt system @xcite . in this year, the dataset has changed in terms of domain (from it to wikipedia) and quality of mt ( a significant decrease in bleu). using previous dataset or officially provided synthetic corpus (such as artificial and escape) @xcite to enlarge the training set might not be appropriate under such circumstance due to the change in data distribution. therefore, we decide to perform transfer learning with the officially released training set and integrate bottleneck adapter layers (bal) @xcite to prevent over-fitting. our model is built based on transformer @xcite and is pre-trained on the wmt 2019 and 2020 news translation corpora. compared with the work by @xcite , we consider that it is more intuitive to use a pretrained nmt model rather than a pre-trained multilingual language model (lm) @xcite . during our experiment, we find that fine-tuning the model only on the officially released corpus could easily reach the performance ceiling. as a result, we wondered whether it is possible to introduce external translations as additional mt candidates for data augmentation so as to provide more diversified features. fortunately, our experiment results demonstrate the effectiveness of such approach. the architecture of our model is shown in figure 1 . the contributions of our work are as follows: add &norm feed forward add &norm multi-head attention input embedding add &norm feed forward add &norm masked multi-head attention output embedding multi-head @xmath0, src, 
, mt, 
, mt ' 
, pe linear softmax pe, </@xmath1. august 1968 verloren die bills einen blowout-haushoch in der vorsaison gegen die houston oilers . </@xmath2, 1968 , the bills suffered a blowout preseason loss to the houston oilers . <@xmath3. august 1968 erlitt die bills einen blowout vorsaison verlust an den houston oilers . <@xmath4. august 1968 erlitten die bills einen verlust vor der saison für die houston oilers . improve feature diversity, which also significantly improves the performance. the dataset contains 7,000 sentences for the training set, 1,000 for the dev and 1,000 for the test. note that it is also used for the from this aspect, we consider the task is easier this year because over correction will not be a serious problem. however, as the corpora size is much smaller than that of the previous year, this year's ape task is challenging in another way. the evaluation metrics used this year, ter @xcite and bleu @xcite , are exactly the same as that of previous years. ## analysis except from focusing on modelling and experimenting, we also conduct an in-depth analysis of the dataset by tagging the pe operations on the dev set. based on the tags, we categorize pe operations into three categories and try to figure out which kind of pe operations can be learned by an ape system. we analysis the en-zh dev set and labeled totally 2196 pe operations, where each sentence has approximately 2.2 corrections. by categorizing these pe operations, we conclude three categories with 10 sub-categories, as described in by observing the output of our system, we find that the first and third categories are relatively difficult for the model to learn in an open domain setting, because of their complexity and uncertainty. for the second category, a pre-trained model has the prior learned from the massive bilingual text, and thereby can be easily fine-tuned to detect and make correction on these mistakes. we believe that further investigation can be performed to explore methods to improve the performance on specific patterns, which is also the research direction of our work. ## conclusion this paper presents our work in the wmt 2020 ape shared task. we adopt transfer learning and data augmentation by fine-tuning a pre-trained transformer on the provided dataset with external mts. the experimental results demonstrate the effectiveness of our method. meanwhile, we achieve competitive results on the test set in the evaluation. apart from that, we also conducted an in-depth analysis on the dev set, and group the pe operations into several fine-grained categories, serving