Building The First E nglish- B razilian P ortuguese Corpus for Automatic Post-Editing (2020)

Abstract: This paper introduces the first corpus for Automatic Post-Editing of English and a low-resource language, Brazilian Portuguese. The source English texts were extracted from the WebNLG corpus and automatically translated into Portuguese using a state-of-the-art industrial neural machine translator. Post-edits were then obtained in an experiment with native speakers of Brazilian Portuguese. To assess the quality of the corpus, we performed error analysis and computed complexity indicators measuring how difficult the APE task would be. We report preliminary results of Phrase-Based and Neural Machine Translation Models on this new corpus. Data and code publicly available in our repository.

## introduction automatic post-editing (ape) is the computational task responsible for fixing systematic and repetitive errors found in black-box machine translation (mt) outputs @xcite . ape is considered appealing for allowing the rapid and cheap customization of general-purpose machine translation models to specific application domains, avoiding the need for new systems to be trained from scratch @xcite . moreover, these models deliver better machine-translated outputs to human translators, reducing human post-editing effort. ape systems are developed based on triples aligning source sentences, machine translation outputs and human post-edits. data is gathered through interfaces where human translators can post-edit and improve the quality of machine-translated documents @xcite . while human post-edits are necessary for training ape systems, there are not many publicly-available resources of this kind. for languages with pre-existing ape corpora, the scarcity of data has been solved by generating artificial triples @xcite . however, even small ape corpora have not been publicly-available for most languages. for example, the wmt shared task on mt automatic post-editing (ape), main source of studies in ape since 2015, provided an english-spanish corpus in its 2015 version, an english-german ape corpus in its 2016-2018 versions (also a german-english in the 2017 one), and a novel english-russian one in its 2019 version. in sum, only three languages have been explored in the main shared-task on ape. in order to fulfill the gap of data scarcity in different languages, we have compiled a novel corpus for automatic post-editing from english into brazilian portuguese, a low-resource language. brazilian portuguese has fewer resources available compared to european portuguese, an official language of the european union and, as such, featuring in several parallel corpora. the source english texts were extracted from the webnlg corpus @xcite , which also provides a meaning representation aligned to each text. the extracted texts were automatically translated into portuguese using a stateof-the-art neural machine translation service. native speakers of brazilian portuguese post-edited the machine-translated output through a web interface specifically designed for our experiment. to assure the quality of the corpus, we submitted the machine and post-edited output to human evaluation, producing a quality label for each sentence. this paper describes our corpus compilation and reports the first results of phrase-based and neural models to automatically post-edit it. the relevance of the corpus and the results produced by our baseline are discussed. further steps in our study are presented in the concluding section of the paper. subject predicate object alfred giles (architect) architect asher and mary isabelle richardson house kendall county, texas placeofdeath alfred giles (architect) england birthplace alfred giles (architect) ↓ sentence 1 the architect of asher and mary isabelle richardson house alfred giles was born in england and died in kendall county texas. sentence 2 alfred giles was born in england and died in kendall county, texas. he designed the asher and mary isabelle richardson house . sentence 3 the architect alfred giles was born in england and he died in kendall county, texas. he was the architect of asher and mary isabelle richardson house. the webnlg corpus has in total 42,901 english verbalizations for 16,095 distinct meaning representations. from this amount, we extracted a sample of 4,148 pairs, corresponding to the test partition of the original corpus. machine translated outputs once the english texts were extracted from the webnlg corpus, we translated them into portuguese foot_1 using deepl translator, an increasingly popular neural mt application in industry. deepl's superior performance over similar mt tools accounts for its selection, even though it would allow us to obtain translations into european portuguese, and not brazilian portuguese, our intended target language for the experiment. human post-edits to obtain the human post-edits, we designed a web interface for this study. 3 for each instance, participants are presented with the original text, a label for its domain category and the machine translation output. they may either edit the machine translation from a text-box (so-called free mode), or on a guided mode, where a set of operations may be used to post-edit the translation. the operations were defined based on neural programmer-interpreter approaches for ape @xcite and consisted of insertion to the right, insertion to the left, delete, and update. prior to post-editing, participants received instructions requesting them (1) to transliterate entity names whenever transliteration was available, (2) not to pause the post-editing session while consulting external sources, and (3) to adapt the machine output to brazilian portuguese whenever necessary. in total, we recruited a group of 37 participants to post-edit the machine translated sentences. portuguese was their l1 and english their l2. 33 of them reported an upper-intermediate proficiency level of english while 4 reported an intermediate one. in order to prevent human errors and human biases, each machine translation output was post-edited by two independent participants. ## automatic post-editing experiment we trained phrase-based and neural models to automatically post-edit the machine translations from the corpus. we explored the impact of using only high quality translations on each model. thus, for each model type, we produced a model using all translations and another one using only high quality ones according to the human evaluation. data from the 8,296 triples in our corpus, we filtered out the ones where the machine translation output had been edited likewise by the two human post-editors, leaving 7,247 unique triples. we then selected texts evaluated as medium, good, and very good (6,646 triples). this high quality triple set was split into training, test, and development on the proportion 0.6, 0.2 and 0.2, resulting in 3,987, 1,330, and 1,329 triples, respectively. the training set was labeled good training. low quality sentences initially filtered out were added to the good training set to create the all training set. hence, the good training, test and development sets contain high quality post-edits only, while the all training set contains all post-edits. ## discussion this study introduces the first ape corpus for english and brazilian portuguese, the latter being a lowresource language. similar to @xcite , our corpus is based on the webnlg dataset @xcite , originally proposed for the task of natural language generation and which consists of instance pairs of meaning representations and their english verbalizations. in this arrangement, the use of meaning representations along with the original text and the machine translation in the task of ape can be investigated on our corpus. our analysis showed the most frequent post-edits performed by human translators in our corpus, such as inserting punctuation, fixing gender agreement and adapting european portuguese to brazilian portuguese. complexity indicators showed high repetition rate in the corpus; machine translations showed good quality as evidenced by mt metrics and also by the fact that 55.3% of them (2292 out of 4148) were not edited by at least one of the post-editors. high repetitiveness indicates higher chance of learning from the training set correction patterns @xcite @xcite , whereas good quality of machine translations represents a challenge on training an ape model to outperform the machine translation tool in our corpus. preliminary results for our corpus with phrased-based and neural ape models show pb models outperform neural models, which may suggest that neural models cannot generalize well with small data. this points to the importance of the training data size, supporting the generation of artificial data @xcite . we can also conclude that large data is better than filtered data, especially for neural models, as the metrics in a further step in our study is the use of multi-source ape systems to reach better results @xcite . specifically for this corpus, not only source sentences, but also rdf triples can be included in the encoding phase along with machine translations. in addition, using artificial data could improve ape as @xcite suggested, which would also fix the problem of having small data. once our ape reaches a reasonable level of improvement, it can be used to finalize the translation of the entire webnlg corpus into brazilian portuguese.