Bring More Attention to Syntactic Symmetry for Automatic Postediting of High-Quality Machine Translations (2023)

Abstract: Automatic postediting (APE) is an automated process to refine a given machine translation (MT). Recent findings present that existing APE systems are not good at handling high-quality MTs even for a language pair with abundant data resources, English–German: the better the given MT is, the harder it is to decide what parts to edit and how to fix these errors. One possible solution to this problem is to instill deeper knowledge about the target language into the model. Thus, we propose a linguistically motivated method of regularization that is expected to enhance APE models’ understanding of the target language: a loss function that encourages symmetric self-attention on the given MT. Our analysis of experimental results demonstrates that the proposed method helps improving the state-of-the-art architecture’s APE quality for high-quality MTs.

## introduction automatic postediting (ape) is an automated process to transform a given machine translation (mt) into a higher-quality text @xcite . since 2015, conference on machine translation (wmt) has been hosting an annual shared task for ape, and most of the recently developed ape systems are within the common framework of representation learning using artificial neural networks to learn postediting patterns from the training data @xcite @xcite . since 2018, all participants in the shared task have used transformer-based models @xcite , but recent findings of the shared task @xcite @xcite cast doubt on whether transformer-based ape models learn good generalizations because such models' ape quality appears to be significantly affected by external factors such as the source-target language pair, the qualitative characteristics of the provided data, and the quality of the given mt. especially, the good quality of the given mts has brought great difficulty in performing ape on the wmt 2019 test data set: the better the given mt is, the harder it is to decide what parts to edit and how to correct these errors @xcite . the thing to notice is that this outcome is not a question of data scarcity because the language pair of this test data set, english-german, is a language pair provided with abundant training, validation, and test data. also, it is not a question of data heterogeneity, either: the domain of this test data set, it, shows a high degree of lexical repetition, which indicates that data sets in this domain use the same small set of lexical items @xcite @xcite . thus, it would be a question of modeling, and one possible solution is to implant deeper knowledge about the target language into the model. to this end, we propose a new method of regularization that is expected to enhance transformerbased ape models' understanding of german translations. specifically, the proposed method is based on feldermodell ( §2), an established linguistic model, which implies the need for proper treatment of the underlying symmetry of german sentence structures. to instill the idea of syntactic symmetry into transformer-based ape models, we introduce a loss function that encourages symmetric self-attention on the given mt. based on experimental results, we conduct a careful analysis and conclude that the proposed method has a positive effect on improving the state-of-the-art architecture's ape quality for high-quality mts. ## linguistic theory in german linguistics, das topologische satzmodell ('the topological sentence model') or das feldermodell ('the field model') @xcite @xcite describes how constituents of a sentence are closely related even if they are far apart from each other. usually, feldermodell divides a clause into das vorfeld ('the prefield'; vf), die linke satzklammer ('the left bracket'; lsk), das mittelfeld ('the middlefield'; mf), die rechte satzklammer ('the right bracket'; rsk), and das nachfeld ('the postfield'; nf). (1) [ heute vf ] [ habe lsk ] [ ich mf ] [ gesehen rsk ] [ zufällig nf ], (2) [ [ dass lsk ] [ du eine tasse kaffee mf ] [ getrunken hast rsk ] nf ]. these parts are all interrelated; lsk and rsk are a typical example: while the former holds a finite verb or a complementizer, the latter holds a past participle, an infinitive, and a particle. in (1), vf holds "heute" ('today'); lsk holds "habe" ('have'); mf holds "ich" ('i'); rsk holds "gesehen" ('seen'); and nf holds "zufällig" ('by chance'). ( 2 ) is an additional nf of (1) and includes its own lsk holding "dass" ('that'); mf holding "du eine tasse kaffee" ('you a cup of coffee'); and rsk holding "getrunken hast" ('drank'). for such analyses, special tree structures such as doppelbaum @xcite ('double tree') can be used, which is a bimodal tree (fig. 1 ), where two cp, c, ip, i, and vp subtrees are 'symmetric' with respect to v. we assume that this structural symmetry is parameterized from the perspective, not only of generative linguistics @xcite , but also of a parametric model @xmath0, where p θ and θ are a probability distribution and the parameter space, respectively. especially, if we look at ape in terms of sequence-to-sequence learning @xcite , the probability distribution of the output sequence (y 1 , ⋯, y ly ) is obtained in the following manner: ## methodology we implement a multi-encoder transformer model consisting of the "joint-final" encoder and the "parallel" decoder, which is a state-of-the-art architecture for ape @xcite , and conduct a controlled experiment without concern for usage of performance-centered tuning techniques. specifically, the joint-final encoder consists of a sourcetext encoder and an mt encoder, which process the given source text and mt, respectively. based on this baseline architecture, we propose a method to encourage the mt encoder to perform symmetric self-attention by minimizing the skewness of each self-attention layer's categorical distribution p self . for each token z i in the given mt (z 1 , ⋯, z lz ). ## experiment in the conducted experiment, all hyperparameters are the same as those of @xcite except the learning rate (appendix a); we basically reproduce their experimental design. data sets sizes training escape-nmt 5,065,187 wmt 2019 13,442 validation wmt 2019 1,000 test wmt 2019 1,023 both the baseline model and the proposed model are trained by using the training data sets and the validation data set listed in ## results and analysis the result of automatic evaluation ( thus, we present an additional evaluation result to further investigate this mismatch between ter improvements and bleu improvements: a relative frequency distribution of successes and failures in ape with regard to the ter difference systems wmt 2019 ter ↓ (σ) bleu ↑ (σ) given mt 16.84 (19.52) 74.73 (25.89) baseline 16.60 † (19.51) 75.11 † (26.21) doppelbaum 16.54 † (19.48) 75.47 †* (26.16) another example ( however, considering the big differences in the proportion of negl (2.35pp), which is the set of the cases where an ape system neglects to postedit the given mt, and the f1 score ( ## conclusion to improve the ape quality for high-quality mts, we propose a linguistically motivated method of regularization that enhances transformer-based ape models' understanding of the target language: a loss function that encourages ape models to perform symmetric self-attention on a given mt. experimental results suggest that the proposed method helps improving the state-of-the-art architecture's ape quality for high-quality mts; we also present a relative frequency distribution of successes and failures in ape and see increases in the proportion of perfect postediting and the f1 score. this evaluation method could be useful for assessing the ape quality for high-quality mts in general. actual cases support that the proposed method successfully instills the idea of syntactic symmetry into ape models. future research should consider different language pairs and different sets of hyperparameters.