The Transference Architecture for Automatic Post-Editing (2020)

Abstract: In automatic post-editing (APE) it makes sense to condition post-editing (pe) decisions on both the source (src) and the machine translated text (mt) as input. This has led to multi-encoder based neural APE approaches. A research challenge now is the search for architectures that best support the capture, preparation and provision of src and mt information and its integration with pe decisions. In this paper we present an efficient multi-encoder based APE model, called transference. Unlike previous approaches, it (i) uses a transformer encoder block for src, (ii) followed by a decoder block, but without masking for self-attention on mt, which effectively acts as second encoder combining src –> mt, and (iii) feeds this representation into a final decoder block generating pe. Our model outperforms the best performing systems by 1 BLEU point on the WMT 2016, 2017, and 2018 English–German APE shared tasks (PBSMT and NMT). Furthermore, the results of our model on the WMT 2019 APE task using NMT data shows a comparable performance to the state-of-the-art system. The inference time of our model is similar to the vanilla transformer-based NMT system although our model deals with two separate encoders. We further investigate the importance of our newly introduced second encoder and find that a too small amount of layers does hurt the performance, while reducing the number of layers of the decoder does not matter much.

## introduction although machine translation (mt) systems are improving rapidly, the resulting translations may still require manual post-editing (pe) to achieve human-acceptable translation. automatic post-editing (ape) is a method that aims to automatically correct errors in machine translated text before performing actual human post-editing (pe) @xcite , thereby reducing the post-editors' workload and increasing productivity @xcite . ape systems learned from human pe data serve as downstream mt post-processing modules to improve the overall performance. ape can therefore be viewed as a 2 nd -stage mt system, translating predictable error patterns in mt output to their corresponding corrections. ape training data minimally involves mt output (mt) and the human post-edited (pe) version of mt, but additionally using the source (src) has been shown to provide further benefits @xcite @xcite . to provide awareness of errors in mt originating from src, attention mechanisms @xcite allow modeling of non-local dependencies in the input or output sequences, and importantly also global dependencies between them (in our case src, mt and pe). the transformer architecture @xcite is built solely upon such attention mechanisms completely replacing recurrence and convolutions. the transformer uses positional encoding to encode the input and output sequences, and computes both self-and cross-attention through so-called multi-head attentions, which can be easily parallelized. multi-head attention allows to jointly attend to information at different positions from different representation subspaces, e.g. utilizing and combining information from src, mt, and pe. in this paper, we present a multi-encoder based neural ape architecture called transference. our model contains a source encoder which encodes src information, a second encoder (enc @xmath0, combines this with the self-attentionbased encoding of mt (enc mt ), and prepares a representation for the decoder (dec pe ) via cross-attention. our second encoder (enc @xmath1, however, without masking, which acts as an encoder. we thus recombine the different blocks of the transformer architecture and repurpose them for the ape task in a simple yet effective way. the suggested architecture is inspired by the two-step approach professional translators tend to use during post-editing: first, the source segment is compared to the corresponding translation suggestion (similar to what our enc @xmath2, then corrections to the mt output are applied based on the encountered errors (in the same way that our dec pe uses the encoded representation of enc @xmath3. the paper makes the following contributions: (i) we propose a multi-encoder model for ape that consists only of standard transformer encoding and decoding blocks, (ii) by using a mix of self-and cross-attention we provide a representation of both src and mt for the decoder, allowing it to better capture errors in mt originating from src; this advances junczys-dowmunt and grundkiewicz (2018) -the wmt 2018 best system (wmt18 smt best ) in terms of bleu and ter, (iii), we analyze the effect of varying the number of encoder and decoder layers @xcite , indicating that the encoders contribute more than decoders in neural ape, and (iv) we present and evaluate an ape architecture inspired by a two-step approach professional translators often use during post-editing. in comparison to the shared task system description paper @xcite , this paper (i) provides more detailed explanations and reformation of different components of the transference architecture, (ii) compares it to a single encoder based transformer architecture where only mt or src concatenated with mt are used as an input, (iii) analyzes results when swapping mt and src in the multi-encoder setup, and (iv) investigates the importance of encoder and decoder by varying the amount of layers. the rest of the paper is organized as follows. in §2, we survey existing literature on ape. in §3, we describe the multi-encoder architecture. §4 describes our experimental setup. §5 reports the results of our approach against a number of baselines. finally, §6 concludes the paper with future directions. ## related research recent advances in ape research are directed towards neural ape, which was first proposed by @xcite and @xcite for the single-source ape scenario which does not consider src, i.e. @xmath4. @xcite also generated a large synthetic training dataset, which we also use as additional training data. exploiting source information as an additional input can help neural ape to disambiguate corrections applied at each time step; this naturally leads to multi-source ape ({src, mt} → pe). a multi-source neural ape system can be configured either by using a single encoder that encodes the concatenation of src and mt @xcite or by using two separate encoders for src and mt and passing the concatenation of both encoders' final states to the decoder @xcite . a few approaches to multi-source neural ape were proposed in the wmt 2017 ape shared task. junczys-dowmunt and grundkiewicz (2017) combine both mt and src in a single neural architecture, exploring different combinations of attention mechanisms including soft attention and hard monotonic attention. @xcite built upon the two-encoder architecture of multi-source models @xcite by means of concatenating both weighted contexts of encoded src and mt. varis and bojar (2017 ) compared two multi-source models, one using a single encoder with the concatenation of src and mt sentences, and a second one using two character-level encoders for mt and src along with a character-level decoder. in the wmt 2018 ape shared task, several adaptations of the transformer architecture were presented for multi-source ape. @xcite introduced a joint encoder that attends over a combination of the two encoded sequences from mt and src. @xcite , the nmt-subtask winner of wmt 2018 (wmt18 nmt best ), employed sequence-level loss functions in order to avoid exposure bias during training and to be consistent with the automatic evaluation metrics. @xcite proposed a multi-source transformer where on the decoder side, they added two additional multi-head attention layers for @xmath5. thereafter another multi-head attention between the output of those attention layers helps the decoder to capture common words in mt which should remain in pe. the ape pbsmt-subtask winner of wmt 2018 (wmt18 smt best ) @xcite ) also presented another transformer-based multi-source ape which uses two encoders and stacks an additional cross-attention component for @xmath6. in contrast to other multi-encoder based approaches and @xcite 's approach, where the authors focused on cross-attention of two encoders with respect to the decoder within the transformer architecture, we propose a novel architecture where the second encoder block is similar to the transformer decoder block but without masking. in the latest edition of @xcite , the submissions are mostly multi-source models extending the transformer implementation @xcite @xcite and adapting bert @xcite to the transformer-based framework @xcite . the winner system ( lopes et al., 2019) (wmt19 nmt best ) uses a single pre-trained bert encoder that receives both the source src and mt strings and applies a bert-based encoder-decoder model. additionally, they add a conservativeness penalty factor during beam decoding to avoid over-corrections in ape. our method outperforms the wmt 2016, 2017, and 2018 winners by 1 bleu point, and yields comparable performance to the wmt 2019 winner, however, without using a bert-based architecture. ## the transference multi-encoder transformer for ape we propose a multi-source transformer model called transference ({src, mt} @xmath7, figure 1 ), which takes advantage of both the encodings of src and mt and attends over a combination of both sequences while generating the post-edited sentence. the second encoder, enc @xmath8, makes use of the first encoder enc src and a sub-encoder enc mt for considering src and mt. here, the enc src encoder and the dec pe decoder are equivalent to the original transformer for neural mt @xcite . our enc @xmath9, the difference being that multihead selfattention is not masked to process mt. the self-attended encoder for src, @xmath10, s 2 , . . . , s k ), returns a sequence of continuous representations, enc src , and the second self-attended sub-encoder for mt, @xmath11, m 2 , . . . , m l ), returns another sequence of continuous representations, enc mt . self-attention at this point provides the advantage of aggregating information from all of the words, including src and mt, and successively generates a new representation per word informed by the entire src and mt context. to do this the internal enc mt representation performs cross-attention over enc src and prepares a final representation (enc @xmath12. the decoder then generates the pe output in sequence, @xmath13, p 2 , . . . , p n ), one word at a time from left to right by attending to previously generated words as well as the final representations (enc @xmath14. to summarize, our multi-source ape implementation extends @xcite by introducing an additional encoding block by which src and mt communicate with the decoder. our proposed approach differs from the wmt 2018 pbsmt winner system (wmt18 smt best ) in several ways: (i) we use the original transformer's decoder without modifications; (ii) one of our encoder blocks (enc @xmath15, thus having one self-attention layer and an additional cross-attention for @xmath16; and (iii) in the decoder layer, the cross-attention is performed between the encoded representation from enc @xmath17. moreover, placing a cross-attention network within the enc @xmath18, during inference, enc @xmath19.e., once per decoding step. our approach also differs from the wmt 2018 nmt winner system: (i) wmt18 nmt best concatenates the encoded representation of two encoders and passes it as the key to the attention layer of the decoder, and (ii), the system additionally employs sequence-level loss functions based on maximum likelihood estimation and minimum risk training in order to avoid exposure bias during training. comparing with wmt19 nmt best , the winner system of wmt 2019 uses a pre-trained deep bidirectional transformer (multilingual bert) @xcite , while our model does not. the main intuition is that our enc @xmath20, process, and share information between src-mt-pe, which efficiently models error patterns and the corresponding corrections. our model performs better than past transformer-based approaches and similar to the bert-based approach (wmt19 nmt best ) without adding the overhead of the pre-trained model, as the experiment section will show. ## experiments we explore our approach on both ape sub-tasks of wmt 2018, where the black box mt (we refer as 1 st -stage mt) system to which ape is applied is either a phrase-based statistical machine translation (pbsmt) or a neural machine translation (nmt) model. for the pbsmt task, we compare against four baselines: the raw smt output provided by the 1 ststage pbsmt, the best-performing systems from wmt ape 2018 (wmt18 smt best ), which are a single model and an ensemble model by @xcite , as well as a transformer directly translating from src to pe (transformer (@xmath21, thus performing translation instead of ape. we evaluate the systems using bleu @xcite and ter @xcite . for the nmt task, we consider three baselines: the raw nmt output provided by the 1 st -stage nmt system, the best-performing system from the wmt 2018 (wmt18 nmt best ) @xcite and wmt 2019 (wmt19 nmt best ) (lopes et al., 2019) nmt ape task. apart from the multi-encoder transference architecture described above ({src, mt} @xmath22, two simpler versions are also analyzed: first, a 'mono-lingual' (@xmath23, and second, an identical single-encoder architecture, however, using the concatenated src and mt text as input ({src + mt} → pe) @xcite . ## results the results of our four models, single-source (@xmath24, multi-source single encoder ({src + pe} → pe), transference model ({src, mt} smt @xmath25, {mt, src} smt @xmath26, and ensemble, in comparison to the four baselines, raw smt, wmt18 smt best @xcite single and ensemble, as well as transformer (@xmath27, are presented in ## conclusions in this paper, we presented a multi-encoder transformer-based ape model that repurposes the standard transformer blocks in a simple and effective way for the ape task: first, our transference architecture uses a transformer encoder block for src, followed by a decoder block without masking on mt that effectively acts as a second encoder combining @xmath29, and feeds this representation into a final decoder block generating pe. the proposed model outperforms the best-performing system of wmt 2018 on the test2016, test2017, dev2018, and test2018 data. moreover, our model is on par with but simpler than wmt 2019 best system since our model does not apply bert or any conservative factor during inference. taking a departure from traditional transformer-based encoders, which perform self-attention only, our second encoder also performs cross-attention to produce representations for the decoder based on both src and mt. we also show that the encoder plays a more pivotal role than the decoder in transformerbased ape, which could also be the case for transformer-based generation tasks in general. our architecture is generic and can be used for any multi-source task, e.g., (i) multi-source translation (ii) document translation to model the associated context during translation, (iii) question generation to generate question from given passage and a short answer text, (iv) question answering task from given passage and question text, (v) summarization, etc.