EDITOR : An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints (2021)

Abstract: We introduce an Edit-Based TransfOrmer with Repositioning (EDITOR), which makes sequence generation flexible by seamlessly allowing users to specify preferences in output lexical choice. Building on recent models for non-autoregressive sequence generation (Gu et al., 2019), EDITOR generates new sequences by iteratively editing hypotheses. It relies on a novel reposition operation designed to disentangle lexical choice from word positioning decisions, while enabling efficient oracles for imitation learning and parallel edits at decoding time. Empirically, EDITOR uses soft lexical constraints more effectively than the Levenshtein Transformer (Gu et al., 2019) while speeding up decoding dramatically compared to constrained beam search (Post and Vilar, 2018). EDITOR also achieves comparable or better translation quality with faster decoding speed than the Levenshtein Transformer on standard Romanian-English, English-German, and English-Japanese machine translation tasks.

## introduction neural machine translation (mt) architectures @xcite make it difficult for users to specify preferences that could be incorporated more easily in statistical mt models @xcite and have been shown to be useful for interactive machine translation @xcite and domain adaptation @xcite . lexical constraints or preferences have previously been incorporated by re-training nmt models with constraints as inputs @xcite or with constrained beam search that drastically slows down decoding @xcite . in this work, we introduce a translation model that can seamlessly incorporate users' lexical choice preferences without increasing the time and computational cost at decoding time, while being trained on regular mt samples. we apply this model to mt tasks with soft lexical constraints. as illustrated in figure 1 , when decoding with soft lexical constraints, user preferences for lexical choice in the output language are provided as an additional input sequence of target words in any order. the goal is to let users encode terminology, domain, or stylistic preferences in target word usage, without strictly enforcing hard constraints that might hamper nmt's ability to generate fluent outputs. our model is an edit-based transformer with repositioning (editor), which builds on recent progress on non-autoregressive sequence generation @xcite . 1 specifically, the levenshtein transformer @xcite showed that iteratively refining output sequences via insertions and deletions yields a fast and flexible generation process for mt and automatic post-editing tasks. editor replaces the deletion operation with a novel reposition operation to disentangle lexical choice from reordering decisions. as a result, editor exploits lexical constraints more effectively and efficiently than the levenshtein transformer, as a single reposition operation can subsume a sequence of deletions and insertions. to train editor via imitation learning, the reposition operation is defined to preserve the ability to use the levenshtein edit distance @xcite as an efficient oracle. we also introduce a dualpath roll-in policy, which lets the reposition and deletion models learn to refine their respective outputs more effectively. experiments on romanian-english, english-german, and english-japanese mt show that editor achieves comparable or better translation quality with faster decoding speed than figure 1 : romanian to english mt example. unconstrained mt incorrectly translates ''gleznȃ'' to ''bullying''. given constraint words ''plague'' and ''ankle'', soft-constrained mt correctly uses ''ankle'' and avoids disfluencies introduced by using ''plague'' as a hard constraint in its exact form. the levenshtein transformer @xcite on the standard mt tasks and exploits soft lexical constraints better: it achieves significantly better translation quality and matches more constraints with faster decoding speed than the levenshtein transformer. it also drastically speeds up decoding compared with lexically constrained decoding algorithms @xcite . furthermore, results highlight the benefits of soft constraints over hard ones-editor with soft constraints achieves translation quality on par or better than both editor and levenshtein transformer with hard constraints @xcite . reordering vs. lexical choice editor's insertion and reposition operations connect closely with the long-standing view of mt as a combination of a translation or lexical choice model, which selects appropriate translations for source units given their context, and reordering model,which encourages the generation of a target sequence order appropriate for the target language. this view is reflected in architectures ranging from the word-based ibm models @xcite , sentence-level models that generate a bag of target words that is reordered to construct a target sentence @xcite , or the operation sequence model @xcite , which views translation as a sequence of translation and reordering operations over bilingual minimal units. by contrast, autoregressive nmt models @xcite do not explicitly separate lexical choice and reordering, and previous non-autoregressive models break up reordering into sequences of other operations. this work introduces the reposition operation, which makes it possible to move words around during the refinement process, as reordering models do. however, we will see that reposition differs from typical reordering to enable efficient oracles for training via imitation learning, and parallelization of edit operations at decoding time (section 3). mt with soft lexical constraints nmt models lack flexible mechanisms to incorporate users preferences in their outputs. lexical constraints have been incorporated in prior work via 1) constrained training where nmt models are trained on parallel samples augmented with constraint target phrases in both the source and target sequences @xcite , or 2) constrained decoding where beam search is modified to include constraint words or phrases in the output @xcite . these mechanisms can incorporate domain-specific knowledge and lexicons which is particularly helpful in low-resource cases @xcite . despite their success at domain adaptation for mt @xcite and caption generation @xcite , they suffer from several issues: constrained training requires building dedicated models for constrained language generation, while constrained decoding adds significant computational overhead and treats all constraints as hard constraints which may hurt fluency. in other tasks, various constraint types have been introduced by designing complex architectures tailored to specific content or style constraints @xcite , or via segment-level ''side-constraints'' @xcite @xcite , which condition generation on users' stylistic preferences, but do not offer fine-grained control over their realization in the output sequence. we refer the reader to yvon and abdul rauf, (2020) for a comprehensive review of the strengths and weaknesses of current techniques to incorporate terminology constraints in nmt. our work is closely related to susanto et al. (2020)'s idea of applying the levenshtein transformer to mt with hard terminology constraints. we will see that their technique can directly be used by editor as well (section 3.3), but this does not offer empirical benefits over the default editor model (section 4.3). ## experiments we evaluate the editor model on standard (section 4.2) and lexically constrained machine translation (sections 4.3-4.4). ## conclusion we introduce editor, a non-autoregressive transformer model that iteratively edits hypotheses using a novel reposition operation. reposition combined with a new dual-path imitation learning strategy helps editor generate output sequences that flexibly incorporate user's lexical choice preferences. extensive experiments show that editor exploits soft lexical constraints more effectively than the levenshtein transformer @xcite while speeding up decoding dramatically compared to constrained beam search @xcite . results also confirm the benefits of using soft constraints over hard ones in terms of translation quality. editor also achieves comparable or better translation quality with faster decoding speed than the levenshtein transformer on three standard mt tasks. these promising results open several avenues for future work, including using editor for other generation tasks than mt and investigating its ability to incorporate more diverse constraint types into the decoding process. richburg, franc ¸ois yvon, the tacl reviewers, and the clip lab at umd for their helpful and constructive comments. this research is supported in part by an amazon web services machine learning research award and by the office of the director of national intelligence (odni), intelligence advanced research projects activity (iarpa), via contract #fa8650-17-c-9117. the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of odni, iarpa, or the u.s. government. the u.s. government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.