Accurate Online Posterior Alignments for Principled Lexically-Constrained Decoding (2022)

Abstract: Online alignment in machine translation refers to the task of aligning a target word to a source word when the target sequence has only been partially decoded. Good online alignments facilitate important applications such as lexically constrained translation where user-defined dictionaries are used to inject lexical constraints into the translation model. We propose a novel posterior alignment technique that is truly online in its execution and superior in terms of alignment error rates compared to existing methods. Our proposed inference technique jointly considers alignment and token probabilities in a principled manner and can be seamlessly integrated within existing constrained beam-search decoding algorithms. On five language pairs, including two distant language pairs, we achieve consistent drop in alignment error rates. When deployed on seven lexically constrained translation tasks, we achieve significant improvements in BLEU specifically around the constrained positions.

## introduction online alignment seeks to align a target word to a source word at the decoding step when the word is output in an auto-regressive neural translation model @xcite @xcite . this is unlike the more popular offline alignment task that uses the entire target sentence @xcite . state of the art methods of offline alignment based on matching of whole source and target sentences @xcite are not applicable for online alignment where we need to commit on the alignment of a target word based on only the generated prefix thus far. an important application of online alignment is lexically constrained translation which allows injection of domain-specific terminology and other phrasal constraints during decoding @xcite @xcite . other applications include preservation of markups between the source and target @xcite , and supporting source word edits in summarization @xcite . these applications need to infer the specific source token which aligns with output token. thus, alignment and translation is to be done simultaneously. existing online alignment methods can be categorized into prior and posterior alignment methods. prior alignment methods @xcite extract alignment based on the attention at time step t when outputting token y t . the attention probabilities at time-step t are conditioned on tokens output before time t. thus, the alignment is estimated prior to observing y t . naturally, the quality of alignment can be improved if we condition on the target token y t @xcite . this motivated @xcite to propose a posterior alignment method where alignment is calculated from the attention probabilities at the next decoder step t + 1. while alignment quality improved as a result, their method is not truly online since it does not generate alignment synchronously with the token. the delay of one step makes it difficult and cumbersome to incorporate terminology constraints during beam decoding. we propose a truly online posterior alignment method that provides higher alignment accuracy than existing online methods, while also being synchronous. because of that we can easily integrate posterior alignment to improve lexicon-constrained translation in state of the art constrained beamsearch algorithms such as vdba @xcite . our method (align-vdba) presents a significant departure from existing papers on alignmentguided constrained translation @xcite ) that employ a greedy algorithm with poor constraint satisfaction rate (csr). for example, on a @xmath0. moreover, the latter does not benefit from larger beam sizes unlike vdba-based methods that significantly improve with larger beam widths. compared to @xcite , our method improves average overall bleu scores by 1.2 points and average bleu scores around the constrained span by up to 9 points. in the evaluations performed in these earlier work, vdba was not allocated the slightly higher beam size needed to pro-actively enforce constraints without compromising bleu. compared to @xcite (vdba), this paper's contributions include online alignments and their use in more fluent constraint placement and efficient allocation of beams. ## posterior online alignment given a sentence @xmath1, . . . , x s in the source language and a sentence @xmath2, . . . , y t in the target language, an alignment a between the word strings is a subset of the cartesian product of the word positions @xcite : a ⊆ {(s, t) : @xmath3, . . . , s; @xmath4, . . . , t } such that the aligned words can be considered translations of each other. an online alignment at timestep t commits on alignment of the t th output token conditioned only on x and @xmath5, y 2 , . . . y t-1 . additionally, if token y t is also available we call it a posterior online alignment. we seek to embed online alignment with existing nmt systems. we will first briefly describe the architecture of state of the art nmt systems. we will then elaborate on how alignments are computed from attention distributions in prior work and highlight some limitations, before describing our proposed approach. ## lexicon constrained translation in the lexicon constrained translation task, for each to-be-translated sentence x, we are given a set of source text spans and the corresponding target tokens in the translation. a constraint c j comprises a pair (c x j , c y j ) where c x @xmath6, p j + 1 . . . , p j + ℓ j ) indicates input token positions, and c y @xmath7, y j 2 . . . , y j m j ) denote target tokens that are translations of the input tokens x p j . . . x p j +ℓ j . for the output tokens we do not know their positions in the target sentence. the different constraints are non-overlapping and each is expected to be used exactly once. the goal is to translate the given sentence x and satisfy as many constraints in @xmath8. since the constraints do not specify target token position, it is natural to use online alignments to guide when a particular constraint is to be enforced. ## experiments we first compare our proposed posterior online alignment method on quality of alignment against existing methods in section 4.2, and in section 4.3, we demonstrate the impact of the improved alignment on the lexicon-constrained translation task. ## conclusion in this paper we proposed a simple architectural modification to modern nmt systems to obtain accurate online alignments. the key idea that led to high alignment accuracy was conditioning on the output token. further, our designed alignment module enables such conditioning to be performed synchronously with token generation. this property led us to align-vdba, a principled decoding algorithm for lexically constrained translation based on joint distribution of target token and source alignments. future work includes increase efficiency of constrained inference and harnessing such joint distributions for other forms of constraints, for example, nested constraints. limitations: all existing methods for hard constrained inference, including ours, come with considerable runtime overheads. soft constrained methods are not accurate enough.