Constraining word alignments with posterior regularization for label transfer (2022)

Abstract: Unsupervised word alignments offer a lightweight and interpretable method to transfer labels from high- to low-resource languages, as long as semantically related words have the same label across languages. But such an assumption is often not true in industrial NLP pipelines, where multilingual annotation guidelines are complex and deviate from semantic consistency due to various factors (such as annotation difficulty, conflicting ontology, upcoming feature launches etc.);We address this difficulty by constraining the alignments models to remain consistent with both source and target annotation guidelines , leveraging posterior regularization and labeled examples. We illustrate the overall approach using IBM 2 (fast_align) as a base model, and report results on both internal and external annotated datasets. We measure consistent accuracy improvements on the MultiATIS++ dataset over AWESoME, a popular transformer-based alignment model, in the label projection task ( +2.7\% at word-level and +15\% at sentence-level), and show how even a small amount of target language annotations help substantially.

## introduction the task of aligning words in parallel sentences (i.e bitexts) originates from statistical machine translation @xcite , where semantic identification was performed based on context similarity in accordance to the well-known distributional hypothesis. the most commonly used statistical aligners are built on top of the so-called ibm models @xcite , a series of structured probabilistic models that, while fully unsupervised, often rely on additional assumptions (such as close-todiagonal alignment) to reach acceptable accuracies. these approaches have since been superseded by neural networks and pretrained embeddings. they nonetheless enjoy a wide popularity across many cheap multilingual word alignments are appealing as they provide a transparent and interpretable way to transfer features from a source language to a target language (see fig. 1 ). they have been used in the past to transfer costly annotations such as part-of-speech @xcite or coreference information from high-to low-resource languages @xcite . however, the reliability of such a strategy depends on the use case at hand and we argue that it can lead to subtle but systematic failures in downstream tasks. in our industrial use case (that of a voice assistant), multilingual named-entity annotation guidelines factor in a great number of aspects (country launches, available features, human-friendly rules for annotators e.t.c) and end up surprisingly riddled with inconsistencies across languages (see we evaluate our method on two annotated datasets and show that it combines the strengths of both approaches: the inferred alignments produce better labels than either the baseline aligners or the n-gram models alone. it also remains fast, interpretable, self-contained and data-efficient, which makes it easy to integrate into industrial nlp pipelines. however, it has the same drawbacks that ibm model 2 has (no fertility modelling -i.e cannot handle a single source word generating multiple words in the target language, n-1 source-target mapping, danger of local optima during training). we release our implementation as fastlabel foot_0 . ## related work statistical word alignment models continue to be widely used to transfer labels from high-to low-resource languages owing to their speed, low memory footprint and interpretability. their most famous exponents are the ibm models 1 to 4 @xcite , a bayesian models hierarchy of increasing sophistication. fast_align @xcite is a fast reparameterization of ibm model 2 that significantly cuts down training and inference time. eflomal @xcite augments ibm model 1 with priors on word order and fertility, and uses markov chain monte carlo (mcmc) to do inference. much of the recent work depart from the bayesian modeling tradition by relying on contextual embeddings to perform the alignment @xcite @xcite . awesome @xcite uses multilingual bert @xcite to extract word alignments, and allows fine-tuning the underlying bert model on parallel corpora to improve alignment quality. while very accurate, they leverage embeddings from computationally expensive neural networks, and as such, they are not self-contained and the errors made by these models are arguably less interpretable than the simpler statistical models presented here. @xcite introduced expectation regularization as a way to encourage unsupervised model predictions to match an expectation from an external prior. @xcite developed the constraint driven learning (codl) framework that is capable of allowing different levels of constraint violation. their formulation, however, did not allow for tractable inference and the authors used beam search to solve the optimization problem. the posterior regularization framework introduced by @xcite allows constraint violations while remaining tractable. applications of statistical word alignment to label projection are numerous. label projection using word alignments is discussed in @xcite , @xcite , östling (2016), @xcite and @xcite . the last three models use the stanford pos tagger @xcite on a high resource source-language and transfer the labels to the target language. ## model formulation we start with the notations and closely follow @xcite for clarity. the source (target) sentence is denoted f (e), of length n (m). the aim is to infer, from bitexts, an alignment a a @xmath0, a 2 , • • • , a m ⟩ from source to target: each a i refers to the position of the source sentence word aligned to the ith word in the target sentence (see figure 1 ). we will assume that each target target word is associated to at most one source word: this n -1 mapping limitation is not a concern in the context of label projection. in the ner (named entity recognition) setup, both source and target sentences may be annotated with ner labels, and we write l the set of possible labels, and ℓ e i (resp. ℓ f j ) the label attached to e i (resp. f j ) ; ℓ e and ℓ f refer to the label sequences of the whole sentences e and f . the parameters of the popular ibm models are usually inferred through maximum likelihood (ml) the examples from our internal dataset show some of the discrepancies present in annotation guidelines across languages -for example, the english token-label pair "and|o" corresponds to "et|length" in french. we also observe inconsistencies arising due to word fertility and tokenization choices -"what" corresponds to "che c' è" (i.e 3 different tokens) in italian and the two words "turn off" corresponds to the single word "desligue" in portuguese. we can therefore rewrite such matching condition as: the set of contraints, one per label configuration per target sentence, is denoted c. in this case, the e-step admits a dual formulation and the optimal alignment distribution q * has a simple expression in terms for the unconstrained p f a : the iterative algorithm closely mimics the classical em coordinate ascent, with the addition of solving the lagrange multipliers (see appendix a). the value of the lagrange multipliers λ * c are computed through gradient ascent over z({λ * c }). ibm model 2 enjoys the property that its alignment probability p f a factors over the words of each target sentence. it is therefore convenient to split c accordingly: to each word e i and each possible ℓ ∈ l, are attached a lagrange multiplier λ e i ℓ and the cost v e i ℓ of labelling e i with ℓ. in such case, z({λ * c }) further decomposes: z({λ * c }) = @xmath1. e @xmath2 the stationary points is reached when v e i ℓ = 0, selecting alignments for which the transferred label distribution matches r(ℓ). are described in appendix c. since our work is an extension of fast_align, we ported the original fast_align foot_3 code to python and extended it to support posterior regularization. just like the original fast_align implementation, we did 5 iterations of expectation-maximization to train the model. the trained alignment model (i.e q * in equation 3) is then evaluated on a held out set of bitexts. for each aligned word pair, the label of the source word (usually from an english sentence) is transferred to the aligned target word. all target words aligned to the "null" token are given a label of "o" (for "other"). we then compare the transferred labels to the true labels of the target sentence to calculate the accuracy. though the label transfer happens at a word level, we report accuracies at the sentence level as well since perfectly annotated sentences are crucial for our industrial use case. the n-gram classifiers in the tables are simple frequency-based classifiers trained on the target language -for a particular ngram in the test set, the classifier annotates the nth word with the most frequent label assigned to that n-gram in the training data. for n-grams that were not present in the training data (even after backingoff to unigrams), the classifier outputs the label "o" (for "other"). these simple classifiers are essentially the same models that are used to do posterior regularization in our experiments -when used as classifiers, they only output the most likely label for a given n-gram while during regularization we use their entire label distribution. 1 ), the diagonal prior used by fast_align (i.e the assumption that words in target sentence are aligned to the words in relatively the same position in the source sentence) can be problematic. the superior performance of fastlabel ( ## results our results are reported in we observe a large drop in performance for fast_align when aligning language from different families (such as english-chinese bitexts), due to the well-known limitations of the diagonal prior assumption. moreover, as observed in