B inary A lign: Word Alignment as Binary Sequence Labeling (2024)

Abstract: Real world deployments of word alignment are almost certain to cover both high and low resource languages. However, the state-of-the-art for this task recommends a different model class depending on the availability of gold alignment training data for a particular language pair. We propose BinaryAlign, a novel word alignment technique based on binary sequence labeling that outperforms existing approaches in both scenarios, offering a unifying approach to the task. Additionally, we vary the specific choice of multilingual foundation model, perform stratified error analysis over alignment error type, and explore the performance of BinaryAlign on non-English language pairs. We make our source code publicly available.

## introduction word alignment refers to the task of uncovering word correspondences between translated text pairs. the automatic prediction of word alignments dates back to the earliest work in machine translation with the ibm models @xcite where they were used as hidden variables that permit the use of direct token to token translation probabilities. while state of the art machine translation techniques have largely abandoned the use of word alignment as an explicit task @xcite other use cases for alignments have emerged including lexical constraint incorporation @xcite , analysing and evaluating translation models @xcite , and cross-lingual language pre-training @xcite . in many real-world applications word alignment must be performed across several languages, often including languages with manually annotated word alignment data and others lacking such annotations. we refer to those languages as high and low-resource languages respectively. while word alignment for high-resource languages can be learned in a few-shot or fully supervised setting depending on the amount of data, for low-resource languages zero-shot learning strategies must be employed due to data scarcity. state-of-the-art supervised techniques formalize the task of word alignment as a collection of squad-style span prediction problems @xcite while in zero-shot settings the best performing methods induce word alignment from the contextualized word embeddings of mulitingual pre-trained language models (mplms) @xcite @xcite . from a practical perspective, this discrepancy in the preferred method adds complexity to the deployment of word alignment models in real-world applications where both high and low-resource languages must be supported. we observe a deeper issue that both span prediction and contextualized word embeddings are sub-optimal as each induces a bias in word alignment models that limits their accuracy. span prediction methods cannot robustly deal with discontinuous word alignments without relying on complex post-processing and hyper-parameter tuning. contextualized word embeddings method cannot deal effectively with untranslated words and one-tomultiple alignments because they rely on a softmax function that normalizes predictions at a sentencelevel while in word alignment; one token being aligned to token t does not mean that another token is less likely to be aligned to t . this poses word alignment as a single-label classification problem, while in reality it is better viewed as a series of binary classifications applied to each possible pair of words. figure 1 shows some cases of oneto-multiple alignments, non-contiguous spans and untranslated words. in this paper, we present binaryalign, a novel word alignment solution that outperforms the state-of-the-art in zero-shot, few-shot and fullysupervised settings. in particular, we reformulate word alignment as a set of binary classification tasks in which an individual alignment prediction is made for each possible pair of words. this reformulation of the task outperforms all previous approaches over five different language pairs with varying levels of supervision. ## related work recently, mplm based approaches have significantly outperformed bilingual statistical methods @xcite @xcite and bilingual neural methods @xcite @xcite @xcite . among those approaches, we distinguish methods that achieve good performance without relying on manually annotated word alignment datasets @xcite @xcite from supervised methods that leverage existing word alignment datasets to train high performing word aligners @xcite . the first type of method relies on the approach of simalign (jalili sabet et al., 2020) which proposes to induce word alignment from the contextualized word embeddings of mplms pre-trained on nonparallel data. awesomealign @xcite builds on top of this approach and proposes to fine-tune mplms on parallel text with different objectives to improve the quality of the contextualized word embeddings. more recently, accalign @xcite showed that models trained to learn language-agnostic sentence-level embeddings also learn strong language-agnostic word-level embeddings and set the state of the art in the zeroshot setting. also, @xcite show that fine-tuning on existing word alignment datasets improves performance of accalign on language pairs unseen during word alignment fine-tuning. our method is different from this body of work because our training and inference objective differ. we formalize word alignment as a binary sequence labeling task while we can see those methods as framing word alignment as a token retrieval task. in terms of approaches trained and evaluated in a supervised setting spanalign @xcite formalizes word alignment as a collection of squad-style span prediction problems which differs from our binary classification objective. however, this method falls short in zero-shot and fewshot when word alignment training data is scarce. to remedy this problem, wspalign @xcite introduces a pre-training method based on weak supervision that significantly improves performance for all amounts of training data. ## method given a sentence x with n words and a translation into another language y with m words, the task of word alignment is to produce an n by m adjacency matrix for the bipartite graph with the words of x on one side and the words of y on the other (refer to figure 1 for an illustration). as our model will employ commonly used subword tokenization preprocessing we assume access to an invertible tokenizer, often implemented in practice with a list of subword units, a greedy algorithm for subword chunking, and leading symbols to denote word continuation in the vocab file @xcite . we present binaryalign, a novel word alignment approach using a binary sequence labeling model, shown in figure 2 . the inputs to this model are a subword tokenized source sentence @xmath0, x 2 , ..., x |x| , a subword tokenized target sentence @xmath1, y 2 , ..., y |y | , and a reference word w @xmath2: j] which is a subspan of x. we model the distribution of a binary alignment vector a of size |y | in which each entry a k indicates if the word in y that contains y k is aligned to w x . we first preprocess x by surrounding w x with unique separator tokens and then cross-encode the source and target sentences with an mplm. for each token y k in the target sentence we pass its final encoded representation through a linear layer to produce a single logit z k . we model a k with a logistic function using z k as its parameter: a supervised signal for token level alignments a k is easily divined from word alignment data and so this form is sufficient to estimate the parameters of the model. however, our true inference-time goal of word to word alignment requires the use of additional heuristics. to motivate these heuristics we formalize w as the inverse of the surjective mapping between the token indices in y and its corresponding words; for any subspan w y of y , w(w y ) returns the token indices in y that compose w y . given an aggregation function agg, we define the probability of the event a ′ that there exists an alignment between word w x in x and word w y in y as preliminary experiments suggest that the maximum aggregation strategy yields slightly superior performance compared to the mean and minimum aggregation strategies; hence, we adopt it for all subsequent experiments. word alignment in its general form is a symmetric problem in that we would expect the same answer if the source and target were swapped. however, like most leading word alignment methods, our method is asymmetric; the source and target sentences are handled differently. this deficiency is empirically detrimental to performance with the common remedy being to perform alignment in both directions and then to merge the two predictions in some manner. we use the following symmetrization technique: letting p @xmath3, we average p @xmath4. while outside the scope of this study, we note that various other options exist and have been explored in previous work such as bidirectional average @xcite or intersection, union and grow-diag-final: the default symmetrization heuristics supported in moses @xcite . ## conclusion we presented binaryalign, a novel word alignment training and inference procedure. in particular, we proposed to reformulate the word alignment problem as a binary token classification task. we showed that because of this reformulation bina-ryalign outperforms existing methods regardless of the degree of supervision. in addition we showed that it overcomes the inherent limitations of previous methods relying on span prediction and softmax. as a result, we made the word alignment task easier to tackle by using a single model for both high and low-resource languages. in the future we plan to explore the use of larger decoder-only or encoder-decoder models such as mt5 @xcite to see how much alignment performance will increase. we also plan on investigating knowledge distillation techniques to improve the inference time of our method.