T rans A lign: Machine Translation Encoders are Strong Word Aligners, Too (2025)

Abstract: In the absence of sizable training data for most world languages and NLP tasks, translation-based strategies such as translate-test—evaluating on noisy source language data translated from the target language—and translate-train—training on noisy target language data translated from the source language—have been established as competitive approaches for cross-lingual transfer (XLT). For token classification tasks, these strategies require label projection: mapping the labels from each token in the original sentence to its counterpart(s) in the translation. To this end, it is common to leverage multilingual word aligners (WAs) derived from encoder language models such as mBERT or LaBSE. Despite obvious associations between machine translation (MT) and WA, research on extracting alignments with MT models is largely limited to exploiting cross-attention in encoder-decoder architectures, yielding poor WA results. In this work, in contrast, we propose TransAlign, a novel word aligner that utilizes the encoder of a massively multilingual MT model. We show that TransAlign not only achieves strong WA performance but substantially outperforms popular WA and state-of-the-art non-WA-based label projection methods in MT-based XLT for token classification.

## motivation and background in recent years, multilingual language models (mlms) have been positioned as the primary tool for cross-lingual transfer (xlt). by fine-tuning on task data in a high-resource source language, mlms can make predictions in target languages with no (zero-shot xlt) or limited (few-shot xlt) labeled examples @xcite @xcite . however, for token classification tasks (e.g., named entity recognition), translation-based xlt strategies, where a machine translation (mt) model is used to either (1) translate the original target language instance into the (noisy) source language before inference, known as translate-test (t-test), or (2) generate noisy target language data by translating the original source language data before training, known as translate-train (t-train) @xcite @xcite @xcite , substantially outperform zeroshot xlt @xcite @xcite , especially for low(er)-resource languages @xcite . translation-based xlt strategies for token classification tasks require the additional step of label projection: mapping the labeled spans from the original to the translated sentence. a broad body of work addressed label projection starting from task-specific @xcite @xcite @xcite and evolving to task-agnostic methods @xcite @xcite . while wa-based label projection @xcite @xcite @xcite which projects labels by establishing pairwise alignments between tokens in the original sentence and corresponding tokens in the translated sentenceserved as baseline throughout, recent work has rendered it less effective than other label projection strategies @xcite @xcite . however, @xcite show that wa-based label projection can perform at least on a par with these state-of-the-art projection methods as long as: (i) a robust algorithm for label projection and (ii) a sentence encoder-based wa model are used. in this work, we hypothesize that multilingual mt models are better suited for producing word alignments than was based on multilingual sen-tence encoders (e.g. labse) @xcite or vanilla encoders (e.g. mbert) @xcite , due to wa and mt being two highly related and interleaved tasks @xcite @xcite . yet, research on extracting word alignments from mt models has largely been limited to extracting alignments from the cross attention mechanism, yielding poor wa performance for transformer-based encoderdecoder mt models @xcite @xcite . contributions. this is why, (1) we propose transalign, a wa that leverages (only) the encoder of nllb @xcite , a massively multilingual encoder-decoder mt model. next, to its vanilla (non-fine-tuned) variant, we explore the impact of further fine-tuning transalign on parallel wa data. (2) we extensively evaluate transalign extrinsically on translation-based xlt for token classification on two established benchmarks covering 28 diverse languages. we find transalign to substantially outperform popular word aligners as well as a state-of-the-art non-wa-based label projection method. furthermore, we evaluate transalign intrinsically on the word alignment task showing its strong performance, particularly on words carrying semantic meaning. ## an mt encoder as a word aligner the task of word alignment aims at finding semantically corresponding pairs of words between a source language sentence @xmath0, x 2 , ..., x n ) and target language sentence @xmath1, y 2 , ..., y m ): extracting alignments. for transalign, we extract word alignments from the contextualized embeddings produced by the encoder of a multilingual encoder-decoder mt model. we separately feed the source language sentence x and target language sentence y through the encoder, obtaining their contextualized representations h x and h y , respectively. following prior work @xcite @xcite , we next obtain the token similarity matrix s xy : we row-and column-normalize the similarity matrix using softmax to obtain ŝxy and ŝyx , which capture the similarity from x to y and y to x. finally, we compute the alignment matrix a by intersecting the two similarity matrices: fine-tuning for word alignment. additionally, we explore fine-tuning transalign on a word alignment-specific objective to further improve performance. different from related work-that employed full fine-tuning @xcite or adapter-based fine-tuning @xcite -we opt for lora @xcite as it does not increase model depth while maintaining parameter efficiency. we resort to the following loss function for wa fine-tuning: ## experiments with label projection as the key remaining application of was, we first evaluate transalign on translation-based xlt for token classification. we then additionally benchmark transalign intrinsically on word alignment itself. ## limitations we focused on choosing well-established and representative tasks for token classification. however, in nlp, multilingual evaluation benchmarks are often created by translating the data from an existing high-resource language followed by postediting. this applies to xsid and some languages of masakha. as a result, the newly introduced languages might contain translation artifacts referred to as translationese. prior work @xcite stated that translation-based xlt strategies might lead to exploitation of translationese, slightly overestimating performance. our intrinsic evaluation points to a potential mismatch between the word alignment task and the extrinsic evaluation on translation-based xlt for token classification. our results suggest that the mismatch stems from the discrepancy of treating all words equally (intrinsic evaluation) against focusing on a specific subset of words (extrinsic evaluation). while we hypothesize as to why mt models perform worse in aligning words with little semantic meaning than sentence encoders, further work is needed to test our hypothesis. niraj aswani and robert gaizauskas. 2005. aligning words in english-hindi parallel corpora. in proceedings of the acl workshop on building and using parallel texts, pages 115-118, ann arbor, michigan. association for computational linguistics. dzmitry bahdanau, kyunghyun cho, and yoshua bengio. 2015. neural machine translation by jointly learning to align and translate. in 3rd international conference on learning representations, iclr 2015, san diego, ca, usa, may 7-9, 2015, conference track proceedings. mehmet talha çakmak, süleyman acar, and gülşen eryigit. 2012. word alignment for english-turkish language pair. in proceedings of the eighth international conference on language resources and evaluation (lrec'12), pages 2177-2180, istanbul, turkey. european language resources association (elra).