The Devil Is in the Word Alignment Details: On Translation-Based Cross-Lingual Transfer for Token Classification Tasks (2025)

Abstract: Translation-based strategies for cross-lingual transfer XLT such as translate-train—training on noisy target language data translated from the source language—and translate-test—evaluating on noisy source language data translated from the target language—are competitive XLT baselines. In XLT for token classification tasks, however, these strategies include label projection, the challenging step of mapping the labels from each token in the original sentence to its counterpart(s) in the translation. Although word aligners (WAs) are commonly used for label projection, the low-level design decisions for applying them to translation-based XLT have not been systematically investigated. Moreover, recent marker-based methods, which project labeled spans by inserting tags around them before (or after) translation, claim to outperform WAs in label projection for XLT. In this work, we revisit WAs for label projection, systematically investigating the effects of low-level design decisions on token-level XLT: (i) the algorithm for projecting labels between (multi-)token spans, (ii) filtering strategies to reduce the number of noisily mapped labels, and (iii) the pre-tokenization of the translated sentences. We find that all of these substantially impact translation-based XLT performance and show that, with optimized choices, XLT with WA offers performance at least comparable to that of marker-based methods. We then introduce a new projection strategy that ensembles translate-train and translate-test predictions and demonstrate that it substantially outperforms the marker-based projection. Crucially, we show that our proposed ensembling also reduces sensitivity to low-level WA design choices, resulting in more robust XLT for token classification tasks.

## introduction in recent years, multilingual language models (mlms) have de facto become the main vehicle of cross-lingual transfer (xlt): fine-tuned on labeled task data in a high-resource source language, mlms can make predictions in target languages with few (few-shot xlt) to no (zero-shot xlt) labeled task instances @xcite @xcite . while both encoder-only @xcite @xcite and decoder-only @xcite @xcite mlms have demonstrated strong xlt performance for sequence classification tasks, in xlt for token classification tasks the comparatively smaller encoder-only mlms, like xlm-r @xcite , continue to outperform the much larger decoder mlms @xcite @xcite . much of the above work highlights translationbased strategies as competitive approaches for xlt, where a machine translation (mt) model is used to either (1) generate noisy target language data by translating the original source language data before training, known as translate-train (t-train), or (2) translate original target language instances into the (noisy) source language before inference, known as translate-test (t-test) @xcite @xcite . more elaborate translation-based xlt strategies have recently been shown to further improve the transfer performance @xcite . the effectiveness of translation-based xlt, however, has predominantly been showcased on sequence-level classification tasks @xcite @xcite . this is in part due to the fact that translation-based xlt for token classification tasks entails the (difficult) step of label projection. traditionally, label projection is tackled with word aligners (was) @xcite @xcite , which map each token in the source sequence to a corresponding token in the target se-quence. recent wa work leverages contextualized embeddings from mlms (e.g., mbert) to produce token alignments @xcite @xcite . although wa research has a long-standing track record in nlp @xcite @xcite @xcite , standard wa evaluation protocols do not include translation-based xlt for token classification tasks. because of this, the impact of low-level design decisions related to token-level xlt using was-such as (i) the exact algorithm for projecting the labels, (ii) filtering techniques to reduce the number of noisily mapped labels, and (iii) the process of inducing word boundaries (i.e. pre-tokenization) to the translated sentence before it can be aligned to the tokens in the original sentence-remain underinvestigated @xcite . in the meantime, marker-based label projection @xcite has largely replaced wa for label projection for token-level xlt. these approaches insert tags (e.g., "[", "]") around labeled spans of interest, either (i) before translation to preserve the markers throughout the translation process and recover the spans afterward @xcite , or (ii) post-translation by means of constrained decoding @xcite ). 1 this line of work explicitly evaluates token-level translation-based xlt, demonstrating strong performance for both t-train and t-test. furthermore, it renders wa-based xlt for token classification tasks inferior @xcite . while these efforts provide the technical details for their proposed marker-based methods, they do not lay out the low-level design details for label projection with was. therefore, we argue that they possibly underestimate translation-based xlt with was due to suboptimal design choices. contributions. because of this, we (1) systematically investigate wa for token-level translationbased xlt. we start by evaluating the effect of lowlevel design decisions covering (i) the exact algorithm for mapping the labels between (multi-)token spans from the source sentence to the target sentence based on word alignments; (ii) filtering strate-1 further methods have been proposed that address label projection by constrained or contextualized generation of labeled spans given the translated input sentence @xcite . we provide details on these methods in app. e and f. in our preliminary experiments these methods performed at most on par with codec (see app. m). ## token-level xlt via word alignment we first detail our pipeline for label projection with wa focusing on the low-level design choices we investigate: mapping labeled spans, filtering strategies, and pre-tokenization @xcite . we then describe our new translation-based xlt approach for token-level tasks that ensembles t-train and t-test. ## experimental setup machine translation. for translation, we utilize the state-of-the-art massively multilingual nllb model with 3.3b parameters @xcite . following prior work @xcite , we decode using beam search with a beam size of 5. evaluation tasks. we evaluate on two established token classification tasks: named entity recognition and slot labeling. our experiments span 29 diverse languages, ranging from high-resource languages, represented well in the pretraining corpus of the base mlm, to low-resource languages, unseen by the mlm. we use english as our source language. 3 named entity recognition (ner). our evaluation includes 18 of 20 languages from masakhaner 2.0 (masakha) @xcite supported by the nllb model used for translation. masakha consists of underserved languages from sub-saharan africa. as source data, we use the english training (14k instances) and validation portions (3250 instances) of conll @xcite . we add a softmax classifier on top of the mlm to predict the class for each token. ## results and discussion starting from the validation portions of our datasets, we assess the impact of low-level design choices related to using wa for token-level translation-based xlt ( §4.1). based on these findings, we compare wa-applying the on average best-performing pretokenization and filtering strategies-against two state-of-the-art marker-based methods, easy and codec, on the test portions of our datasets ( §4.2). last, we provide further ablations in §4.3, analyzing the impact of decoder-turned encoder llms, wa, and mt model on the xlt performance. ## conclusion in this work, we thoroughly investigated the role of word aligners (was) in translation-based crosslingual transfer for token classification tasks. our evaluation on two established benchmarks covering 29 languages, revealed that low-level design decisions related to label projection via wa can have a substantial effect on translation-based xlt strategies, in particular translate-test. we then showed that an optimized application of wa-based label projection can match or even surpass the transfer performance of recent marker-based approaches @xcite , contrary to their findings. further, we proposed a more sophisticated wa-based transfer approach that ensembles predictions of translate-train and translate-test. we demonstrated that the proposed ensemble not only substantially increases transfer performance but also reduces the sensitivity to low-level design decisions of wa-based label projection.