Improving Word Alignment Using Semi-Supervised Learning (2025)

Abstract: Word alignment plays a crucial role in various natural language processing tasks, such as serving as cross-lingual signals for sentence embedding, reducing hallucination and omission in machine translation, and facilitating the construction of training data for simultaneous speech translation.Current state-of-the-art approaches usually rely on: (1) supervised data and large-scale weakly supervised data constructed from Wikipedia and (2) multilingual Transformer encoder-based models.However, we find that the current state-of-the-art encoder-based method, BinaryAlign, suffers from the issue of insufficient labeled data, and we further improve it with self-training with a small amount of parallel data. In addition, considering the impressive performance of multilingual large language models on many natural language processing tasks, we also explore the possibility of using these decoder-based large language models as word aligners. We observe that although fine-tuning large language models with labeled data produces acceptable results, augmenting the training with pseudo-labeled data further enhances model performance. Based on the findings, we propose a semi-supervised framework to improve the large language model-based word aligners. Experimental results demonstrate that the proposed method with a small amount of parallel data outperforms the current state-of-the-art method on various word alignment datasets.

## introduction word alignment aims to identify correspondences between source and target words in a translation sentence pair, as shown in figure 1 . although word alignment was initially proposed to enhance statistical machine translation @xcite , advancements in both word alignment and deep learning techniques have broadened its application the english words hino in green and hino in blue are different for the japanese word 日野 , which apprears in the second position (zero-indexed) of the japanese sentence. the english word of in red does not have the corresponding japanese word (null word alignment). to a wide range of tasks in natural language processing (nlp). for example, @xcite use word alignment in cross-lingual language model pretraining. word alignment is utilized by @xcite to improve cross-lingual sentence embedding. @xcite leverage word alignment preference data to mitigate hallucination and omission in machine translation models. word alignment can also be utilized in the simultaneous speech translation domain for data curation @xcite . other possible applications include alleviating the overtranslation and under-translation problems in machine translation @xcite , xml-structured parallel text segment extraction @xcite and constrained neural machine translation @xcite . representative studies for improving word alignment with supervised methods include spanalign @xcite , wspalign @xcite and binaryalign @xcite . spanalign @xcite reformulates the word alignment task as a squad-style span prediction question answering task. ws-palign @xcite relaxes the requirement of figure 2 : trend of f1 scores as the amount of unlabeled parallel sentences in the training data increases, using our semi-supervised framework on the test set of the japanese-english word alignment dataset, @xcite . note that one unit (653) is the size of the labeled data. the yellow dotted line denotes the f1 score of the current state-of-the-art method, binaryalign @xcite . manually labeled data in spanalign with a largescale weak supervision pretraining dataset (2 million noisy sentence pairs) based on entity links from wikipedia and contextual word embeddings from multilingual encoder-based language models. binaryalign @xcite further reformulates the word alignment task as a binary sequence labeling task, achieving the new state-ofthe-art performance. these supervised methods for word alignment are highly dependent on (1) manually labeled datasets @xcite or large-scale weak word alignment signals extracted from wikipedia entity links and contextual word embeddings of encoder-based models to relax the strict requirement of labeled data @xcite and (2) multilingual encoder models @xcite @xcite . this raises the following questions: (1) can the potential of multilingual encoder-based models be fully leveraged by only using the labeled datasets for the current state-of-the-art method, bi-naryalign? if not, is there a way of using a small amount of data to improve it instead of utilizing a large-scale weakly supervised pre-training dataset like wspalign? (2) given the success of llms across a wide range of nlp tasks, how effective are multilingual llms for the word alignment task? we find that the issue of insufficient labeled data in word alignment prevents both multilingual encoder-based and decoder-based large language models from fully utilizing their advantages, leading to suboptimal performance as shown in figure 2 . figure 2 presents the trend of f1 scores on the japanese word alignment dataset kftt, as we progressively increase the number of parallel sentences used for training our multilingual encoder (semialign-e) and decoder (semialign-d) models. based on the aforementioned findings, we propose a semi-supervised framework named semialign, which is short for leveraging semisupervised learning to improve word alignment, designed to alleviate the challenge of limited labeled data in word alignment by utilizing a small amount of unlabeled parallel text. we demonstrate the effectiveness of our method for both multilingual encoder-based and decoder-based language models. the contributions of this paper can be summarized as follows: ## methodology background. given a source sentence @xmath0, . . . , x n ] and its target sentence @xmath1, . . . , y m ], where x i denotes the i-th word in the source sentence (1 ≤ @xmath2, word alignment aims to find a set of source and target word pairs in sentences x and y: the output space is a set of two-element ordered tuples. x i and y j in each pair of a are the words that ## results and analysis the main results are presented in a small amount of unlabeled data boosts the performance of binaryalign. we re-train bi- utilizing labeled data is not enough for multilingual llm-based word aligners. for a better comparison and investigation of factors that affect the word alignment performance of decoderbased language models, we mainly fine-tune the llama-3.1-8b-instruct for all languages with the labeled data as the baseline. for romanian-english, we use the aya-expanse-8b model because we find that its performance on this language pair is better. for the sft version of these large language models, the number of epochs is 5. the other hyperparameters are the same as the main experiments. in zero-shot setting. the results of zero-shot setting are shown in few-shot setting. we present the results of different methods in the few-shot setting in in this setting, we only use 32 labeled examples to train our base models, and generate pseudolabeled examples for unlabeled data using the ## related work word alignment. the research community has proposed various methods to improve word alignment based on the multilingual transformer encoder models @xcite . there are two categories of research that focus on improving word alignment performance on multilingual encoder-based language models @xcite . one category consists of methods based on contextual word embeddings extracted from encoder models @xcite @xcite . for example, simalign @xcite proposes three new alignment methods based on contextual word embeddings. the other category involves supervised learning for word alignment. spanalign @xcite redefines the word alignment task as a span prediction-based question answering task. wspalign @xcite proposes large-scale weakly pre-training for word alignment, using entity links and contextual word embeddings for entity and common words, due to the different representation level of them @xcite . binaryalign @xcite reformulates the word alignment task as a binary sequence labeling task, achieving the new state-of-the-art performance. both of the above categories are based on the multilingual encoder language models. however, considering that autoregressive decoder-based llms have achieved impressive performance on many nlp tasks, such as coding, reasoning and machine translation @xcite @xcite @xcite , we also explore the possibility of using multilingual llms as word aligners, an avenue that remains relatively unexplored by the research community. semi-supervised learning. semi-supervised learning aims to improve various machine learning systems with labeled data and unlabeled data @xcite . recently, researchers have successfully applied semi-supervised learning to various emergent domains, such as sentence semantics @xcite , multimodal domains @xcite @xcite , aspect sentiment quad prediction @xcite , improving reasoning ability of large language models @xcite and reward modeling @xcite . in this work, we investigate the application of semi-supervised learning on word alignment through self-training and pseudo-labeling. ## conclusion in this paper, we find that the current state-of-theart method, binaryalign, can be further improved with semi-supervised learning with a small amount of unlabeled data. we also explore the possibility of utilizing multilingual llms as word aligners. we observe that fine-tuning multilingual llms using the labeled data is insufficient, and not comparable to the current state-of-the-art binaryalign. therefore, we also utilize the pseudo-labeled data from the encoder-based word alignment models to improve multilingual llm-based word aligners in our proposed framework. both multilingual encoder models and llms that are trained with the proposed method achieve competitive performance, outperforming the current state-of-the-art method on average. the extensive experimental results demonstrate the effectiveness of the proposed method.