Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment (2022)

Abstract: Word alignment which aims to extract lexicon translation equivalents between source and target sentences, serves as a fundamental tool for natural language processing. Recent studies in this area have yielded substantial improvements by generating alignments from contextualized embeddings of the pre-trained multilingual language models. However, we find that the existing approaches capture few interactions between the input sentence pairs, which degrades the word alignment quality severely, especially for the ambiguous words in the monolingual context. To remedy this problem, we propose Cross-Align to model deep interactions between the input sentence pairs, in which the source and target sentences are encoded separately with the shared self-attention modules in the shallow layers, while cross-lingual interactions are explicitly constructed by the cross-attention modules in the upper layers. Besides, to train our model effectively, we propose a two-stage training framework, where the model is trained with a simple Translation Language Modeling (TLM) objective in the first stage and then finetuned with a self-supervised alignment objective in the second stage. Experiments show that the proposed Cross-Align achieves the state-of-the-art (SOTA) performance on four out of five language pairs.

## introduction word alignment which aims to extract the lexicon translation equivalents between the input sourcetarget sentence pairs @xcite @xcite , has been widely used in machine translation @xcite @xcite , transfer text annotations (fang i s r a e l figure 1 : an example from @xcite . there is a misalignment between "以" and "to" and "for". red boxes denote the gold alignments. and cohn, 2016; @xcite , typological analysis @xcite , generating adversarial examples @xcite , etc. statistical word aligners based on the ibm translation models @xcite , such as giza++ @xcite and fastalign @xcite , have remained popular over the past thirty years for their good performance. recently, with the advancement of deep neural models, neural aligners have developed rapidly and surpassed the statistical aligners on many language pairs. typically, these neural approaches can be divided into two branches: neural machine translation (nmt) based aligners and language model (lm) based aligners. nmt based aligners @xcite @xcite @xcite take alignments as a by-product of nmt systems by using attention weights to extract alignments. as nmt models are unidirectional, two nmt models (source-to-target and target-to-source) are required to obtain the final alignments, which makes the nmt based aligners less efficient. as opposed to the nmt based aligners, the lm based aligners generate alignments from the contextualized embeddings of the directionless multilingual language models. they extract contextualized embeddings from lms and induce alignments based on the matrix of embed-ding similarities @xcite . while achieving some improvements in the alignment quality and efficiency, we find that the existing lm based aligners capture few interactions between the input source-target sentence pairs. specifically, simalign @xcite encodes the source and target sentences separately without attending to the context in the other language. @xcite further propose awesome-align, which considers the cross-lingual context by taking the concatenation of the sentence pairs as inputs during training, but still encodes them separately during inference. however, the lack of interaction between the input source-target sentence pairs degrades the alignment quality severely, especially for the ambiguous words in the monolingual context. figure 1 presents an example of our reproduced results from awesome-align. the ambiguous chinese word "以" has two different meanings: 1) a preposition ("to", "as", "for" in english), 2) the abbreviation of the word "以色列" ("israel" in english). in this example, the word "以" is misaligned to "to" and "for" as the model does not fully consider the word "israel' in the target sentence. intuitively, the crosslingual context is very helpful for alleviating the meaning confusion in the task of word alignment. based on the above observation, we propose cross-align, which fully considers the crosslingual context by modeling deep interactions between the input sentence pairs. specifically, cross-align encodes the monolingual information for source and target sentences independently with the shared self-attention modules in the shallow layers, and then explicitly models deep cross-lingual interactions with the cross-attention modules in the upper layers. besides, to train cross-align effectively, we propose a two-stage training framework, where the model is trained with the simple tlm objective @xcite to learn the cross-lingual representations in the first stage, and then finetuned with a self-supervised alignment objective to bridge the gap between training and inference in the second stage. we conduct extensive experiments on five different language pairs and the results show that our approach achieves the sota performance on four out of five language pairs. foot_0 compared to the existing approaches which apply many complex training objectives, our approach is simple yet effective. our main contributions are summarized as follows: ## method in this section, we first introduce the model architecture and then illustrate how we extract alignments from cross-align. finally, we describe the proposed two-stage training framework in detail. ## experimental settings in this section, we first describe the details of datasets and implementation, then present the baselines, and finally introduce evaluation measures. ## conclusion this paper presents a novel lm based aligner named cross-align, which models deep interactions between the input sentence pairs. cross-align first encodes the source and target sentences sep-arately with the shared self-attention modules in the shallow layers, then explicitly constructs crosslingual interactions with the cross-attention modules in the upper layers. additionally, we propose a simple yet effective two-stage training framework, where the model is first trained with a simple tlm objective and then finetuned with a self-supervised alignment objective. experimental results show that cross-align achieves new sota results on four out of language pairs. in future work, we plan to improve the alignment quality on more lowresource language pairs.