M irror A lign: A Super Lightweight Unsupervised Word Alignment Model via Cross-Lingual Contrastive Learning (2022)

Abstract: Word alignment is essential for the downstream cross-lingual language understanding and generation tasks. Recently, the performance of the neural word alignment models has exceeded that of statistical models. However, they heavily rely on sophisticated translation models. In this study, we propose a super lightweight unsupervised word alignment model named MirrorAlign, in which bidirectional symmetric attention trained with a contrastive learning objective is introduced, and an agreement loss is employed to bind the attention maps, such that the alignments follow mirror-like symmetry hypothesis. Experimental results on several public benchmarks demonstrate that our model achieves competitive, if not better, performance compared to the state of the art in word alignment while significantly reducing the training and decoding time on average. Further ablation analysis and case studies show the superiority of our proposed MirrorAlign. Notably, we recognize our model as a pioneer attempt to unify bilingual word embedding and word alignments. Encouragingly, our approach achieves 16.4X speedup against GIZA++, and 50X parameter compression compared with the Transformer-based alignment methods. We release our code to facilitate the community: https://github.com/moore3930/MirrorAlign .

## introduction word alignment, aiming to find the word-level correspondence between a pair of parallel sentences, is a core component of the statistical machine translation @xcite . it also has benefited several downstream tasks, e.g., computeraided translation @xcite , semantic role labeling @xcite , crosslingual dataset creation @xcite , cross-lingual modeling @xcite , and cross-lingual text generation @xcite . recently, in the era of neural machine translation @xcite , the attention mechanism plays the role of the alignment model in translation system. unfortunately, @xcite show that attention mechanism may in fact dramatically diverge with word alignment. the works of @xcite ; @xcite also confirm this finding. although there are some studies attempt to mitigate this problem, most of them are rely on a sophisticated translation architecture @xcite . these methods are trained with a translation objective, which computes the probability of each target token conditioned on source tokens and previous target tokens. this will bring tremendous parameters and noisy alignments. most recent work avoids the noisy alignment of translation models but employed too much expensive human-annotated alignments @xcite . given these disadvantages, simple statistical alignment tools, e.g., fastalign @xcite and giza++ (och and ney, 2003) 2 , are still the most representative solutions due to their efficiency and unsupervised fashion. we argue that the word alignment task, intuitively, is much simpler than translation, and thus should be performed before translation rather than inducing alignment matrix with heavy neural machine translation models. for example, the ibm word alignment model, e.g., fastalign, is the prerequisite of smt. however, related research about lightweight neural word alignment without nmt is currently very scarce. inspired by cross-lingual word embeddings @xcite , we propose to implement a super lightweight unsupervised word alignment model in section 3, named mirroralign, which encourages the embedding distance between aligned words to be closer. we also provide the theoretical justification from mutual information perspective for our proposed contrastive learning objective in section 3.4, demonstrating the reasonableness of our method. figure 1 shows an english sentence, and its corresponding chinese and japanese sentences, and their word alignments. the links indicate the correspondence between english⇔chinese and english⇔japanese words. if the chinese word "举行" can be aligned to english word "held", the reverse mapping should also hold. specifically, a bidirectional attention mechanism with contrastive estimation is proposed to capture the alignment between parallel sentences. in addition, we employ an agreement loss to constrain the attention maps such that the alignments follow symmetry hypothesis @xcite . our contributions can be summarized as follows: ## related work word alignment studies can be divided into two classes: statistical models statistical alignment models directly build on the lexical translation models of @xcite , also known as ibm models. the most popular implementation of this statistical alignment model is fastalign @xcite and giza++ @xcite . for optimal performance, the training pipeline of giza++ relies on multiple iterations of ibm model 1, model 3, model 4 and the hmm alignment model @xcite . initialized with parameters from previous models, each subsequent model adds more assumptions about word alignments. model 2 introduces non-uniform distortion, and model 3 introduces fertility. model 4 and the hmm alignment model introduce relative distortion, where the likelihood of the position of each alignment link is conditioned on the position of the previous alignment link. fastalign @xcite , which is based on a reparametrization of ibm model 2, is almost the existing fastest word aligner, while keeping the quality of alignment. in contrast to giza++, our model achieves nearly 15× speedup during training, while achieving the comparable performance. encouragingly, our model is at least 1.5× faster to train than fastalign and consistently outperforms it. neural models most neural alignment approaches in the literature, such as @xcite , rely on alignments generated by statistical systems that are used as supervision for training the neural systems. these approaches tend to learn to copy the alignment errors from the supervising statistical models. @xcite use attention to extract alignments from a dedicated alignment layer of a neural model without using any output from a statistical aligner, but fail to match the quality of giza++. @xcite represents the current state of the art in word alignment, outperforming giza++ by training a single model that is able to both translate and align. this model is supervised with a guided alignment loss, and existing word alignments must be provided to the model during training. @xcite can produce alignments using an end-to-end neural training pipeline guided by attention activations, but this approach underperforms giza++. the performance of giza++ is only surpassed by training given y j , we can calculate context vector in source part. the nce training objective is encouraging the dot product of this context vector and y j to be large. the process in the other direction is consistent. by stacking all of the soft weights, two attention maps a @xmath0, which will be bound by an agreement loss to encourage symmetry. the guided alignment loss using giza++ output. @xcite introduce a discriminative neural alignment model that uses a dot-productbased distance measure between learned source and target representation to predict if a given sourcetarget pair should be aligned. alignment decisions are conditioned on the neighboring decisions using convolution. the model is trained using gold alignments. @xcite uses guided alignment training, but with large number of modules and parameters, they can surpass the alignment quality of giza++. they either use translation models for alignment task, which introduces a extremely huge number of parameters (compared to ours), making the training and deployment of the model cumbersome. or they train the model with the alignment supervision, however, these alignment data is scarce in practice especially for low resource languages. these settings make above approaches less versatile. instead, our approach is fully unsupervised at word level, that is, it does not require gold alignments generated by human annotators during training. moreover, our model achieves comparable performance and is at least 50 times smaller than theirs, i.e., #parameters: 4m (ours) vs. 200m (above). ## our approach our model trains in an unsupervised fashion, where the word level alignments are not provided. therefore, we need to leverage sentence-level supervision of the parallel corpus. to achieve this, we in-troduce negative sampling strategy with contrastive learning to fully exploit the corpus. besides, inspired by the concept of cross-lingual word embedding, we design the model under the following assumption: if a target token can be aligned to a source token, then the dot product of their embedding vectors should be large. figure 2 shows the schema of our approach mirroralign. ## analysis alignment case study figure 4 shows an alignment example. our model correctly aligns "do not believe" in english to "glauben nicht" in german. our model, based on word representation, makes better use of semantics to accomplish alignment such that inverted phrase like "glauben nicht" can be well handled. instead, fastalign, relied on the positional assumption foot_4 , fails here. ## conclusion and future work in this paper, we presented a super lightweight neural alignment model, named mirroralign, that has achieved better alignment performance compared to fastalign and other existing neural alignment models while preserving training efficiency. we empirically and theoretically show its effectiveness over several language pairs. in the future, we would further explore the relationship between clwes and word alignments. a promising attempt is using our model as a bridge to unify cross-lingual embeddings and word alignment tasks.