Graph Neural Networks for Multiparallel Word Alignment (2022)

Abstract: After a period of decrease, interest in word alignments is increasing again for their usefulness in domains such as typological research, cross-lingual annotation projection and machine translation. Generally, alignment algorithms only use bitext and do not make use of the fact that many parallel corpora are multiparallel. Here, we compute high-quality word alignments between multiple language pairs by considering all language pairs together. First, we create a multiparallel word alignment graph, joining all bilingual word alignment pairs in one graph. Next, we use graph neural networks (GNNs) to exploit the graph structure. Our GNN approach (i) utilizes information about the meaning, position and language of the input words, (ii) incorporates information from multiple parallel sentences, (iii) adds and removes edges from the initial alignments, and (iv) yields a prediction model that can generalize beyond the training sentences. We show that community detection algorithms can provide valuable information for multiparallel word alignment. Our method outperforms previous work on three word alignment datasets and on a downstream task.

## introduction word alignments are crucial for statistical machine translation @xcite and useful for many other multilingual tasks such as neural machine translation @xcite , typological analysis @xcite @xcite and annotation projection @xcite @xcite . the rise of deep learning initially led to a temporary plateau, but interest in word alignments is now increasing, demonstrated by several recent publications @xcite @xcite . while word alignment is usually considered for bilingual corpora, our work addresses the problem of word alignment in multiparallel corpora, containing sentence level parallel text in more than two languages. examples of multiparallel corpora are jw300 @xcite , pbc @xcite which covers the highest number of languages (1334), and tatoeba. 1 while the perlanguage amount of data provided in such corpora is less than bilingual corpora, they support highly low-resource languages, many of which are not covered by existing language technologies @xcite . therefore, these corpora are essential for studying many of the world's low-resource languages. we consider the task of word alignment for multiparallel sentences. the basic motivation is that the alignment between words in languages u and v can benefit from word-level alignments of u and v with a translation in a third language w . following up on the work of imani @xcite , we model multilingual word alignments with tools borrowed from graph theory (community detection algorithms) combined with neural network based models, specifically, the graph neural network (gnn) model of @xcite . gnns were proposed to extend the powerful current generation of neural network models to the processing of graph-structured data and they have gained increasing popularity in many domains @xcite @xcite . gnns can incorporate heterogeneous sources of signal in the form of node and edge features. we use this property to take into account properties of the whole alignment graph, notably its tendency to cluster into communities, see figure 1 . with our new proposed methods, we obtain improved results on word alignment for three language pairs: english-french, finnish-hebrew and finnish-greek. as a demonstration of the importance of high-quality alignments, we use our word alignments to project annotations from highresource to low-resource languages. we improve a part-of-speech tagger for yoruba by training it over a high-quality dataset, which is created using annotation projection. ## predicting and using multiparallel after the gnn model is trained on multiple mpwa graphs, it is used to infer an alignment probability matrix between tokens in a source language and tokens in a target language for a multiparallel sentence, see §3.1.4. our method predicts new alignment links from this matrix, independently of initial edges. therefore, given an initial bilingual alignment, it is not limited to adding edges, but it can also remove them. ## conclusion and future work we introduced graph neural networks and community detection algorithms for multiparallel word alignment. by incorporating signals from diverse sources as node features, including community features, our gnn model outperformed the baselines and prior work, establishing new state-of-the-art results on three pbc gold standard datasets. we also showed that our gnn model improves downstream task performance in low-resource languages through annotation projection. we have only used node features to provide signals to gnns. in the future, other signals can be added in the form of edge features to further boost the performance.