Structural Supervision for Word Alignment and Machine Translation (2022)

Abstract: Syntactic structure has long been argued to be potentially useful for enforcing accurate word alignment and improving generalization performance of machine translation. Unfortunately, existing wisdom demonstrates its significance by considering only the syntactic structure of source tokens, neglecting the rich structural information from target tokens and the structural similarity between the source and target sentences. In this work, we propose to incorporate the syntactic structure of both source and target tokens into the encoder-decoder framework, tightly correlating the internal logic of word alignment and machine translation for multi-task learning. Particularly, we won’t leverage any annotated syntactic graph of the target side during training, so we introduce Dynamic Graph Convolution Networks (DGCN) on observed target tokens to sequentially and simultaneously generate the target tokens and the corresponding syntactic graphs, and further guide the word alignment. On this basis, Hierarchical Graph Random Walks (HGRW) are performed on the syntactic graphs of both source and target sides, for incorporating structured constraints on machine translation outputs. Experiments on four publicly available language pairs verify that our method is highly effective in capturing syntactic structure in different languages, consistently outperforming baselines in alignment accuracy and demonstrating promising results in translation quality.

## introduction word alignment @xcite aims to find the correspondence between tokens in a sentence pair. neural machine translation (nmt) @xcite works by taking an end-to-end approach to incrementally predict the target translation from a source sentence, where no explicit word alignment is required during model training or decoding. recently, there has been an german chinese 1-hop 2-hop syntax-graph you re naive nsubj acomp conj cc t u es naï ve du bist naiv 你 很 天真 nsubj advmod sb oa regardless of direction and type: (1) the dependencies between 'you @xcite in english match the dependencies between 'du @xcite german. although nmt-based aligners have proven to be effective and achieved the state-of-the-art alignment accuracy, they suffer from two major limitations. first, due to the autoregressive property @xcite , they @xcite @xcite ) only leverage partial target context. the latest works @xcite alleviate this deficiency to exploit both sides of the target content to compute better target-to-source attention (alignment), by abandoning autoregressive decoder and sacrificing the translation ability. in addition, there are also related works @xcite proposing syntax-aware nmt models without word alignment task. however, they simply utilize the syntactic structure of source tokens and ignore to capture the syntactic structure of target tokens. in summary, the syntactic structure of both source and target tokens has not been thoroughly explored to guide accurate alignments, while the similarity of dependencies across diverse languages has not been utilized for producing translation outputs with high-quality and favorable generalization capabilities. second, they @xcite typically use multi-task learning architecture to jointly learn the word alignment and translation with elaborately designed loss functions. however, this is computationally expensive for training and the internal logic between the two subtasks is not well correlated. to alleviate mentioned problems, we propose to simultaneously consider the syntactic structure of both source and target tokens. according to the similar dependencies across language pairs, the syntactic graphs of target tokens are first sequentially inferred through introduced dynamic graph convolution networks. hierarchical graph random walks are then performed based on the built syntactic graphs at both ends, as well as the initialized multi-scale and trainable "hidden graphs" @xcite . we found that by correlating cross-linguistic dependencies without any additional guided loss, word alignment and translation can be more effectively integrated into a unified learning framework, efficiently correlating the internal logic between subtasks while improving the interpretability of the model. our contributions are as follows: (1) we introduce dynamic graph convolution networks to sequentially infer the syntactic graphs of target tokens and further guide the word alignment learning. (2) hierarchical graph random walks are further performed to incorporate both local and global structural constraints for producing translation outputs. ## approach our work is inspired by the fact that tokens in different languages have similar dependencies under the same semantics. as shown in figure 1 , the dependencies between tokens with the same semantics in the english-german pair are highly similar, while the similarity of dependencies between english and french (chinese) can also be implicitly captured. ## conclusion we propose a multi-task learning framework that tightly correlates the internal logic of word alignment and machine translation, by fully exploits the syntactic structure of both source and target tokens and the similarity of dependencies at both ends. experiments show that our proposal achieves the new state-of-the-art results among all neural methods in word alignment, while producing high-quality translations. we leave it for future work to extend our study to more downstream tasks and systems in natural language processing.