A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT (2020)

Abstract: We present a novel supervised word alignment method based on cross-language span prediction. We first formalize a word alignment problem as a collection of independent predictions from a token in the source sentence to a span in the target sentence. Since this step is equivalent to a SQuAD v2.0 style question answering task, we solve it using the multilingual BERT, which is fine-tuned on manually created gold word alignment data. It is nontrivial to obtain accurate alignment from a set of independently predicted spans. We greatly improved the word alignment accuracy by adding to the question the source token’s context and symmetrizing two directional predictions. In experiments using five word alignment datasets from among Chinese, Japanese, German, Romanian, French, and English, we show that our proposed method significantly outperformed previous supervised and unsupervised word alignment methods without any bitexts for pretraining. For example, we achieved 86.7 F1 score for the Chinese-English data, which is 13.3 points higher than the previous state-of-the-art supervised method.

## introduction over the last several years, machine translation accuracy has been greatly improved by neural networks @xcite @xcite @xcite . however, word alignment tools, which were developed during the age of statistical machine translation @xcite such as giza++ @xcite , mgiza @xcite and fastalign @xcite , remain 1 our implementation is available at https://github.com/nttcslab-nlp/word align/ widely used because the improvement of word alignment accuracy has become stagnant. this situation is unfortunate because word alignment could be used for many downstream tasks including projecting linguistic annotation @xcite , projecting xml markups @xcite , and enforcing terminology constraints (pre-specified translation) @xcite . we could also use it for the user interfaces of post-editing to detect such problems as under-translation @xcite . word alignment has a long research history. here, we focus on approaches that use neural networks because they are the state-of-the art. most previous works that use them for word alignment @xcite @xcite achieved accuracies that are basically comparable to giza++. however, the accuracy of recent works @xcite @xcite based on the transformer @xcite , which is the state-of-the art neural machine translation model, have started to outperform giza++. @xcite made the attention of the transformer more closely resembled the word alignment, and achieved better accuracy than giza++ when they used alignments obtained from it for supervision. @xcite added an alignment layer using a full target context on top of the transformer and trained it with a loss function that encouraged contiguous alignment and bidirectional agreement. they outperformed giza++ without giza++ output for supervision. @xcite proposed a supervised word alignment method using the hidden states of the transformer, and significantly outperformed fastalign (11-27 f1 points) using a small number of gold word alignments (1.7k-5k sentences). however, both @xcite and @xcite required more than a million parallel sentences to pretrain their models. applying these methods to low-resource language pairs and domains is difficult. in this paper, we present a novel supervised word alignment method that requires no parallel sentences for pretraining and can be trained from fewer gold word alignments (150-300 sentences). it formalizes word alignment as a collection of squad-style span prediction problems @xcite and solves them with multilingual bert @xcite . we experimentally show that our proposed model significantly outperformed both @xcite and @xcite . our main contribution is that we make supervised word alignment more practical. @xcite argued that supervised word alignment is a viable option. they concluded that alignment annotation could be performed rapidly 4.4 sentences per minute by annotators with minimal experience using a web-based crowd-sourcing interface. assuming that a small amount of gold word alignment data, which can be annotated in a couple of hours, our proposed method could be used on 104 languages supported by the multilingual bert. ## analysis 4.1 symmetrization heuristics to show the effectiveness of our proposed symmetrization heuristics (bidi-avg), the accuracies are greatly affected by the orthography of the target language. for languages whose words are not delimited by white spaces, such as chinese and japanese, the span prediction accuracy "to english" is significantly higher than that of "from english". in this case, grow-diagfinal outperforms bidi-avg. by contract, for languages with spaces between words, such as german, romanian, and french, no significant differences exist between the "to english" and "from english" accuracies. in this case, bidi-avg is better than grow-diag-final. in en-fr, intersection achieves the best accuracy, probably because the dataset is very noisy. since the proposed bidi-avg works relatively well for all cases, we used the heuristics as the default symmetrization method in our experiments. ## conclusion we presented a novel supervised word alignment method using the multilingual bert, which requires as few as 300 training sentences to outperform previous supervised and unsupervised methods. we made supervised word alignment practical because our method does not require any bitexts for pretraining, and it can be fine-tuned to a specific guideline using fewer gold word alignments. future works include using other multilingual pretraining models such as xlm-roberta @xcite for a more accurate model and distilmbert @xcite for a more compact model. one significant merit of framing word alignment as a squad-style span prediction task is that we can easily import the progress of the latest question answering and multilingual language modeling technologies. our cross-language span prediction method can be used for any alignments between two sequences. we have already applied it to bilingual sentence alignment @xcite and we plan to extend it to other related problems.