Leveraging Highly Accurate Word Alignment for Low Resource Translation by Pretrained Multilingual Model (2023)

Abstract: Recently, there has been a growing interest in pretraining models in the field of natural language processing. As opposed to training models from scratch, pretrained models have been shown to produce superior results in low-resource translation tasks. In this paper, we introduced the use of pretrained seq2seq models for preordering and translation tasks. We utilized manual word alignment data and mBERT-based generated word alignment data for training preordering and compared the effectiveness of various types of mT5 and mBART models for preordering. For the translation task, we chose mBART as our baseline model and evaluated several input manners. Our approach was evaluated on the Asian Language Treebank dataset, consisting of 20,000 parallel data in Japanese, English and Hindi, where Japanese is either on the source or target side. We also used in-house 3,000 parallel data in Chinese and Japanese. The results indicated that mT5-large trained with manual word alignment achieved a preordering performance exceeding 0.9 RIBES score on Ja-En and Ja-Zh pairs. Moreover, our proposed approach significantly outperformed the baseline model in most translation directions of Ja-En, Ja-Zh, and Ja-Hi pairs in at least one of BLEU/COMET scores.

## introduction in recent years, there has been a growing body of research on sequence-to-sequence (seq2seq) models that are based on pretraining @xcite @xcite . since the introduction of the transformer architecture @xcite , the quality of machine translation has greatly improved. however, when it comes to low-resource translation tasks, the performance of this type of parameter randomization model often suffers due to the limited size of available datasets @xcite @xcite . to address this challenge, many researchers have proposed using unsupervised methods, such as mapping monolingual vector embeddings to a common cross-lingual embedding space @xcite , or leveraging large-scale pretraining models that have been successfully applied to various nlp tasks @xcite . in this paper, we propose applying a pretrained seq2seq model for preordering and translation tasks. specifically, we investigate different sizes of mt5s @xcite and mbart @xcite , in order to evaluate their performance on preordering when using manual word alignment data. for the translation process, we choose mbart as our baseline model, and we evaluate the translation results using both the original sequence and the generated preordering sequence as input. our approach was evaluated on the asian language treebank dataset @xcite , consisting of 20,000 parallel data in japanese, english and hindi, where japanese is either on the source or target side. to compare the effects on different datasets, we also used the in-house data which comprised 3,000 parallel data in chinese and japanese. the results indicated that mt5-large trained with manual word alignment achieved a preordering performance exceeding 0.9 when evaluated using the ribes score on ja-en and ja-zh pairs. moreover, our proposed approach significantly outperformed the baseline model in most of the translation directions of language pairs of ja-en, ja-zh, and ja-hi in terms of at least one of the bleu @xcite and comet (wmt20-comet-da) @xcite scores. ## related work in recent years, researchers have conducted more and more studies on seq2seq models based on pretraining. while learning the rules of sequence generation remains the most crucial feature of these models, some studies have explored the application of preordering to training, resulting in improved results. @xcite discussed the importance of maintaining consistency between input source word order and output target word order for improved translation accuracy in neural machine translation (nmt) models. @xcite proposed a transfer learning approach for nmt that trains the model on an assisting source-target language pair and improves translation quality in extremely low-resource scenarios. however, both methods rely on separately pretraining a translation model using a large-scale parallel corpus and handle preordering based on the syntax tree. in contrast, @xcite proposed a framework for low-resource translation that focuses on preordering and highly accurate word alignment using an smt model. their solution outperformed the transformer model, but they did not explore the use of large-scale pretrained seq2seq models. our work focused on low-resource translation tasks and utilizes large-scale pretrained multilingual models for fine-tuning the preordering and translation procedures. ## seq2seq models in general, seq2seq models take a sequence of tokens as input from the source sequence @xmath0, s 2 , . . . , s k and produce a sequence of tokens as output for the target sequence @xmath1, t 2 , . . . , t m , where s i (@xmath2, . . . , k) and t j (@xmath3, . . . , m) represent the tokens in the source and target sequences, respectively. in terms of structure, seq2seq models consist of an encoder and a decoder @xcite . the encoder converts the input sequence into a high-dimensional vector representation, while the decoder maps the high-dimensional vectors to the output dictionary based on the encoder's output. this framework has been applied to various tasks, including machine summarization @xcite , question-answering systems @xcite , and machine translation @xcite . since seq2seq models can learn the rules governing the input and output sequences, we aim to use them for preordering and translation.