Improving Neural Machine Translation by Bidirectional Training (2021)

Abstract: We present a simple and effective pretraining strategy – bidirectional training (BiT) for neural machine translation. Specifically, we bidirectionally update the model parameters at the early stage and then tune the model normally. To achieve bidirectional updating, we simply reconstruct the training samples from “src \rightarrow tgt” to “src+tgt \rightarrow tgt+src” without any complicated model modifications. Notably, our approach does not increase any parameters or training steps, requiring the parallel data merely. Experimental results show that BiT pushes the SOTA neural machine translation performance across 15 translation tasks on 8 language pairs (data sizes range from 160K to 38M) significantly higher. Encouragingly, our proposed model can complement existing data manipulation strategies, i.e. back translation, data distillation, and data diversification. Extensive analyses show that our approach functions as a novel bilingual code-switcher, obtaining better bilingual alignment.

## introduction recent years have seen a surge of interest in neural machine translation @xcite @xcite where it benefits from a massive amount of training data. but obtaining such large amounts of parallel data is not-trivial in most machine translation scenarios. for example, there are many lowresource language pairs (e.g. english-to-tamil), which lack adequate parallel data for training. although many approaches about fully exploiting the parallel and monolingual data are proposed, e.g. back translation @xcite , knowledge distillation @xcite and data diversification @xcite , the prerequisite of these approaches is to build a wellperformed baseline model based on the parallel data. however, @xcite ; @xcite ; @xcite em-pirically reveal that nmt runs worse than their statistical or even unsupervised counterparts in lowresource conditions. here naturally arise a question: can we find a strategy to consistently improve nmt performance given the parallel data merely? we decide to find a solution from human learning behavior. @xcite ; @xcite ; @xcite show that bidirectional language learning helps master bilingualism. in the context of machine translation, both the @xmath0, which motivates many recent studies, e.g. dual learning @xcite and symmetric training @xcite . however, their approaches rely on external resources (e.g. word alignment or monolingual data) or complicated model modifications, which limit the applicability of the method to a broader range of languages and model structures. accordingly, we turn to propose a simple data manipulation strategy and transfer the bidirectional relationship through bidirectional training ( §2.2). the core idea is using a bidirectional system as an initialization for a unidirectional system. specifically, to make the most of the parallel data, we first reconstruct the training samples from " -→ b : @xmath1: source+@xmath2, where the training data was doubled. then we update the model parameters with ← → b in the early stage, and tune the model with normal " -→ b @xmath3. we validated our approach on several benchmarks across different language families and data sizes, including iwslt21 en↔de, wmt16 en↔ro, wmt19 en↔gu, iwslt21 en↔sw, wmt14 en↔de, wmt19 en↔de, wmt17 zh↔en and wat17 ja↔en. experimental results show that the proposed bidirectional training (bit) consistently and significantly improves the translation performance over the strong transformer @xcite . also, we show that bit can complement existing data manipulation strategies, i.e. back translation, knowledge distillation and data diversification. extensive analyses in §3.3 confirm that the performance improvement indeed comes from the better cross-lingual modeling and our method works like a novel code-switching method. ## conclusion and future works in this study, we propose a pretraining strategy for nmt with parallel data merely. experiments show that our approach significantly improves translation performance, and can complement existing data manipulation strategies. extensive analyses reveal that our method can be viewed as a simple yet better bilingual code-switching approach, and improves bilingual alignment quality. encouragingly, with bit, our system @xcite got the first place in terms of bleu scores in iwslt2021 foot_7 low-resource track. it will be interesting to integrate bit into our previous systems @xcite and validate its effectiveness on industrial level competitions, e.g. wmt foot_8 . it is also worthwhile to explore the effectiveness of our proposed bidirectional pretraining strategy on multilingual nmt task @xcite .