Revamping Multilingual Agreement Bidirectionally via Switched Back-translation for Multilingual Neural Machine Translation (2024)

Abstract: Despite the fact that multilingual agreement (MA) has shown its importance for multilingual neural machine translation (MNMT), current methodologies in the field have two shortages: (i) require parallel data between multiple language pairs, which is not always realistic and (ii) optimize the agreement in an ambiguous direction, which hampers the translation performance. We present B idirectional M ultilingual A greement via S witched B ack- t ranslation ( BMA-SBT ), a novel and universal multilingual agreement framework for fine-tuning pre-trained MNMT models, which (i) exempts the need for aforementioned parallel data by using a novel method called switched BT that creates synthetic text written in another source language using the translation target and (ii) optimizes the agreement bidirectionally with the Kullback-Leibler Divergence loss. Experiments indicate that BMA-SBT clearly improves the strong baselines on the task of MNMT with three benchmarks: TED Talks, News, and Europarl. In-depth analyzes indicate that BMA-SBT brings additive improvements to the conventional BT method.

## introduction conventional multilingual neural machine translation (mnmt) leverages independent parallel data during the training process. in comparison, the multilingual agreement (ma) explicitly minimizes the output difference between two source inputs written in different languages but with the same meaning. despite its success in from-scratch training on mt @xcite , current methodologies suffer from at least two disadvantages that limit their scope of usage. firstly, conventional figure 1: an illustrated example that can be benefited from multilingual agreement optimized in a bidirectional manner. the words in green are the correct translation, and the words in red are the wrong translation. here, chinese is incorrectly translated since it does not have past tense for verbs, and german is incorrectly translated due to the shared subword unit with different meanings between glanz (german, shine) and glance (english, take a brief look at). best viewed in colour. ma leverages word alignment tools to create codeswitching sentence-level data @xcite . this process usually requires authentic parallel data between multiple language pairs. for example, assuming we would like to enhance chinese to english and german to english, conventional ma assumes the existence of parallel data from chinese to german, which however sometimes does not exist. secondly, the direction of agreement-based learning can be bidirectional @xcite , while the direction of conventional multilingual agreement is usually ambiguous. however, since languages usually have different linguistic clues and they are helpful to each other, we argue that optimizing the multilingual agreement explicitly in a bidirectional manner can help the languages to learn from each other and hence further enhance cross-lingual learning. figure 1 depicts such a case that can be benefited from bidirectionally enhanced ma. the underlying reason is that both of the source inputs have crosslingual ambiguities here. since chinese does not have past tense verbs, it is intuitive to use some auxiliary languages with past tense. furthermore, since german shares partial vocabulary subwords with english under mnmt, this introduces crosslingual ambiguities and using a language that does not share its vocabulary subwords with english, e.g., such as chinese, could be helpful. as a side note, since ma was proposed as a method for from-scratch training for mt, it was unclear whether conventional ma is also effective as a fine-tuning technique for pre-trained models. furthermore, how to appropriately apply backtranslation to a multilingual setting is also an understudied subject despite its importance. this paper proposes bma-sbt, a novel mnmt framework that (i) exempts the need for parallel data between multiple language pairs and (ii) optimizes the ma in a bidirectional manner. to exempt the need for parallel data, we propose switched back-translation to produce synthetic text in some different auxiliary source languages with the translation target. 1 to optimize the ma in an explicit bidirectional manner, we use a bidirectional kullback-leibler divergence loss instead of the codeswitching for conventional ma. this enforces the original source language and the synthetic auxiliary language to have the same outputs as the target reference translation in a bidirectional manner. we conduct experiments on three mt benchmarks: ted talks @xcite , news benchmark (news-commentary) and europarl @xcite . experimental results indicate that bma-sbt clearly improves the strong pre-trained baselines on all three benchmarks. in-depth analyses indicate that bma-sbt effectively mitigates cross-lingual ambiguities. in summary, we make three key contributions: results indicate that bma-sbt brings additive improvement to conventional bt and bidirectionality is important for ma. also, this is the first work that demonstrates the usefulness of ma as a fine-tuning technique. ## conclusions despite the fact that multilingual agreement (ma) has shown its effectiveness in from-scratch training for mnmt, the conventional ma has at least two shortages that limit its usages: (i) needs authentic extra parallel data, which can be often unrealistic and (ii) has an ambiguous direction for agreementbased learning. we propose bma-sbt as a novel and universal fine-tuning framework for pre-trained mt models that (i) exempts the need for authentic parallel data by creating synthetic parallel text written in a different source language and (ii) specifies the direction of agreement-based learning with bidirectional kl divergence loss. experimental results on three multilingual machine translation datasets illustrate that bma-sbt can obviously improve the strong pre-trained baseline system. an in-depth investigation indicates that bma-sbt brings additive improvements to the conventional bt methods for neural machine translation.