Language Tokens: A Frustratingly Simple Approach Improves Zero-Shot Performance of Multilingual Translation (2022)

Abstract: This paper proposes a simple yet effective method to improve direct (X-to-Y) translation for both cases: zero-shot and when direct data is available. We modify the input tokens at both the encoder and decoder to include signals for the source and target languages. We show a performance gain when training from scratch, or finetuning a pretrained model with the pro- posed setup. In the experiments, our method shows nearly 10.0 BLEU points gain on in-house datasets depending on the checkpoint selection criteria. In a WMT evaluation campaign, From- English performance improves by 4.17 and 2.87 BLEU points, in the zero-shot setting, and when direct data is available for training, respectively. While X-to-Y improves by 1.29 BLEU over the zero-shot baseline, and 0.44 over the many-to-many baseline. In the low-resource setting, we see a 1.5 ∼ 1.7 point improvement when finetuning on X-to-Y domain data.

## introduction neural machine translation (nmt) has witnessed significant advances since the introduction of the transformer model @xcite . this model has shown impressive performance for bilingual translation commonly from and to english @xcite . it has also been shown that the proposed model could be easily extended to multiple language pairs @xcite @xcite , to and/or from english, by simple modifications to the basic architecture. this holds promise for improved performance for low-resource pairs through transfer learning, as well as better training and deployment costs per language pair. this setting is referred to as multilingual neural machine translation (mnmt). the mainstream method of training mnmt is to introduce an additional input tag at the encoder to indicate the target language, while the decoder uses the usual begin-of-sentence ( bos ) token. this simple modification to the bilingual architecture is shown to work well up to hundreds of language pairs @xcite , given a corresponding increase in the number of parameters to handle the increased training data. despite the emergence of modified architectures which add language-specific parameters, like language specific subnetworks (lass) @xcite , and adapters @xcite , the basic architecture remains the most effective choice for deploying large scale production systems. ## motivation while mnmt was originally focused on english-centric translation, there is increasing interest interest in direct translation 1 rather than pivoting through a common language (ex. english). in @xcite , the authors mine direct translation data by matching the english part of english-centric corpora, then use modified temperature sampling to alleviate the overrepresentation of english as target. another work @xcite leverages public direct translation data including ccmatrix @xcite and ccaligned @xcite as well as improved sampling and sparse modeling to develop a 100-by-100 direct translation model. while the availability of direct translation training data helps improve the corresponding directions, the zero-shot case remains of particular importance considering the difficulty in maintaining coverage of the set of rapidly increasing directions, and handling the corresponding increase in data size and compute time and resources. hence the interest in techniques that improve zero-shot translation performance, benefit from parallel training data as it becomes available, and which can be easily applied to the basic architecture and pretrained models. related works. @xcite approaches the off-target translation problem using gradient projection and no direct training data. zhang, williams, titov, and sennrich (2020) improves the zero-shot case using online back-translation and by specializing layers (ex. layer-norm) for the target language. rios, müller, and sennrich (2020) utilizes separately-trained vocabularies per language. @xcite proposes an alignment loss to enforce source language invariance. @xcite proposes tagging input tokens by the source language and indicating the target language directly to a shared decoder. proposal. we propose a simple yet effective method that improves the performance of direct translation: the input tokens used in mnmt are changed to st-t instead of t-b (see figure 1 ). the encoder takes tokens for both the source and target languages ( s,t ) while the decoder takes one for only the target language ( t ). it is shown that using these modified tokens significantly improves the performance on direct translation pairs without any parallel x ⇔y translation data-training only on english-centric data (e ⇔x ). remarkably, these gains are quickly obtained if we start from a model trained using the baseline tokens and continue training after adding the new tokens. in subsequent experiments, we also show that some gains are still observed if we continue training the baseline model using a mix of direct (x ⇔y ) and englishcentric training data-suggesting the method extends to the non zero-shot case as well. the paper is organized as follows: we describe the proposed method in section 3. this is followed by describing the data and the models used in our experiments in section 4 and section 5 respectively. section 6 gives the experimental results (domain finetuning results in section 6.2). finally, section 7 gives the conclusion. 1 also known as x-y translation. in the rest of the paper we refer to translation between any two language pairs not involving english as direct or x-y translation. ## approach in basic mnmt, the source sentence is followed by a token indicating the target language at the encoder side, and with the begin-of-sentence token at the decoder side. this setup is t-b in figure 1 . in the proposed method, we perform a simple modification by adding both the source and target tokens to the input at the encoder side foot_0 , and the target at the decoder side. this setup is st-t in figure 1 . y. @xcite shows that adding the tgt language to the decoder input helps english-to-x translation. in a recent submission to wmt21, @xcite uses a src token at the encoder and a tgt token at the decoder, which can be observed from the public evaluation code foot_1 . in initial experiments we try several variants of indicating the languages to the model. we find that most are similar for the english-centric case, but the proposed method ( st-t ) performs the best for zero-shot direct translation. ## data in initial experiments we build a model for 10 european languages using in-house data (section 4.1). follow-up experiments use wmt data @xcite as well as other publicly available data covering 6 languages (section 4.2). most experiments use englishcentric training data, some use direct training data (section 4.3), and some use domain data (section 4.5). validation data is described in section 4.4. tables are in appendix a. ## models all experiments use the same architecture and configuration. we use the transformer encoderdecoder architecture @xcite as the base model and opt for a deep encoder and a shallower decoder as presented in @xcite and @xcite , with 24 encoder layers and 12 decoder layers. dimensions are 1024 for model width, 4096 for the feed-forward hidden layer, and 16 attention heads. we use pre-layer normalization which is becoming more common for similar architectures @xcite . we use a vocabulary of size 128, 000 with the sentencepiece tokenizer 7 . the model size is 0.6b parameters. all models are trained by the radam optimizer @xcite . see appendix a, ## experimental results in this section, we show experimental results using the wmt model as described in section 5. we first continue training the wmt model using the proposed tokens on english-centric data only, then we continue training using direct data 8 . in other experiments, we use x ⇔y data from start, once with each of t-b and st-t tokens. in all cases, we report the bleu score for both the direct and english-centric dev sets. see ## conclusion this paper proposes a simple and effective method to improve direct translation for both the zero-shot case and when direct data is available. the input tokens used for mnmt are changed from t-b (@xmath0. moreover, the performance of the new tokens can be readily obtained if we continue training the baseline model with the new tokens but the same training data. for a wmt-based setting, we see around 1.3 bleu points improvement for zero-shot direct translation and around 0.4 bleu point improvement when using direct data for training. in both cases the english-centric performance is also improved-by as much as 3.97 in wmt21 for one setup. comet scores see noticeable bumps as well-by 2.56 points. on x ⇔y dev, and 15.23 points on e ⇒ x dev. on another front, the proposed tokens are effective when finetuning a general model for direct translation using domain data. for three tested domains and two translation directions, we see significant improvements over the baseline. results for eu10 (section 3.1) suggest a stronger potential given more similar language and domain sets.