Pre-training Synthetic Cross-lingual Decoder for Multilingual Samples Adaptation in E -Commerce Neural Machine Translation (2022)

Abstract: Availability of the user reviews in vernacular languages is helpful for the users to get information regarding the products. Since most of the e-commerce websites allow the reviews in English language only, it is important to provide the translated versions of the reviews to the non-English speaking users. Translation of the user reviews from English to vernacular languages is a challenging task, predominantly due to the lack of sufficient in-domain datasets. In this paper, we present a pre-training based efficient technique which is used to adapt and improve the single multilingual neural machine translation (NMT) model for the low-resource language pairs. The pre-trained model contains a special synthetic cross-lingual decoder. The decoder for the pre-training is trained over the cross-lingual target samples where the phrases are replaced with their translated counterparts. After pre-training, the model is adapted to multiple samples of the low-resource language pairs using incremental learning that does not require full training from the very scratch. We perform the experiments over eight low-resource and three high resource language pairs from the generic domain, and two language pairs from the product review domains. Through our synthetic multilingual decoder based pre-training, we achieve improvements of upto 4.35 BLEU points compared to the baseline and 2.13 BLEU points compared to the previous code-switched pre-trained models. The review domain outputs from the proposed model are evaluated in real time by human evaluators in the e-commerce company Flipkart.

## introduction neural machine translation models @xcite are effective for a specific language pair or domain when trained on a large amount of parallel corpus. it is often difficult to obtain such a large corpus, especially in non-english languages and in specialized domains such as product reviews @xcite . currently, in the e-commerce domain, providing the translation of the user reviews in vernacular languages is a need. in a multilingual country like india where english is not a primary language, reviews in local languages will be very helpful for the users as well as e-commerce platforms like flipkart. as of december 2021, flipkart leads 1 in the indian e-commerce market with a market share of 31.9%. in the process of building a oneto-many multilingual translation system to translate the low-resource review domain data on the ecommerce platform flipkart from english to multiple indian languages, we propose a synthetic decoder based pre-training approach. to see the impact of the proposed model on translation quality, we perform experiments over the general domain data available publicly. along with it, we also evaluate our model for review domain data using english-hindi, english-french and english-tamil testset. recently, pre-training based nmt @xcite models have attracted attention to improve the translation quality of low as well as high resource language pairs @xcite . pre-training based models first train a parent model over a large dataset and then use the learnt weights to fine-tune for a spe-cific low-resource language pair or domain @xcite . these approaches have some limitations, e.g. these use some special symbols in the parent models which may not be present in the data during the training of child model. as the samples are taken from the same languages, these approaches fail to capture the cross-lingual information in two languages @xcite . fine-tuning also has a limitation that it is not able to remember the information of the parent model's language pairs while training over the child model (new language pair or domain). to resolve this, source side code-switching is used to generate synthetic parallel samples to train the parent model and later use it for finetuning over new language pair @xcite . these approaches use the parent model's weights to fine-tune for a bi-lingual translation task. in our work, we perform random phrase substitution at the target side of a parallel sample to capture the shared target context. our final trained model is a multilingual translation model which can translate the source sentence into multiple languages. multilingual adaptation helps the incoming pairs to learn from each other because of the shared parameter training. also, unlike @xcite @xcite , we use incremental learning instead of fine-tuning where the model can adapt over the incoming input samples from different language pairs without forgetting the information of previously adopted language pairs. incremental learning allows to update the model even with a small size of available parallel samples without full retraining. ## related work pre-training a nmt model and fine-tuning it for specific translation tasks is one of the popular approaches for dealing with the resource-scarce language scenario. pre-trained language models (lms) like bert (bidirectional encoder representations from transformers) @xcite have been used to improve the nmt models @xcite . @xcite introduced elmo to the encoder of the nmt model as a pre-trained lm to improve the performance of the nmt model. @xcite used bi-directional self-attention lm in the nmt by weighted-fusion mechanism and knowl-edge transfer paradigm to improve the learning of encoder and decoder. @xcite incorporated the representations from the bert into the encoder and decoder layers of the nmt model. but such large parameters also added extra overhead and delay in the inference time. the recent studies of @xcite and @xcite used code-switching at source side to train the parent model. the trained parent model is used for fine-tuning over the specific bi-lingual translation direction. yang et al. (2020b); lin et al. (2020) trained a multilingual parent model. unsupervised pre-training has also been popular in several natural language understanding problems, such as word embedding representation @xcite , pre-trained context representation @xcite and sequence-to-sequence pre-training @xcite . in this pretraining, scale of data is found to be a very important attribute. multilingual nmt @xcite @xcite @xcite is also a useful paradigm where a model trained in a parameter sharing fashion shares the information among the language pairs. in multilingual nmt, low-resource language pairs leverage the information of the high-resource languages. @xcite added language specific tags before each source sentence in the parallel corpus, merged all the data from multiple language pairs and trained them in a single nmt model. @xcite used shared attention to transfer information between multiple encoderdecoders in a multilingual nmt. @xcite performed the training of massively multilingual nmt models. they showed that training a many-to-many multilingual model is helpful in low-resource scenarios. by keeping this in mind, we also use pre-training to improve a multilingual nmt. unlike yang et al. (2020b); lin et al. (2020) , we use the pre-trained nmt model to adapt over multilingual nmt using incremental learning instead of bi-lingual pair using fine-tuning. ## dataset we need two types of corpora i.e. parallel and monolingual. for the experiments, we include a total of 11 language pairs; out of which 3 belong to the european language pairs, and the rest 8 are low-resource english-indian language pairs. the data statistics are shown in ## methodology our proposed approach has four modules: i. training cross-lingual word mapping, ii. generation of synthetic phrase table for source-target phrase pairs, iii. generation of synthetic cross-lingual target samples and training the parent model and iv. adapting new input samples from multiple language pairs using incremental learning. ## experimental setting parent model is trained using the transformer @xcite based encoder-decoder nmt model. our training setup is described as follows: the tokens of training, evaluation and validation sets are segmented into the subword units using the bpe technique @xcite proposed by @xcite . we perform 30,000 and 10,000 join operations for high and low-resource languages, respectively. we learn the bpe vocabulary using the monolingual data and apply it over the parallel samples. we use 6 layers at encoder and decoder sides each, 8-head attention, hidden layer of size 512, embedding vector of size 512, learning rate of 0.0002, and the minimum batch size of 3800 tokens. the evaluation results are based on the bleu metric @xcite . the new samples from the low-resource child pairs are tokenized and true-cased. here, we also apply the subword operation using the learned vocabulary from the monolingual data as mentioned above. here, before adding the new parallel samples to the parent models using incremental learning, we add language specific tags before the source sentence of each parallel sample. adding a tag before the sample @xcite is for differentiating between parent samples and new incoming samples as well as highlighting the difference between the parallel samples from different languages too. for example, we append ##hi before source sentence of each english-hindi parallel sample. similar to this, we use the tags like ##fr, ##de, ##es, ##bn, ##gu, ##mr, ##bn, ##gu, ##mr, ##pa, ##ml, ##ta and ##te for french, german, spanish, bengali, gujarati, marathi, punjabi, malayalam, tamil and telugu languages, respectively. ## results and analysis we evaluate our proposed models and compare with the multilingual models for indian languages as the baseline. we also compare our method with existing two pre-trained models, i.e. csp @xcite and mrasp @xcite . for the low-resource indian languages, we fine tune csp and mrasp models for multilingual child model. for the experiments over indian languages using wat21 dataset, we augmented it with european languages dataset. we report the evaluation results of both word based and phrase based models. from to see the impact of synthetic data used to train the parent model, we also perform the experiments by training the parent model over multiple fractions of synthetic data samples. we split the parent data in 30%, 50% and 100% of total size. in ## conclusion in this paper, we have devised a pre-training based learning where the parent model is trained on the source and cross-lingual target samples. we pre-train a one-to-many multilingual parent model with synthetic decoder and use incremental learning to adapt over new incoming bi-lingual parallel samples from multiple language pairs. our objective to train such a pre-training model is to capture a cross-lingual context at the target side and use it to adapt the new multilingual parallel samples from the low-resource language pairs. we have performed experiments over 8 lowresource and 3 high-resource language pairs. we also perform experiments over two product review domain datasets from english-french and english-hindi language pairs. through our synthetic multilingual decoder based pre-training, we achieve upto 3.22 and 4.35 bleu points improvements for high and low-resource language pairs, respectively over the baseline. from the perspective of the e-commerce platforms, our proposed parent model is able to adapt new samples for multiple language pairs and provide us a single translation model which can translate the english sentence into multiple languages. the proposed model is evaluated by real time evaluators at flipkart for english-to-tamil and english-to-hindi review domain testsets. the human evaluation results show the increment of upto 6% output samples with the good label. in the future, we aim to utilize language relatedness in the multilingual setting. we believe that language relatedness in terms of vocabulary overlap, syntax sharing and subword learning can help to improve the translation quality in a multilingual model.