Enhancing Low-Resource NMT with a Multilingual Encoder and Knowledge Distillation: A Case Study (2024)

Abstract: Neural Machine Translation (NMT) remains a formidable challenge, especially when dealing with low-resource languages. Pre-trained sequence-to-sequence (seq2seq) multi-lingual models, such as mBART-50, have demonstrated impressive performance in various low-resource NMT tasks. However, their pre-training has been confined to 50 languages, leaving out support for numerous low-resource languages, particularly those spoken in the Indian subcontinent. Expanding mBART-50’s language support requires complex pre-training, risking performance decline due to catastrophic forgetting. Considering these expanding challenges, this paper explores a framework that leverages the benefits of a pre-trained language model along with knowledge distillation in a seq2seq architecture to facilitate translation for low-resource languages, including those not covered by mBART-50. The proposed framework employs a multilingual encoder-based seq2seq model as the foundational architecture and subsequently uses complementary knowledge distillation techniques to mitigate the impact of imbalanced training. Our framework is evaluated on three low-resource Indic languages in four Indic-to-Indic directions, yielding significant BLEU-4 and chrF improvements over baselines. Further, we conduct human evaluation to confirm effectiveness of our approach. Our code is publicly available at https://github.com/raypretam/Two-step-low-res-NMT.

## introduction neural machine translation (nmt) models @xcite @xcite have shown impressive results on benchmark datasets, mainly containing large amounts of parallel data. however, these models face challenges when applied to low-resource languages or languages with rich and diverse mor-phology. previous approaches have leveraged pretrained models trained on extensive corpora @xcite @xcite @xcite @xcite to address these limitations. pre-trained multilingual seq2seq-based models based on an encoder-decoder framework such as mbart-50 @xcite have been successfully used for various low-resource nmt tasks. despite being pre-trained with 50 languages, it needs more support for numerous low-resource languages. expanding the capabilities of mbart-50 to encompass new languages entails a cumbersome process involving the collection of substantial amounts of monolingual data and the execution of pre-training with denoising objectives after initializing mbart-50. this process is time-consuming and may decrease performance on the initial 50 languages when incorporating new ones, a phenomenon known as catastrophic forgetting @xcite . in contrast, encoder-based pretrained model xlm-r @xcite is designed to accommodate 100 languages, making it suitable for a wide range of low-resource cross-lingual natural language understanding (nlu) tasks. both crosslingual and machine translation (mt) functionalities share certain similarities. in cross-lingual scenarios, training and evaluation occur across different languages, while mt systems process input in one language and produce output in another. this distinction prompts several experimental research questions, including: 1) how does an xlm-r based nmt model perform on low-resource morphologically rich languages, particularly those not covered by mbart-50? 2) given that low-resource nmt may be affected by training imbalances leading to performance degradation, can the application of knowledge distillation further enhance the results? to address the two aforementioned experimen-tal research questions, we utilize our base model, which follows a seq2seq framework. here, we initialize the encoder with the multilingual pretrained model xlm-r large, while decoder layers are initialized from scratch, we call this base approach as xlm-mt. similar frameworks have been explored in previous studies @xcite , with our approach sharing similarities with @xcite , who employed it for zero-shot crosslingual nmt tasks and froze the embedding layers. however, our base approach differs in considering only decoder training. thereafter, we apply complementary knowledge distillation (ckd) @xcite to the base xlm-mt model to address training imbalances. the objective of this complementary knowledge distillation is to train the student model with knowledge which complements the teacher model and avoid knowledge forgetting, and we refer to this as xlm-mt+ckd. we empirically evaluate our model across three indic languages and observe significant improvement in bleu and chrf scores. finally, we use human evaluation to assess the fluency, relatedness, and correctness of our output. our contributions are as follows: ## methodology given a source language sentence @xmath0, x 2 , . . . , x s ), and its corresponding target language translation @xmath1, y 2 , . . . , y t ), an nmt model is trained to predict the translated sequence y ′ using the maximum log-likelihood estimation (mle) objective. the probability of predicting the target sequence y ′ is computed as p(y ′ |x; θ) = t @xmath2:t-1 , x 1:s , θ), where θ represents the model parameters. ## experimental setup dataset: for our experiments, we specifically select three indic languages, namely kannada, and punjabi that are not included in mbart-50, to assess the effectiveness of our approach. we use the samanantar dataset @xcite for training all our nmt models which contains parallel sentences for 11 indic language pairs. we consider three languages in 4 directions, namely hindi-kannada, kannada-hindi, kannada-punjabi, and punjabi-kannada, containing 2.1 million and 1.1 million parallel sentences respectively. we use the flores-200 (team, 2022) containing 997 and 1012 sentences as our validation and test set respectively. implementation details: we implement our approach using the fairseq toolkit @xcite . we use adam optimizer (kingma and ba, 2017) with β 1 = 0.9 and β = 0.98. following the work by @xcite , we use learning rates 5e -3 and 1e -3 for the base model and ckd, respectively. we set maximum updates of 200k for the base model training and 40k for the ckd. we use 12 layers with 16 attention heads in the decoder. we use the 'large' variant of xlm-r that has 550 million parameters for our experiments. we set the number of teachers to 1 and α = 0.95. we set batch @xmath3, and used beam @xmath4, and following @xcite we averaged the last five checkpoints. we use bleu-4 @xcite and chrf @xcite score to evaluate our approach. all the models have been trained on single a100 gpus. none of the training methods consumed more than 96 hours. baselines we employ various baseline models for comparison with our approach. to ensure a fair assessment, we train all baseline models using identical training data and assess their performance on the flores dataset. transformer @xcite selective kd @xcite refers to the process of distilling and transferring specific, relevant knowledge from a teacher model to a student model. instead of transferring all the knowledge indiscriminately, this approach involves selecting and distilling the most valuable and informative aspects of the teacher model's knowledge. mbert-mt @xcite , integrates bert into the nmt process. initially, bert is employed to extract representations for an input sequence. subsequently, these representations are fused with each layer of the nmt model's encoder and decoder using attention mechanisms. sixtp @xcite is a sequence-to-sequence (seq-to-seq) model. in its initialization, the encoders are initialized with the xlm-r large model, while the decoder is initialized randomly. the model undergoes a two-stage fine-tuning process. in the initial stage, the encoder layers are frozen, and fine-tuning is performed on the decoders. subsequently, in the second stage, the model is trained in an end-to-end fashion. ## results we also investigate various variants of our model to validate the effectiveness of our architecture and present results in ## analysis how would the method perform with the languages that mbart-50 supported? in addition to the language pairs outlined in section 3, we extend our exploration to include language pairs supported by mbart-50, facilitating effec- ## human evaluation the experts' responses for each parameter. the annotation experts received compensation according to industry standards for their work. we briefed them on the objectives and explicit usage of their annotations 7 case study ## related work neural machine translation (nmt) aims to translate a given source sentence into a target sentence. typically, an nmt model comprises an encoder, a decoder, and an attention mechanism. the encoder transforms the input sequence into hidden representations while the decoder maps these representations to the target sequence. the attention mechanism, pioneered by @xcite , enhances alignment between words in the source and target languages. different architectures can be employed for the encoder and decoder, including lstm (long short-term memory), cnn (convolutional neural network), and transformer. the transformer architecture, introduced by @xcite , consists of three sublayers. transformer has demonstrated state-of-the-art performance in nmt tasks @xcite . prior studies @xcite @xcite @xcite have investigated the integration of pre-trained language encoders into nmt models to bolster supervised translation performance. @xcite introduce a bert-fused model that extracts representations from input sentences and integrates them into the encoder and decoder using attention mechanisms. recent research @xcite focuses on developing and refining encoderdecoder-based multilingual trained language models for nmt. @xcite present mbart, a transformer-based encoder-decoder model explicitly tailored for nmt applications. wei et al. finetune the multilingual encoder-based model for low-resource nmt, and they focus on improving the mpe for a more universal representation across languages. @xcite have examined a two-stage framework utilizing an encoder-based multilingual language model for zero-shot neural machine translation. numerous studies in nmt have incorporated the knowledge distillation (kd) framework. @xcite introduced word-level kd for nmt and later proposed sequence-level kd to enhance overall performance. investigating the efficacy of various token types in kd, @xcite suggested strategies for selective kd. @xcite successfully transferred internal hidden states from teacher models to students, achieving positive results. various kd approaches have also been employed in non-auto-regressive machine translation tasks to enhance outcomes. @xcite improved non-autoregressive model performance by distilling information from an autoregressive model. @xcite conducted systematic experiments highlighting the importance of knowledge distillation in training non-auto-1. source (kannada): udaharanage, obbaru, motaru karugale rastegala abhivrd'dhige mula karana endu helabahudu. translation: for example, one could say that motor cars were the root cause of the development of roads. reference (punjabi): udaharana vajon, koi kahi sakada hai ki motara kara sarakan nu zaruri taura'te vikas vala lai jandi hai. translation: for example, one could say that the motor car essentially leads to development of roads. xlm-mt+ckd: udaharana vajon, koi kahi sakada hai ki motara kara sarakan nu zaruri taura'te vikas vala lai jandi hai. translation: for example, one could say that the motor car essentially leads to development of roads. 2. source (hindi) : translation: like some other experts, he is skeptical about whether diabetes can be cured, noting that these findings have no relevance to people who already have type 1 diabetes. reference (kannada): madhumehavannu gunapadisalu sadhyave emba bagge itare itara kelavu tajñarante avaru kuda sansaya vyaktapadisuttare, ī sansodhanegal . u īgagale t . aip 1 madhumeha hondiruva janarige yavude prayojanagal . annu nīdilla. translation: like some other experts, he doubts whether diabetes can be cured, because these conclusions are not practical for people who have previously had type 1 diabetes. xlm-mt+ckd: itara kelavu tajñarante, avaru madhumehavannu gunapadisabahude endu sansayapaduttare, ekendare ī tīrmanagal . u ī hinde t . aip 1 madhumeha hondiruva vyaktigal . ige prayogikavagiruvudilla. translation: like some other experts, he doubts whether diabetes can be cured, because these conclusions are not practical for people who have previously had type 1 diabetes. ## conclusion in this paper, we empirically explored the methods for improving low-resource nmt, particularly for indic languages. we investigated several strategies for initialization of encoder and decoder, along with the knowledge distillation techniques. we conducted experiment on three low-resource indic languages in four indic-to-indic directions belonging to two language families, specifically focusing on those not covered by mbart-50. further, we perform additional analysis on languages supported by mbart-50 and high-resource language pairs.