Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translation (2023)

Abstract: Despite advances in multilingual neural machine translation (MNMT), we argue that there are still two major challenges in this area: data imbalance and representation degeneration. The data imbalance problem refers to the imbalance in the amount of parallel corpora for all language pairs, especially for long-tail languages (i.e., very low-resource languages). The representation degeneration problem refers to the problem of encoded tokens tending to appear only in a small subspace of the full space available to the MNMT model. To solve these two issues, we propose Bi-ACL, a framework which only requires target-side monolingual data and a bilingual dictionary to improve the performance of the MNMT model. We define two modules, named bidirectional autoencoder and bidirectional contrastive learning, which we combine with an online constrained beam search and a curriculum learning sampling strategy. Extensive experiments show that our proposed method is more effective than strong baselines both in long-tail languages and in high-resource languages. We also demonstrate that our approach is capable of transferring knowledge between domains and languages in zero-shot scenarios.

## introduction multilingual neural machine translation (mnmt) makes it possible to train a single model that supports translation from multiple source languages into multiple target languages. this has attracted a lot of attention in the field of machine translation @xcite @xcite . mnmt is appealing for two reasons: first, it can transfer the knowledge learned by the model from high-resource to low-resource languages, especially in zero-shot scenarios @xcite ; second, it uses only one unified model to translate between multiple language pairs, which saves on training and deployment costs. although significant improvements have been made recently, we argue that there are still two major challenges to be addressed: i) mnmt models suffer from poor performance on long-tail languages (i.e., very low-resource languages), for which parallel corpora are insufficient or nonexisting. we call this the data imbalance problem. for instance foot_0 , 21% of the language pairs in the m2m_100 model @xcite have a bleu score of less than 1 and more than 50% have a bleu score of less than 5. only 13% have a bleu score over 20. for example, the average bleu score for the language pairs with irish as the target language is only 0.09. ii) degeneration of mnmt models stems from the anisotropic distribution of token representations, i.e., their representations reside in a narrow subset of the entire space @xcite . this is called the representation degeneration problem. it can lead to a prevalent issue in large-scale mnmt: the model copies sentences from the source sentence or translates them into the wrong language (off-target problem; @xcite . to address the data imbalance problem, prior work has attempted to improve the performance of a machine translation model without using any parallel data. on the one hand, unsupervised machine translation @xcite attempts to learn models relying only on monolingual data. on the other hand, bilingual dictionaries have shown to be helpful for machine translation models @xcite . what these approaches have in common is that they only require data that is both more accessible and cheaper than parallel data. as an example, 70% of the languages in the world have bilingual lexicons or word lists available @xcite . representation degeneration is a prevalent problem in text generation @xcite and machine translation models @xcite . contrastive learning @xcite aims to bring similar sentences in the model close together and dissimilar sentences far from each other in the representation space. this is an effective solution to the representation problem in machine translation @xcite . however, the naïve contrastive learning framework that utilizes random non-target sequences as negative examples is suboptimal, because they are easily distinguishable from the correct output @xcite . to address both problems mentioned above, we present a novel multilingual nmt approach which leverages plentiful data sources: target-side monolingual data and a bilingual dictionary. specifically, we start by using constrained beam search @xcite to construct pseudo-parallel data in an online mode. to overcome the data imbalance problem, we propose training a bidirectional autoencoder, while to address representation degeneration, we use bidirectional contrastive learning. finally, we use a curriculum learning @xcite sampling strategy. this uses the score given by token coverage in the bilingual dictionary to rearrange the order of training examples, such that sentences with more tokens in the dictionary are seen earlier and more frequently during training. in summary, we make the following contributions: i) we propose a novel approach that uses only target-side monolingual data and a bilingual dictionary to improve mnt performance. ii) we define two modules, bidirectional autoencoder and bidirectional contrastive learning, to address the data imbalance and representation degeneration prob-lem. iii) we show that our method demonstrates zero-shot domain transfer and language transfer capability. iv) we also show that our method is an effective solution for both the repetition @xcite and the off-target @xcite problems in large-scale mnmt models. ## related work multilingual neural machine translation. mnmt is rapidly moving towards developing large models that enable translation between an increasing number of language pairs. fan et al. (2021) proposed m2m_100 model that enables translation between 100 languages. @xcite and costa-jussà et al. (2022) extend the current mnmt models to support translation between more than 200 languages using supervised and self-supervised learning methods. autoencoder. an autoencoder (ae) is a generative model that is able to generate its own input. there are many variants of ae that can be useful for machine translation. @xcite and @xcite propose using a variational autoencoder to improve the performance of machine translation models. a variant of the same generative model is the denoising autoencoder, which is an important component of unsupervised machine translation models @xcite . however, the utility of autoencoders has not been fully explored for mnmt. to the best of our knowledge, we are the first to propose training an autoencoder using only target-side monolingual data and a bilingual dictionary to improve low-resource mnmt. contrastive learning. contrastive learning is a technique that clusters similar data together in a representation space while it simultaneously separates the representation of dissimilar sentences. it is useful for many natural language processing tasks @xcite . recently, @xcite and @xcite used contrastive learning to improve machine translation and obtained promising results. however, these methods use the random replacing technique to construct the negative examples, which often leads to a significant divergence between the semantically similar sentences and the ground-truth sentence in the model representation space. this large changes makes the model more difficult to distinguish correct sentence from incorrect ones. we use small perturbations to construct negative examples, ensuring their proximity to the ground-truth sentence within the semantic space, which significantly mitigates the aforementioned issue. ## method our goal is to overcome the data imbalance and representation degeneration issues in the mnmt model. we aim to improve the performance of mnmt without using parallel data, instead relying only on target-side monolingual data and a bilingual dictionary. our approach contains three parts: online pseudo-parallel data construction (section 3.1), bidirectional autoencoder (section 3.2) and bidirectional contrastive learning (section 3.3). figure 1 illustrates the overview of our method. the architectures of the bidirectional autoencoder (left) and bidirectional contrastive learning (right) are presented in figure 2 . ## experiments datasets. we conduct three group of experiments: bilingual setting, multilingual setting and highresource setting. in the bilingual setting, we focus on improving the performance on a specific long-tail language pair. we choose 10 language pairs at random that have @xmath0.5 in the original m2m_100 model and a considerable amount of monolingual data in the target language in newscrawl. 4 the language pairs cover the following languages (iso 639-1 language code foot_4 ): en, ta, kk, ar, ca, ga, bs, ko, ka, tr, af, hi, jv, ml. in the multilingual setting, we aim to improve the performance on long-tail language pairs, which share the same target language. we randomly select 10 languages where the average bleu score on the language pairs with the same target language is less than 2.5. for the languages not covered from news-crawl, we use the monolingual data from ccaligned foot_6 (el-kishky et al., 2020). the languages we use are: ta, hy, ka, be, kk, az, mn, gu. for the high-resource setting, we aim to validate whether our proposed method also works for high-resource languages. we randomly select 6 language pairs that cover the following language codes: en, de, fr, cs. dictionaries. we extract bilingual dictionaries using the wiktextextract foot_7 tool. for pairs not involving english, we pivot through english. given a source language ℓ s and a target language ℓ t , the intersection of the two respective bilingual dictionaries with english creates a bilingual dictionary d ℓs→ℓt dict from ℓ s to ℓ t . the statistics of the dictionaries can be seen in appendix d.1. ## results the performance of the language pair is poor in the original m2m_100 model. in addition, using the curriculum learning sampling strategy further improves our model's performance. ## analysis in this section, we conduct additional experiments to better understand the strengths of our proposed methods. we first investigate the impact of four components on the results through an ablation study (section 6.1). then, we evaluate the zeroshot domain transfer ability and language transfer ability of our method (section 6.2). finally, we evaluate some impact factors (the quality of bilingual dictionary and the amount of monolingual data) on our proposed method (section 6.3) and present a case study to show the strengths of our approach in solving the repetition problem and offtarget issues in mnmt model. ## conclusion to address the data imbalance and representation degeneration problem in mnmt, we present a framework named bi-acl which improves the performance of mnmt models using only target-side monolingual data and a bilingual dictionary. we employ a bidirectional autoencoder and bidirectional contrastive learning, which prove to be effective both on long-tail languages and high-resource languages. we also find that bi-acl shows language transfer and domain transfer ability in zeroshot scenarios. in addition, bi-acl provides a paradigm that an inexpensive bilingual lexicon and monolingual data should be fully exploited when there are no bilingual parallel corpora, which we believe more researchers in the community should be aware of.