Language-Aware Multilingual Machine Translation with Self-Supervised Learning (2023)

Abstract: Multilingual machine translation (MMT) benefits from cross-lingual transfer but is a challenging multitask optimization problem. This is partly because there is no clear framework to systematically learn language-specific parameters. Self-supervised learning (SSL) approaches that leverage large quantities of monolingual data (where parallel data is unavailable) have shown promise by improving translation performance as complementary tasks to the MMT task. However, jointly optimizing SSL and MMT tasks is even more challenging. In this work, we first investigate how to utilize **intra-distillation** to learn more *language-specific* parameters and then show the importance of these language-specific parameters. Next, we propose a novel but simple SSL task, **concurrent denoising**, that co-trains with the MMT task by concurrently denoising monolingual data on both the encoder and decoder. Finally, we apply **intra-distillation** to this co-training approach. Combining these two approaches significantly improves MMT performance, outperforming three state-of-the-art SSL methods by a large margin, e.g., 11.3% and 3.7% improvement on an 8-language and a 15-language benchmark compared with MASS, respectively.

## introduction multilingual machine translation (mmt) @xcite comes with the problem of designing architectures where certain parameters are shared and certain parameters are more language-specific. in order to mitigate negative interference across languages, recent studies have investigated language-specific parameters, including searching for more languagespecific parameters @xcite , or adding extra language-specific components to the original work done during an internship at meta ai research 1 code is released at https://github.com/ fe1ixxu/cd_id_mmt . figure 1 : concurrent denoising is a complementary task to the mmt task. both tasks are applied with intradistillation, where we forward pass model twice for the translation and masked inputs and each time we disable different subsets of parameters (illustrated by different colors). then, for each task, we not only minimize the difference between the target and two outputs (e.g., minimize difference(p 1 , t p ) and difference(p 2 , t p ) in the mmt task), we also minimize the difference between two translated outputs as well as two denoised outputs (e.g., minimize difference(p 1 , p 2 ) for mmt). the difficulty of scaling mmt to low-resource and long-tail languages arises due to the scarcity of abundantly available parallel aligned data. previous works @xcite @xcite @xcite try to tackle this by collecting massive amounts of monolingual data and using various types of self-supervised learning (ssl) objectives, such as denoising autoencoder (dae) @xcite or masked sequence to sequence (mass) @xcite as auxiliary tasks to co-train with the mmt task, to compensate for the scarcity of parallel data for low-resource languages. following this line, we secondly aim to propose a more effective ssl objective. with the goal of learning language-aware mmt models and designing more effective ssl methods for mmt, we introduce two approaches. the first approach is intra-distillation (id) @xcite , which performs a forward pass through the model k times foot_0 , and in each pass disables a different set of parameters. this enforces consistent contributions between these disabled parameters by minimizing the difference between the k outputs. id was originally proposed by @xcite to achieve a balanced parameter contribution in a model. here, we study the effectiveness of id in learning languagespecific parameters for mmt models. next, we introduce concurrent denoising (cd) which is an auxiliary self-supervised task jointly trained with the mmt task. cd predicts the same masked sentences both on the encoder and decoder side with a shared projection layer to facilitate the consistent understanding between encoder and decoder representations. we show that cd outperforms several state-of-the-art ssl methods for translation. finally, we apply id to our cotraining scheme to further improve the mmt performance by learning more language-specific parameters. the overall framework is illustrated in figure 1 and we summarize our main contributions below. ## proposed self-supervision method we extend our study of language-awareness to mmt models co-trained with self-supervised objectives that have been shown to improve translation performance. we first propose a simple but effective self-supervised learning objective, concurrent denoising (cd), and then investigate the effectiveness of id in helping improve multitask optimization challenges of co-training cd and models are pruned starting with the least languagespecific parameters. mmt tasks together by learning more languagespecific parameters. ## analysis 6.1 ablation study the final loss, described in equation 8, has 6 loss terms. except for the translation loss, we ablate the relative contribution of all the other 5 loss terms to the translation task performance. in ## conclusions we show extensive analysis that intra-distillation training helps multilingual translation by learning more language-specific parameters. we propose concurrent denoising, improving upon multiple state-of-the-art self-supervised learning methods. moreover, we demonstrate that applying intradistillation to the above co-training scheme offers further improvements to translation performance.