Addressing Asymmetry in Multilingual Neural Machine Translation with Fuzzy Task Clustering (2022)

Abstract: Multilingual neural machine translation (NMT) enables positive knowledge transfer among multiple translation tasks with a shared underlying model, but a unified multilingual model usually suffers from capacity bottleneck when tens or hundreds of languages are involved. A possible solution is to cluster languages and train individual model for each cluster. However, the existing clustering methods based on language similarity cannot handle the asymmetric problem in multilingual NMT, i.e., one translation task A can benefit from another translation task B but task B will be harmed by task A. To address this problem, we propose a fuzzy task clustering method for multilingual NMT. Specifically, we employ task affinity, defined as the loss change of one translation task caused by the training of another, as the clustering criterion. Next, we cluster the translation tasks based on the task affinity, such that tasks from the same cluster can benefit each other. For each cluster, we further find out a set of auxiliary translation tasks that benefit the tasks in this cluster. In this way, the model for each cluster is trained not only on the tasks in the cluster but also on the auxiliary tasks. We conduct extensive experiments for one-to-many, manyto-one, and many-to-many translation scenarios to verify the effectiveness of our method.

## introduction neural machine translation (nmt) has achieved great success in recent years @xcite @xcite . the conventional bilingual translation model well handles the translation task for a single language pair, but it is infeasible to train an individual model for each language pair since there are thousands of despite its simplicity, multilingual nmt model often suffers from representation bottleneck caused by language interference @xcite when massive number of languages involved @xcite , which leads to performance degradation compared to bilingual translation models. several methods have been proposed to break the bottleneck, such as designing language specific modules @xcite , hidden units @xcite or routing path @xcite . among them, the most intuitive approach is grouping the languages into several clusters and training one multilingual nmt model for each cluster. these methods use similarity between language representations as clustering criterion, in which the representations come from language embedding vectors of a pretrained universal multilingual model or sparse language vectors of a multilingual knowledge base @xcite . however, language similarity is not enough for modeling the relationship among translation tasks because of two reasons. first, a translation task includes a source language and a target language, while the language similarity only consider the feature of a single language. second, the similarity metric cannot handle the asymmetric relationship between translation tasks. @xcite ; @xcite show the asymmetry in multi-task learning and multilingual translation in which one task may benefit from another but not vice versa. in this work, we propose a fuzzy task clustering method for multilingual nmt to address the asymmetric problem among translation tasks. we use the task affinity @xcite as the clustering criterion, which measures the loss change of one task caused by the training of another task. we first train a universal multilingual model to obtain the affinity between each two tasks. then we cluster tasks based on the affinity score such that tasks within each cluster can symmetrically benefit each other. for each cluster, we also select a set of auxiliary tasks that asymmetrically benefit the tasks in this cluster. finally, we build separate models for each cluster, and train the model on both the tasks in the cluster and the corresponding auxiliary tasks. in this way, the model for each cluster can exploit the auxiliary tasks to facilitate training and improve the model quality. compared to previous language clustering based methods, the proposed method improves positive knowledge transfer between tasks by introducing auxiliary tasks. to further address the increased burden of model training due to auxiliary tasks involved, we therefore propose a dynamic data sampling strategy, in which the sampling weights of auxiliary tasks gradually decrease to allow the optimization process to focus more on the tasks that used in inference. to summarize, our method has the following advantages: ## method in this section, we first describe the translation task affinity used for modeling the asymmetric relationship between tasks (section 3.1), followed by the translation task clustering method that clusters the tasks and finds out the auxiliary tasks for each cluster (section 3.2). we finally build a model for each cluster and train the model with dynamic data sampling strategy (section 3.3). ## conclusion in this work, we have proposed a fuzzy task clustering method to address the asymmetric problem in multilingual nmt based on task affinity. the task affinity is defined by the loss change of one task after a step training of another task. based on the affinity, we cluster translation tasks, each of which contains tasks that are symmetric. each cluster is further equipped with auxiliary tasks that can benefit the model training of this cluster. experiments show that our fuzzy task clustering method significant outperforms the strong baselines. we also show the effectiveness of incorporating auxiliary tasks in a multilingual translation model. in the future, we plan to explore more efficient and effective clustering criterion by exploiting large-scale pre-trained multilingual models. biao zhang, philip williams, ivan titov, and rico sennrich. 2020. improving massively multilingual neural machine translation and zero-shot translation. in proceedings of the 58th annual meeting of the association for computational linguistics, acl 2020, online, july 5-10, 2020, pages 1628-1639. association for computational linguistics.