Latent Group Dropout for Multilingual and Multidomain Machine Translation (2022)

Abstract: Multidomain and multilingual machine translation often rely on parameter sharing strategies, where large portions of the network are meant to capture the commonalities of the tasks at hand, while smaller parts are reserved to model the peculiarities of a language or a domain. In adapter-based approaches, these strategies are hardcoded in the network architecture, independent of the similarities between tasks. In this work, we propose a new method to better take advantage of these similarities, using a latent-variable model. We also develop new techniques to train this model end-to-end and report experimental results showing that the learned patterns are both meaningful and yield improved translation performance without any increase of the model size.

## introduction multidomain and multilingual machine translation aim to develop one single model to perform translation for multiple domains and multiple language pairs, respectively. 1 these paradigms are motivated by the compactness of the resulting translation system @xcite , the hypothetical positive knowledge transfer between similar domains @xcite or between languages in the same family @xcite . however, having all the tasks use exactly the same model parameters can cause negative interference between unrelated tasks @xcite . hence, the recent development of approaches relying on a partial sharing of the parameters, eg. using adapter layers as studied in @xcite @xcite . if these techniques have proven effective for building strong baselines, they fail to fully take advantage of the similarities that exist between domains and tasks, as documented eg. in @xcite . this is because the partition of the parameter space between generic or task-specific subparts, and their allocation to each task, is hard-coded in the network, irrespective of the actual commonalties and differences in the data space. in this work, we study and develop a new method, multi-task group dropout, aimed to take into account the similarity between tasks in a more effective way, by learning the network organization from the data. to this end, we introduce a set of latent variables in the model, to account for the unseen association between tasks and regions of the representation space and show how training can still be performed end-to-end using a variational surrogate of the log-likelihood loss function. our experiments with multilingual and multidomain machine translation confirm that this method can automatically detect similarities in the data, meaning that related tasks use the same subparts of the network. our results also show that this method is comparable to using adapter layers in a number of empirical comparisons; however, contrarily to adapters, these performance are obtained without any increase of the model size. our contributions are primarily methodological and can be summarized as follows: in our approach, we also require to know the task @xmath0. . . n d -1] for each training and test instance. the structure of our transformer networks @xcite is however based on the notion of groups of nodes in the computation graph. at the input of each transformer layer @xmath1. . . l], we partition all input state vectors into n p groups of nodes, and zero-out a task-dependent subset of these groups. the assignment of tasks to groups will be learned from the data, under the constraint that each task only activates exactly k groups of active nodes, while the all the other values are nullified, akin to a dropout process (see figure 1 ). formally, a group dropout mask m d l is a n p -dimensional binary vector containing exactly k ones: group p (∈ [0, . . . , n p -1]) is retained for task ## conclusions and outlook in this work, we have presented a novel method for multdomain and multilingual translation. it allows us to jointly search for an optimal assignment of sub-networks to tasks and to learn the parameters of the underlying network. our method relies on a sound mathematical framework and an end-to-end optimization procedure; it only adds a small number of extra parameters. the additional training cost is also reasonable, amounting to 100k iterations in the multidomain setting, given the observed gains in performance. experimentally, we achieve a large improvement over a transformer baseline; our performance are also comparable to that of a strong a multi-task baseline using residual adapter modules which rely on a large number of extra parameters. for multilingual translation, our model outperforms multilingual transformer and language adapters in 3 our of 4 settings. lamgd seems specially beneficial for training languages with little parallel data, which can take advantage of the resources that are available for related languages. besides, we provided an thorough analysis of the similarities between learned sub-networks and demonstrate a strong correlation between the learned similarities and the proximity of the corresponding tasks (domain or language). there are several ways in which our methodology can be improved. in future work, we would first like to provide an complete variational framework to model both the number of groups, k and the selection of the dropout masks. second, we also intend to dispense with the domain information during inference: this would mean replacing the dependency on d in the variational distribution by a dependency on the input x. another interesting direction will be to consider adapting the size and capacity allocated to each domain / language, depending on the difficulty of the associated translation task. addressing these questions will allow to us replace heuristic choices in the architecture design with an increased dependency on the training data.