Multi-Agent Mutual Learning at Sentence-Level and Token-Level for Neural Machine Translation (2020)

Abstract: Mutual learning, where multiple agents learn collaboratively and teach one another, has been shown to be an effective way to distill knowledge for image classification tasks. In this paper, we extend mutual learning to the machine translation task and operate at both the sentence-level and the token-level. Firstly, we co-train multiple agents by using the same parallel corpora. After convergence, each agent selects and learns its poorly predicted tokens from other agents. The poorly predicted tokens are determined by the acceptance-rejection sampling algorithm. Our experiments show that sequential mutual learning at the sentence-level and the token-level improves the results cumulatively. Absolute improvements compared to strong baselines are obtained on various translation tasks. On the IWSLT’14 German-English task, we get a new state-of-the-art BLEU score of 37.0. We also report a competitive result, 29.9 BLEU score, on the WMT’14 English-German task.

## introduction neural machine translation (nmt) has achieved significant progress over recent years @xcite @xcite @xcite . conventional training of the nmt models with hard targets limits the models' generalization ability @xcite . this has led to a rapid growth of research in developing more regularized models. teacher-student (t/s) learning @xcite @xcite is an effective method to handle this problem. it has been widely applied in many cases, e.g. model compression @xcite , domain adaptation @xcite and low-resource machine translation @xcite . t/s learning is a strategy that trains a student model with both hard targets and soft posteriors produced by a pre-trained teacher model @xcite . because training with soft targets provides smoother output distribution, t/s learning could outperform the single model training @xcite @xcite . however, does a teacher always outperform a student? in order to evaluate the pros and cons of different models, we conduct experiments on two different architectures. in this paper, we propose a two-step multi-agent mutual learning scheme, where one agent learns from other agents at the sentence-level as the first step, which we call sentence-wise mutual learning. when it becomes "smart", as the second step it will compare its own predicted tokens with other agents and only learn those poorly predicted tokens, which we call token-wise mutual learning. mutual learning is first proposed by @xcite for image classification tasks. compared to t/s learning, there is no fixed teacher model. the cotrained agents are teachers to one another. for sentence-wise mutual learning, each agent learns from other agents at sentence-level. when all of the agents converge after the first step, they continue to learn from other agents at the tokenlevel. each agent selects and learns the tokens that it predicts poorly. the poorly predicted tokens are determined by acceptance-rejection sampling @xcite . for every two agents, the target tokens can be divided into two subsets, where one agent performs poorly in one subset and src die evolution ist ein andauerndes thema hier auf der ted-konferenz gewesen, aber heute möchte ich ihnen die ansicht eines zu dem thema geben. ref evolution has been a perennial topic here at the ted conference, but i want to give you today one doctor's take on the subject. arch 1 (init 1 ) evolution has been a serious topic here at the ted conference, but i want to give you today the view of an ark on the subject. arch 1 (init 2 ) evolution has been a severe subject here at the ted conference, but today i want to give you the view of a doctor. arch 2 now, evolution is a continuous topic in the ted conference today, but today i want to give you the view of a doctor on the subject. to the best of our knowledge, this is the first work using multi-agent mutual learning for nmt tasks. the token-level knowledge distillation is also applied for the first time. our contributions are summarized as follows: alternatively, an ensemble integrated by multiple co-trained agents can also be treated as a teacher to all agents @xcite . dual learning dual learning @xcite or multi-agent dual learning @xcite is to leverage the duality between the primal task and the dual task. the source and target domains for these two tasks are opposite. even though both dual learning and mutual learning introduce extra models compared to traditional training method, the source and target sentences for all agents in mutual learning stay the same. there is only one task for mutual learning, i.e. translation from source sentences to target sentences. chen et al. (2017) propose a training method at sentencelevel and token-level for pivot-based zero-resource nmt. however, we have different definitions for the sentence-level and token-level translation. both the sentence-level and token-level translation in @xcite are considered as the sentencelevel translation in our work. the token-level translation in this paper means one agent only learns the poorly predicted tokens from other agents. ## experimental setup datasets in this paper, we run experiments on multiple benchmark mt datasets to evaluate the effectiveness of the proposed method, including iwslt'14 german-english (de-en), iwslt'14 dutch-english (nl-en), wmt'16 romanian-english (ro-en) and wmt'14 english-german (en-de). the amount of parallel sentence pairs for different mt tasks is shown in following @xcite , we split iwslt'14 de-en, iwslt'14 nl-en and wmt'16 ro-en datasets into various amounts of parallel sentence pairs for training, validation and testing. on wmt'14 en-de, we choose the wmt'16 training set and sample 4.5m parallel sentence pairs for training @xcite , employ newstest 2013 as the validation set and use newstest 2014 as the testing set @xcite . for all language pairs, we use byte-pair encoding @xcite with shared vocabularies. model architecture we mainly employ three types of the transformer model @xcite , i.e. transformer small, transformer base and transformer big, implemented in the fairseqpy toolkit @xcite . transformer base and transformer big stay the same as @xcite . the difference between transformer small and transformer base is that each encoder and decoder layer in transformer small employs a word representation size of 512, a feed-forward layer dimension of 1024 and 4 attention heads. transformer small is used for the small-scale iwslt'14 de-en and iwslt'14 nl-en datasets with a dropout rate of 0.3. transformer base is applied for the middle-scale wmt'16 ro-en and large-scale wmt'14 en-de with a dropout rate of 0.1. transformer big is also employed for the largescale wmt'14 en-de with a dropout probability of 0.3. we also train a convolutional sequence to sequence network (convs2s) @xcite and a seven encoder and decoder layer transformer small (transformer small7) on iwslt'14 de-en and iwslt'14 nl-en as our baselines. ## results sml and tml are conducted sequentially. firstly, we train our multiple agents at the sentence-level until convergence. secondly, we train again the agents with the best performance from sml at the token-level till convergence. ## conclusion we extend mutual learning to machine translation tasks at both the sentence-level and the token-level, where multiple agents are co-trained and distill 1 the results in this row are obtained with the average checkpoint from top 10 checkpoints. in this way, we can have strong baselines. the other results are obtained from the best checkpoint of the best agent. the trick of checkpoint averaging does not improve the results for ml. knowledge to one another throughout the training procedure. firstly, the agents learn the whole sentence from one another. after convergence, they only learn the poorly predicted tokens from other agents. sampling of poorly predicted tokens is done with acceptance-rejection sampling. with our two-step mutual learning, agents could learn from one another at different levels and are improved cumulatively. extensive experiments show significant improvements for both steps. we improve the state-of-the-art for iwslt'14 german-english from 36.3 @xcite to 37.0 points without using additional data. on wmt'14 english-german, we report 28.7 and 29.9 vs. 27.3 and 28.4 @xcite with transformer base and transformer big, respectively. we plan to extend the work by looking into more sophisticated training schedules for the agents with different architectures and applying backtranslation to ml.