M ULTI -A GENT D UAL L EARNING
A BSTRACT
Dual learning has attracted much attention in machine learning, computer vision and natural language processing communities.
The core idea of dual learning is to leverage the duality between the primal task (mapping from domain X to domain
Y ) and dual task (mapping from domain Y to X ) to boost the performances of both tasks.
Existing dual learning framework forms a system with two agents (one primal model and one dual model) to utilize such duality.
In this paper, we extend this framework by introducing multiple primal and dual models, and propose the multi-agent dual learning framework.
Experiments on neural machine translation and image translation tasks demonstrate the effectiveness of the new framework.
In particular, we set a new record on IWSLT 2014 German-to-English translation with a 35 . 44 BLEU score, achieve a 31 .
03 BLEU score on WMT 2014 English-toGerman translation with over 2 .
6 BLEU improvement over the strong Transformer baseline, and set a new record of 49 .
61 BLEU score on the recent WMT
English-to-German translation.
I NTRODUCTION
Motivated by the dual nature of many tasks, e.g., English-to-German vs. German-to-English in machine translation, photo-to-Monet vs. Monet-to-photo in image translation, and speech recognition vs. speech synthesis, dual learning is proposed (He et al., 2016) and applied to many applications including neural machine translation (NMT) (He et al., 2016; Xia et al., 2017a;b), image-to-image translation (Zhu et al., 2017; Yi et al., 2017; Luo et al., 2017), question answering (Duan et al., 2017; Sun et al., 2019) and image captioning (Huang et al., 2018).
The basic idea of dual learning is to leverage the duality between the two tasks as the feedback signal to regularize training.
The regularization has been implemented with different ways in existing works, such as maximizing the likelihood of data reconstruction (He et al., 2016), constraining joint probabilistic consistency (Xia et al., 2017b), and encouraging the model-level information sharing (Xia et al., 2018).
Among them, the principle of likelihood maximization of data reconstruction has been mostly adopted (Zhu et al., 2017; He et al., 2016), in which dual learning is formulated as a two-agent game: One agent targets at learning the mapping f : X (cid:55)→ Y , while the other learning the mapping of opposite direction g : Y (cid:55)→ X .
Here X and Y are two domains.
In dual learning, x ∈ X is ﬁrst mapped to ˆ y ∈ Y through f and then reconstructed to ˆ x ∈ X by g (ˆ y ) .
The distortion between x and ˆ x , denoted as ∆ x ( x, ˆ x ) , is used as the feedback signal to regularize training.
Similar reconstruction error can also be constructed in Y and further used for training.
In such a two-agent game, g and f can be viewed as an evaluator for each other.
g is used to evaluate the quality of ˆ y generated by f and return the feedback signal ∆ x ( x, g (ˆ y )) back to f , and vice versa.
The quality of such evaluation plays an important role to improve the training of the mapping functions.
In the current dual learning framework, only one agent g is used to evaluate and provide feedback signals to the mapping function f
in the other direction.
In this work, we introduce multiple agents into the learning system to further exploit the potential of dual learning.
The agents in the same direction have similar capability and certain level of diversity to map one domain to the other domain, i.e., X → Y or Y → X .
Different agents can be obtained by training multiple f ’s and g ’s independently with different random seeds for initialization and data access order.
Then for the output of each f (or g ), multiple g ’s (or f ’s) will provide feedback signals.
Intuitively, more agents can lead to more reliable and robust feedback, just like the majority voting of multiple experts, and it is expected to achieve better ﬁnal performance.
We name this new dual learning framework with multiple agents as multi-agent dual learning 1 .
Although multiple agents are involved in multi-agent dual learning, we still focus on training two mappings f 0 :
X → Y and g 0 :
Y → X , similar to the traditional two-agent dual learning.
The additionally introduced agents play the role of facilitating the training of f 0 and g 0 .
The parameters of the additional agents are ﬁxed during training and have no part to play during testing/inference.
Therefore, the inference of multi-agent dual learning is of the same complexity as it is of the standard two-agent dual learning.
More precisely, apart from the two agents f 0 and g 0 , we further employ 2( N − 1) additional agents, N ≥ 2 , which are pre-trained before optimizing f 0 and g 0 and subsequently held constant.
Among them, N − 1 agents are leveraged to model the mapping f
i : X (cid:55)→ Y and the others N − 1 agents
model
g
i : Y (cid:55)→ X , i ∈ { 1 , 2 , · · · , N − 1 } .
2 All these 2( N − 1) agents are used to facilitate the training of f 0 and g 0 in a similar manner as the standard dual learning with minor revisions: To help the training of f 0 , for any x ∈ X , we ﬁrst map x to ˆ y by f 0 , build the feedback signals by adaptively summing ∆ x ( x, g i (ˆ y )) for any i ∈ { 0 , 1 , · · · , N − 1 } , and then regularize training with the feedback signals.
The training of g 0 is performed similarly.
We conduct experiments on two benchmark tasks of dual learning, NMT and image-to-image translation, to verify our proposed framework.
For NMT, we work on the English ↔ German translation task with different scales of bilingual data.
On IWSLT 2014 translation task with 153 K bilingual data and no monolingual data, we achieve 29 . 52 and 35 .
BLEU scores for English → German and German → English translations respectively, setting a new record on this task.
On WMT 2014 translation task with 4 .
5 M bilingual sentence, we achieve 30 .
03 and 33 .
64 BLEU scores without/with monolingual data, which are the best results of the same settings.
On the recent WMT 2018 English → German translation task, we set a new record of 49 .
61 BLEU score, improving the previous best system by over 1 . 3 BLEU score.
On WMT 2016 unsupervised NMT task, where no bilingual data is available and only 50 M monolingual data is provided, we achieve 19 .
BLEU scores, outperforming all the previous pure NMT based systems.
We work on two groups of tasks for image-to-image translation, paint ↔ photo translations (paint including Van Gogh, Monet, Ukiyo-e and Cezanne), and cityscapes label ↔ photo translation.
Experimental results show that we can generate better images than CycleGAN.
The remaining part of this paper is organized as follows.
We introduce the basic framework in
Section 2.
The applications to NMT and image translation are presented in Section 3 and Section 4.
We conclude this paper and discuss the future directions in the last section.
F RAMEWORK
In this section, we mathematically formulate the multi-agent dual learning framework, and compare it with previous learning settings.
Consider two domains of interests, X and Y .
Let D x and D y denote the collection of training data from X and Y respectively, D x ⊂ X , D y ⊂ Y .
We aim to learn two agents f 0 :
X (cid:55)→ Y , and g 0 :
Y → X .
Without loss of generality, we refer to learning f 0 and g 0 as the primal task and the dual task respectively.
Let ∆ x ( x, x (cid:48) ) be a mapping X × X → R , representing the dissimilarity/distance/error between x and x (cid:48) where x, x (cid:48) ∈ X , and ∆ y ( y, y (cid:48) ) denote the dissimilarity between the two elements y and y (cid:48) in space Y .
The standard dual learning loss (He et al., 2016) is formulated as
[Equation]
where |D x | and |D y | denote the number of elements in D x and D y .
In a multi-agent dual learning framework, as a prerequisite there are N − 1 pretrained primal models
f i : X → Y and N − 1 dual models g i : Y → X , i = { 1 , 2 , · · · , N − 1 } .
Each f i is pretrained by minimizing the cross entropy loss − (cid:80) ( x,y ) log P
( y | x ; f i ) via stochastic gradient descent where ( x, y ) is the paired data, or using unsupervised learning techniques to obtain like unsupervised NMT (Lample et al., 2018) or unsupervised image translation (Zhu et al., 2017).
Different methods can be used to obtain diverse agents
f i , i ∈ { 1 , 2 , · · · , N − 1 } , including using various random seeds to affect weight initialization and input order of the training samples, different model architectures, or training over different subsets.
The same holds for g i , i ∈ { 1 , 2 , · · · , N − 1 } .
In the training process of f 0 and g 0 , all these pretrained 2( N − 1) models remain ﬁxed, and are linearly aggregated with f 0 , g 0 into two mixed models F α and G β .
Speciﬁcally, given any α i ≥and β i ≥ 0 for any i ∈ { 0 , 1 , · · · , N − 1 } ,
[Equation]
For multi-agent dual learning, the duality feedback signal is built upon F α and G β .
Following the basic framework of dual learning (He et al., 2016), for any x ∈ X , all agents ﬁrst cooperate to generate a ˆ y ∈ Y by ˆ y =
F α ( x ) , and then jointly reconstruct the ˆ x ∈ X through ˆ x = G β (ˆ y ) .
The reconstruction error between y ∈ Y and ˆ y =
F α ( G β ( y )) is similarly constructed.
The out-coming dual learning loss is deﬁned as
[Equation]
In this way, N agent pairs (prima-dual model pairs) are involved to jointly train and improve f 0 and g 0 .
When N = 1 , Our algorithm degenerates to standard dual learning.
No labeled information is required for building (cid:96) dual .
We can apply it on either labeled data or unlabeled data.
Note that Eqn.
(3) is the loss about duality.
Other training objectives could also be included.
For example, in NMT, if bilingual data is available, the cross entropy loss can be included to guide the training; in image-to-image translation, the GAN loss could also be included to enforce the generated images into the correct categories.
In the next two sections, we will discuss how to adapt
Eqn. (3) to different applications.
Discussion .
While there are existing works using multiple agents to boost the model performance, none of them has leveraged the duality.
We use the X → Y mapping task to compare the previous work with our proposed framework, following the notations deﬁned in Section 2.
(1) Ensemble learning (Zhou, 2012) is a straightforward way to combine multiple models during inference.
To predict the label of x ∈ X , all agents vote together and the ﬁnal label would be arg min y ∈Y (cid:80) N − 1 i =
α i (cid:96) ( f i ( x ) , y ) , where (cid:96) is the loss function over space X × Y .
The α
i ’s can be simply set as 1 /N or adaptively set according to the quality of each agent.
There are several differences between ensemble learning and our work: 1) ensemble learning does not use multiple agents in training as we do; 2) our multi-agent dual learning uses only one model f 0 in inference, which is more efﬁcient than ensemble learning that uses multiple agents; and 3) duality is not considered in conventional ensemble learning.
(2) Knowledge distillation with multiple agents (Hinton et al., 2015; Kim & Rush, 2016).
Knowledge distillation consists of two steps: ﬁrst, all f
i ’s generate soft labels for x ∈ X , e.g., ˆ y = arg min y ∈Y (cid:80) N − 1 i =
α i (cid:96) ( f i ( x ) , y ) ; the generated pairs ( x, ˆ y ) ’s are together used to train a new model.
Each ( x, ˆ y ) is regarded as labeled data without evaluating the quality of ˆ y or considering whether it is good enough for model training.
In our proposed framework, we leverage the duality to build a feedback loop so as to evaluate the quality of generated pairs.
A PPLICATION TO N EURAL M ACHINE T RANSLATION
In this section, we introduce how to adapt the proposed multi-agent dual learning framework to
Neural Machine Translation (NMT), and present evaluation on several public translation datasets.
3. A DAPTION
Note that for NMT, X and Y stands for the collection of all possible sentences of two given languages, while D x and D y is the dataset.
Denote the parameters of mapping f
and g 0 as θ f 0 and θ grespectively.
Following the common practice in NMT, the ∆ x and ∆ y is speciﬁed as the negative log-likelihood, that is, for any x ∈ D x or x ∈ X ,
[Equation]
For ease of reference
[Equation]
we brieﬂy denote Eqn.
(4) as
[Equation]
Similarly, for any y ∈ Y , we have
[Equation]
Following Hassan et al.
[Equation]
we minimize the upper bounds of ∆ x and ∆ y for ease of optimization, which are respectively denoted as ¯∆ x and ¯∆ y :
[Equation]
where two ≥ ’s hold due to Jensen’s inequality.
Then we turn to minimize ˜ (cid:96)
[Equation]
Directly calculating the gradients for ¯∆ x w.r.t θ f
and θ g 0 is difﬁcult in that: i) Y is exponentially large since it represents all possible sentences of a language, which makes the computation intractable .
ii)
All f i ’s
and g i ’s
( i ≥ 1) need to be loaded into the memory, although they will not be updated, which makes the computation inefﬁcient.
Similar issues exist for computing the gradients of ¯ ∆ y w.r.t θ f {+
+} and θ g 0 .
We use importance sampling to address the above two difﬁculties.
Deﬁne γ
=
(0 , 1 N − 1 , · · · , 1 N − 1 ) , thus F γ and G γ represent the combined model from all the pre-trained agents without the target models f 0 and g 0 (see Eqn.
Deﬁne
[Equation]
Then the gradients for ¯∆ x w.r.t θ f 0 is: ∂
¯∆ x ∂θ f 0 = − (cid:80) ˆ y ∈Y P (ˆ y | x ; F γ ) ∂δ ( x, ˆ y )
∂θ f 0 , and can be estimated by:
(i)
Given an x ∈ D x and y ∈ D y , sample a ˆ y according to the distribution P ( ·| x ; F γ ) ; sample an ˆ x according to the distribution P ( ·| y ; G γ ) ; (ii) Estimate the gradients of ¯∆ x w.r.t θ f 0 as follows (and similarly for θ g 0 without loss of generality):
[Equation]
Our method is summarized in Algorithm 1.
We do the ofﬂine sampling by sampling ˆ x ’s and ˆ y ’s in advance (step 3); and computing the probabilities related to f i ’s
and g i ’s
i ≥ 1 in advance (step 4).
When bilingual data is available, denoted as B = { ( x k , y k ) } Mk =1 where M is the number of training data, we can also apply the negative log-likelihood loss on bilingual data.
Following Eqn. (9) and Eqn.
{+(10), calculate Grad f 
 and Grad g 0 as follows:+}
[Equation]
and
[Equation]
− 1 |B ˆ xy | (cid:80) (ˆ x,y ) ∈B
ˆ xy ∂φ ( y, ˆ x )
∂θ g 0 ;
If bilingual data is available, sample a batch B xy ⊂ B of size K , calculate
Grad f 0 ← Grad f 0 − 1 K ∇ θ f 0 (cid:80) ( x,y ) ∈B xy log P ( y | x ; f 0 ) and
Grad g 0 ← Grad g 0 − 1 K ∇ θ g 0 (cid:80) ( x,y ) ∈B xy log P ( x | y ; g 0 ) ;
Update the parameters: θ f 0 ← θ f 0 − η Grad f 0 , θ g 0 ← θ g 0 − η Grad g 0 ; 10 end
3. E XPERIMENT S ETTINGS
Dataset We use multiple benchmark NMT datasets to evaluate the effectiveness of the proposed framework, including IWSLT 2014 English ↔ German translation, WMT 2014 English ↔ German translation and WMT 2016 unsupervised English ↔ German translation.
English and German is denoted as “En” and “De” respectively for ease of reference.
For IWSLT
En ↔ De translation, following Edunov et al.
(2018b), we lowercase all the sentences, and split them into training/validation/test set with 153 k / 7 k / 7 k sentences respectively.
For WMT 2014 En ↔ De translation, we choose WMT 2014 training set and ﬁlter out 4 .
5 M sentences pairs following Gehring et al.
(2017) and Vaswani et al.
We concatenate newstest 2012 and newstest 2013 as the validation set and use newstest 2014 as the test set.
We also select 8 M English and 8 M German monolingual sentences from newscrawl 2013 as monolingual dataset.
For unsupervised En ↔ De, following Lample et al.
(2018), we choose 50 M monolingual English and German sentences and use newstest 2016 as the test set for a fair comparison with the previous work.
We preprocess the words into word-pieces in the same way as Wu et al.
Model Architecture We use Transformer (Vaswani et al., 2017) as the basic model structure.
For
IWSLT En ↔ De, we use the transformer small conﬁguration with 4 and 8 blocks to verify the generality of our model with different structures (a 4 -block setting refer to transformer with 4 encoder blocks and 4 decoder blocks), where the word embedding dimension, hidden state dimension and non-linear layer dimension is set to be 256 , 256 and 1024 respectively.
For the WMT task, following Vaswani et al.
(2017), we use the transformer big setting with 6 blocks, where the word embedding dimension, hidden state dimension and non-linear layer dimension is 1024 , 1024 and 4096 respectively.
For the unsupervised NMT task, we choose transformer base setting following Lample et al.
(2018), where the above three dimensions are 512 , 512 and 2048 .
The dropout rates for the three settings are 0 .
1 and 0 .
1 respectively.
Optimization and Evaluation We choose Adam (Kingma & Ba, 2015) to optimize the network.
The η in Algorithm 1 is set as 2 × 10 − 4 .
The learning rate decay rule and the two β ’s of Adam are the same as Vaswani et al.
For the three tasks, we use one, eight and four M40 GPUs to train those networks for three, ﬁve and six days respectively.
Beam search is applied to generate translations for all the models, where the beam sizes for the three tasks are 6 , 4 and 4 respectively.
The evaluation metric is BLEU score (Papineni et al., 2002), which is the geometric mean of n -gram precisions, n ∈ { 1 , 2 , 3 , 4 } .
Baselines We implement three types of important baselines 3 , including back translation (Sennrich et al., 2015; Edunov et al., 2018a), knowledge distillation (Kim & Rush, 2016) and the two-agent dual learning (He et al., 2016).
Take the f : X (cid:55)→ Y translation as an example: 1) To use back translation, a reversed translation model g : Y (cid:55)→ X is ﬁrst trained on the original bilingual dataset, then the dataset { ( g ( y ) , y ) | y ∈ D y } is constructed, which is then concatenated with the original dataset and train another model f . 2)
To use knowledge distillation, we pre-train a teacher model f T : X → Y , and use it to generate the corresponding dataset { ( x, f T ( x )) | x ∈ D x } , which is then mixed with the ground truth data and used to train a new model 4 .
For these two baselines, we generate sentences using both a single model and multiple models.
Two-agent dual learning, which uses f together with g to build the feedback signal to regularize the training (see Eqn.
(1)), which we denote as “Dual-1”.
Our multi-agent models are denoted as “Dual- i ” ( i > 1 )
3. R ESULTS OF NMT ON IWSLT DATASET
We regard the English and German sentences in the IWSLT14 bilingual corpus as D x and D y in
Algorithm 1.
We use 4-block networks (i.e. F γ in Algorithm 1) to generate translations, which are then used to train both 4-block (4B) and 8-block (8B) networks.
We compare dual learning (Duali ) with the standard baseline (Standard), knowledge distillation (KD- i ) and back translation (BT- i ) with different number of primal-dual model pairs i ∈ { 1 , 5 } .
Speciﬁally, Dual- 1 is the two-agent dual learning baseline, and Dual- 5 is our multi-agent model with 4 additional pairs of agents.
The results of IWSLT En ↔ De are presented in Table 1.
We can observe that: 1)
Dual learning has brought signiﬁcant improvement over all the baselines (Standard, KD and BT) in both single-agent and multi-agent settings, demonstrating the effectiveness of dual learning; 2) Involving multiple agents into the learning system leads to better performances, which shows the importance of additional feedback signals from other agents.
In particular, Dual- 5 outperforms Dual- 1 by around 0 .
BLEU in 4B setting and around 0 .
8 BLEU in 8B setting, which proves that dual learning can beneﬁt from cooperating with more agents and demonstrates the effectiveness of our proposed algorithm.
Particularly, dual learning with N = 5 agent pairs under 8B setting sets a new record of 35 . 44 BLEU in De → En and 29 .
52 BLEU in En → De translations.
[Table]
Study on different number of agents .
We study the performances of our proposed algorithm with respect to different numbers of agent pairs, i.e., N and explore the optimal value of N .
We enumerate within N = { 1 , · · · , 5 } and did not try larger N values due to the limitation of computational resources.
As we can see from Figure 1, more agents can bring better performance for all settings.
While it appears that the performance becomes better with more pair of agents involved into the learning system, the gain becomes more and more marginal (e.g., the maximum difference for
N = 4 and N = 5 across all settings is around 0 .
1 BLEU).
Furthermore, larger N leads to higher computational costs such as GPU resources.
Thus, it is the most practical to leverage N = 3 agent pairs, where we can beneﬁt from the substantial gain over baselines without too much computational costs.
We employ N = 3 for the rest of our experiments.
Study on diversity of agents .
We further study the inﬂuence of diversity among the agents.
We show that diversity is important but not the focus of our algorithm.
Detailed results and discussion is presented in Appendix A due to the space limitation.
[Figure]
Study on translation tasks for other language pairs .
We carry out another three groups of
IWSLT14 experiments, English ↔ Spanish (Es), English ↔ Russian (Ru) and English ↔ Hebrew (He) translations to verify the generality of our framework on different languages.
As is illustrated in Table 2, we achieve consistent improvements across different translation tasks (for all languages and all translation directions), further demonstrating the effectiveness and robustness of our algorithm [Table]
[Table]
R ESULTS OF NMT ON WMT DATASET
We work with two settings on WMT En ↔ De task, where (1) only bilingual data is available; and (2) additional monolingual data is provided.
The results are summarized in Table
[Table] [Table]
{+En → De (Mono) 28 .29 .
 De → En (Mono) 32 .32 .
+} From this table, we can see that the proposed algorithm achieves the best performances for all settings, and we observe that: 1) With WMT 2014 bilingual data only, the proposed multi-agent dual learning (Dual- 3 ) achieves the state-of-the-art results to our knowledge, with 30 .
BLEU for
En → De and 33 .
32 BLEU for De
→ En 5 . 2)
Compared with the traditional two-agent dual learning in He et al.
(2016), we can achieve 0 .
61 BLEU score improvement on En → De and 0 .
33 on De
→ En with bilingual data only.
For the setting of using monolingual data, the improvements are 1 .
10 and 0 . 66 respectively.
Although KD and BT can bring improvements under different settings, the results are not as consistent as dual learning.
For example, the performance of BT is not as good as KD under bilingual setting, but is better than KD under monolingual setting.
We conjecture the reason is that with feedback signals to a generated sentence in the dual learning framework, the dataset is not only enlarged through sampling, but also guaranteed with good quality of generation.
Study on generality of the algorithm .
We apply our method to the state-of-the-art systems on
WMT 2016, 2017 and 2018 English-to-German translation challenge to verify the generality of our method.
We work with the degenerated case where f 0 is warm started with a state-of-the-art model provided by FAIR 6 and g 0 is ﬁxed.
We adopt the commercial system Google Translator (crawled on Oct 16, 2018), the champion system of WMT
En → De: MS-Marian (Junczys-Dowmunt, 2018), and FAIR’s single and ensemble models (Edunov et al., 2018a) as our baselines.
The models are evaluated by sacreBLEU 7 .
More details are left in Appendix B.
As can be seen from Table 4, our method is capable of further improving the strong model by a large margin.
Both single and ensemble models with our algorithm outperform the previous best systems and set state-of-the-art results on these tasks, demonstrating the effectiveness of our method.
[Table]
3. R ESULTS OF NMT WITH M ONOLINGUAL D ATA O NLY
Unsupervised NMT is studied recently to learn two translation models without bilingual data (details in Appendix C).
We pre-train two unsupervised NMT models with different initialization, use them to translate the 50 M monolingual sentences, and apply KD, BT and dual learning algorithms.
[Table]
As is shown in Table 5, with one pair of agents, KD-1 and BT-1 obtains almost the same results as the standard baseline, while Dual-1 achieves slightly better results on En → De translation.
When we increase the number of agents, there are substantial improvements for KD-3, BT-3 and Dual-3.
Especially, multi-agent dual learning (Dual-3) achieves 19 .
26 and 23 .
85 BLEU scores, setting new records for unsupervised NMT with pure NMT models.
A PPLICATION TO I MAGE T RANSLATION
In this section, We apply the multi-agent dual learning framework to image translation tasks.
Different from Ciregan et al.
which also uses multiple agents (several columns of deep neural networks), we leverage the duality of translation tasks and only use one model during inference.
We follow the setting of CycleGAN (Zhu et al., 2017), the most popular implementation of image translation that combines the ideas of GAN and dual learning.
[Table]
4. A DAPTION
Following the common practice in image translation (Zhu et al., 2017), for each term in Eqn.
(3), the ∆ x and ∆ y are speciﬁed as the L 1 norm of pixel-level difference between two images.
We set α = β = ( 1 N , 1 N , · · · , 1 N ) .
Since there is few literature focusing on generating consensus images using multiple models, we switch Eqn.
(3) to a simpler form:
[Equation]
where D x and D y are the datasets of domain X and domain Y .
The remaining parts of our model, including the model components, the objective functions, are exactly the same as the standard CycleGAN.
There are two discriminators d X : X (cid:55)→ [0 , 1] and d Y : Y (cid:55)→ [0 , 1] used to differentiate the generated images from natural images.
The outputs of d X and d Y represent the probability that the input image is a natural one.
The GAN loss (Goodfellow et al., 2014) is deﬁned as
[Equation]
During the optimization process, the two discriminators will try to maximize (cid:96) GAN , while the f 0 and g 0 will minimize (cid:96) dual ( D x , D y ; F α , G β ) + λ(cid:96) GAN in cooperation with other provided agents, where λ is a hyper-parameter to be tuned.
In the experiments, we follow Zhu et al.
(2017) to set λ as 10 .
4. S ETTINGS
We use the same datasets, model conﬁguration and optimization algorithm as Zhu et al.
We implement our method based on CycleGAN 8 .
We choose two tasks, the photo ↔ painting translations where the art styles include Monet, Van Gogh, Ukiyo-e, Cezanne, and cityscapes label ↔ photo translation.
All the images are cropped to 256 × 256 .
The conﬁguration details can be found at
Appendix D.1, which follows the same settings as Zhu et al.
4. R ESULTS OF P AINTING - TO -P HOTO T RANSLATION
The task of painting → photo translation is to translate an input painting with speciﬁc art style to the original photo captured by a camera.
We present case studies on painting → photo translations in
Figure 2.
Indeed, there are several cases that our algorithm is clearly better than CycleGAN:
For the images in the top-left corner, CycleGAN fails to recover the tree while our algorithm succeeds, with many details correctly included like the branches and leaves.
In the bottom right corner, the texture of the ground is kept by our results, which is missing from the images generated by CycleGAN.
We further present quantitative evaluation with Fr´echet Inception Distance (brieﬂy, FID), which is introduced in Heusel et al.
and veriﬁed to be a reasonable metric by Lucic et al.
(2018), to evaluate the quality of painting → photo translations.
FID captures the distance between generated images and the real ones, and a smaller FID score indicates the better model quality.
As is shown in Table 6, our method yields consistent improvements across all datasets.
More results in photo → painting are left in Appendix D.2 due to space limitation.
[Table]
4. R ESULTS OF C ITYSCAPES L ABEL - TO -P HOTO T RANSLATION
We present case studies and quantitative evaluation of label → photo translation on cityscapes images in Figure 3 and Table 7.
As can be seen from Figure 3, both CycleGAN and our method can transfer the input label to a matching natural scene, while results of ours are clearer with less noise.
For example, in the top left corner, our method can generate ﬂatten roads, clearer cars and buildings and the baseline model fails.
In Table 7, we present quantitative evaluation with “FCN-scores” following Isola et al.
Our method achieves better results on the three evaluation metrics.
More details about FCN-score and results on photo → label translation are left in Appendix D.3.
[Table]
C ONCLUSION
In this paper, we proposed a new framework, multi-agent dual learning, in which multiple primal models and dual models are involved in the learning process.
We empirically veriﬁed the effectiveness of the proposed framework on multiple machine translation tasks and image translation tasks.
Extending the multi-agent dual learning framework to more applications would be an interesting direction for future research.
For example, we can apply our framework to improve the recent non-autoregressive machine translation, which is known to have high inference efﬁciency yet poor translation quality (Gu et al., 2018;
Wang et al., 2019; Guo et al., 2019). It is also worth studying how to apply the framework to other dual learning paradigms, including dual supervised learning (Xia et al., 2017b) and dual transfer learning (Wang et al., 2018).
Furthermore, it is challenging yet important to improve the training efﬁciency while maintaining the substantial improvements. A CKNOWLEDGEMENT
This work is partially supported by the National Science Foundation under Grant No. 1801652 .
A S TUDY ON DIVERSITY OF AGENTS
We further study how the diversity among agents inﬂuences the performances of our algorithm on the IWSLT De ↔ En dataset.
We compare three group of agents: (1) Agents with independent runs (“ Independent Run ”) : We use the same agents as Section 3.3, which have the same model architecture and hyperparameters, and are trained through independent runs with different random seeds.
Agents with different architectures (“ Different Arch ”) : The agents have different hidden dimensions ( 256 , 512 ) with different numbers of heads ( 2 , 4 ).
They are also trained through independent runs with different random seeds.
Intuitively, this group of agents with different architectures could be more diverse than the ﬁrst group.
(3) Agents from a same run (“ Homogeneous ”)
: In comparison to the ﬁrst two groups where different methods (i.e., different random seeds, different model architecture) are used to encourage diversity among agents, in this group, we use model checkpoints at different (but very close) iterations from a same run.
At the late stage of training, the learning rate becomes smaller.
Thus the diversity among agents would naturally be very poor.
[Figure]
We train 8 -block ( 8 B) networks as Section 3.3 with N = { 1 , ..., 5 } pairs of agents from the above three groups respectively.
The same training and inference procedures as Section 3.3 are applied here.
As is shown from results in Figure 4, we have the following observations and conclusions:
Diversity among agents plays an important role in our method.
The Homogeneous agents with poor diversity has the worst performances among the three groups.
That is, introducing more homogeneous agents brings very marginal gain.
A more diverse group of agents can potentially bring further improvements, yet diversity is not the key focus of our algorithm.
Independent Run achieves comparable performances with Different Arch , suggesting that our framework can achieve substantial improvements with a reasonable level of diversity among agents.
B D ETAILS ON WMT18 E N → D E CHALLENGES
For the study on generality of the algorithm on {+WMT
 En → De+} challenges, the training data, F γ ( · ) and G γ ( · ) are the same as those used in Section 3.4.
We work with a degenerated case where [-g 
 (i.e.,-]g(i.e., the De → En model) is pre-trained before optimizing f 0 and subsequently hold constant.
The f 0 is warm started by the models from https://dl.fbaipublicfiles.com/fairseq/ models/wmt18.en-de.ensemble.tar.bz2 , which are well trained with tremendous resources.
We degenerate our algorithm to optimize the single direction due to the limitation that only
En → De model checkpoints are released.
We train the model on 8 M40 GPUs for 2 days with PyTorch implementation of our algorithm based on FairSeq Toolkit 9 .
The batch size is 4096 tokens.
We use Adam optimizer and the learning rate is automatically calculated by the system based on the reloaded checkpoint.
The dropout ratio is 0 . 3 .
We use beam search with beam size 5 to generate candidates.
C I NTRODUCTION TO U NSUPERVISED NMT
We brieﬂy introduce the recent state-of-the-art unsupervised NMT algorithm in (Lample et al., 2018).
Similar to the standard NMT model with bilingual training data, an unsupervised NMT model is also based on the encoder-decoder architecture.
The source sentences and target sentences are mapped into a same vocabulary using BPE techniques.
The shared embedding is pretrained with fastText Bojanowski et al.
(2017) to get good initial values.
The proposed model can handle both source-to-target translation and target-to-source translation like Johnson et al.
The training loss usually consists of two parts, a language model loss and a back-translation loss.
Let P
X→Y denote the translation from space X to space Y , and so for the other similar notations.
(1) Language model loss, implemented by a denoising autoencoder.
Mathematically,
[Equation]
where C ( · ) is a noise model with randomly dropping several words, swapping words, etc.
(2) Back translation loss, implemented by back-translating the monolingual data and feeding into the reversed models.
Mathematically,
[Equation]
(13) in which ˆ y ( x ) = arg max u ∈Y P
X→Y
( u | x ) and ˆ x ( y ) =
arg max v ∈X P Y→X ( v | y ) .
Note that the four P ··· ’s are implemented in a single encoder-decoder based model, where each translation task has a different “task embedding”, i.e., a learnable vector indicating the translating directions.
D I MAGE - TO -I MAGE T RANSLATION D. N ETWORK ARCHITECTURE
We follow Zhu et al.
(2017) to conﬁgure the model architectures for all tasks on image-to-image translation.
For the generator, the network contains 2 stride-2 convolutions, 9 residual blocks, andconvolutional layers with stride 12 , (i.e., the transposed convolutional layers).
Instance normalization is applied to the network.
For the discriminator, the 70 × 70 PatchGANs (Isola et al., 2017), which aim to classify whether 70 × 70 overlapping image patches are real or fake.
Such a patch-level discriminator architecture has fewer parameters than a full-image discriminator and can work on arbitrarily-sized images in a fully convolutional fashion.
D. R ESULTS OF PHOTO - TO - PAINTING TRANSLATION
We present results on photo → painting translation in Figure 5.
As we can see, both standard CycleGAN and our algorithm successfully translates the inputs to the corresponding target domains.
Our model outperforms the baseline in two aspects: 1) multi-agent dual learning can generate clearer images with less block artifact (the Monet column and Ukiyo-e column); and 2) the images generated by our algorithm are faithful to the original images in semantics.
For example, the sun is missing from CycleGAN generated images in the top Ukiyo-e column and the bottom Cezanne column, while our algorithm is able to keep the sun.
D. R ESULTS OF C ITYSCAPES PHOTO - TO - LABEL TRANSLATION
FCN Evaluation FCN-scores are used in Zhu et al.
(2017) and Isola et al.
(2017) to quantitatively evaluate Cityscape photo to semantic label translation.
A pre-trained fully convolutional network (Long et al., 2015), brieﬂy, FCN, is used to convert the photo to the corresponding semantic label, and then calculate the per-pixel accuracy, per-class accuracy and mean class IntersectionOver-Union(Class IOU) (Cordts et al., 2016) against the ground truth labels.
Results
on photo → label translation The example cases and quantitative evaluation with FCNscores of photo → labels on cityscapes images are shown in Figure 6 and Table 8 10 respectively.
We can observe improvements but the gap is not signiﬁcant.
Note that CycleGAN is a very strong baseline, and we leave further improving photo → label translation to future work.
[Table]