S IMULATED +U NSUPERVISED L EARNING W ITH A DAPTIVE D ATA G ENERATION AND B IDIRECTIONAL M APPINGS
A BSTRACT
Collecting a large dataset with high quality annotations is expensive and timeconsuming.
Recently, Shrivastava et al.
(2017) propose Simulated+Unsupervised (S+U) learning :
It ﬁrst learns a mapping from synthetic data to real data, translates a large amount of labeled synthetic data to the ones that resemble real data, and then trains a learning model on the translated data.
Bousmalis et al.
(2017b) propose a similar framework that jointly trains a translation mapping and a learning model.
While these algorithms are shown to achieve the state-of-the-art performances on various tasks, it may have a room for improvement, as they do not fully leverage ﬂexibility of data simulation process and consider only the forward (synthetic to real) mapping.
Inspired by this limitation, we propose a new
S+U learning algorithm, which fully leverage the ﬂexibility of data simulators and bidirectional mappings between synthetic and real data.
We show that our approach achieves the improved performance on the gaze estimation task, outperforming (Shrivastava et al., 2017).
I NTRODUCTION
Collecting a large annotated dataset is usually a very expensive and time-consuming task, and sometimes it is even infeasible.
Recently, researchers have proposed the use of synthetic datasets provided by simulators to address this challenge.
Not only synthetic datasets can be easily annotated but one can also generate an arbitrarily large amount of synthetic data.
In addition to that, recent advances in computer technologies enabled synthesis of high-quality data.
Speciﬁcally, a variety of computer vision applications have beneﬁtted from advanced computer graphics technologies.
For instance, Wood et al.
(2016) manipulate a 3D game graphics engine, called Unity , to synthesize photo-realistic images of human eye regions.
Then, using a million of synthetic images with labels, they achieve the state-of-the-art performance on the cross-domain appearance-based gaze estimation task (Sugano et al., 2014).
Another important application that is heavily beneﬁtting from synthetic data is autonomous driving.
A few recent works show that an inﬁnite amount of realistic driving data can be collected from high-quality video games such as GTA V (Grand Theft Auto V)
(Richter et al., 2016; Johnson-Roberson et al., 2017; Lee et al., 2017).
Speciﬁcally, Richter et al.
(2016) show that a semantic segmentation model trained only with synthetic data can even outperform the model trained with real data if the amount of synthetic data is large enough.
In (Lee et al., 2017), the authors collect a synthetic dataset of vehicle accidents, which is hardly collectable from the real world, train an accident prediction model with the synthetic data, and then apply the trained model to a real accident dataset.
As a result, they show that the model trained with large synthetic dataset can outperform the model trained within small real dataset.
Further, researchers also propose the use of simulated environments for building algorithms for autonomous drones (Microsoft, 2017), autonomous truck driving (Im, 2017), etc. [Figure] Training Testing
While the use of synthetic datasets is increasingly popular, it is not clear how one should systematically address a machine learning problem when simulated data is given with unlabeled real data.
Recently, Shrivastava et al.
(2017) propose Simulated+Unsupervised (S+U) learning , which is one the ﬁrst methodologies that guides how one can use synthetic data to improve the performance of learning algorithms.
They ﬁrst learn a translation mapping from synthetic data to real data using a modiﬁed GAN (Generative Adversarial Networks) architecture (Goodfellow et al., 2014), map the synthetic data to the real-data domain, and then train a learning model with this mapped data.
Using this methodology, they achieve the state-of-the-art performances for the cross-domain gaze estimation on the MPIIGaze dataset (Sugano et al., 2014).
A contemporary paper by Bousmalis et al.
(2017b) proposes a similar approach.
They also show how one can accommodate both the task-speciﬁc loss and the domain-speciﬁc loss to further improve the quality of image transfer.
Even though the works of (Shrivastava et al., 2017; Bousmalis et al., 2017b) present interesting solutions to deal with simulated data, their solutions have some room for improvements for the following reasons.
First, their approaches assume a ﬁxed synthetic data, and does not leverage the ﬂexibility of data simulation process.
Since the data simulator can be freely manipulated, one may hope for further performance improvements.
In addition to that, their approaches make a use of the forward (synthetic to real) mapping only, while recent works have shown the efﬁcacy of bidirectional mappings between two domains (Zhu et al., 2017a;
Kim et al., 2017).
Inspired by these limitations, we propose a new S+U learning framework, which fully leverages both the ﬂexibility of data simulators and bidirectional mappings between synthetic data and real data.
In this work, we propose a new S+U learning framework, consisting of three stages, visualized in
Fig.
The ﬁrst stage is where we fully leverage the ﬂexibility of data simulators.
We ﬁrst predict the labels of the unlabeled real data, which is available in the training phase.
We then update simulation parameters so that the synthetic label distribution matches the distribution of the predicted labels, and this procedure is repeated a few times.
The second stage learns a bidirectional mapping between the synthetic data and the real data, using the cyclic image-to-image translation frameworks (Zhu et al., 2017a;
Kim et al., 2017).
To better preserve labels while translating images, we make a few modiﬁcations to the existing framework.
In the last stage, we translate test (real) data to the synthetic domain, and then apply to them an inference model trained on the synthetic images.
Note that our approach does not require additional training of the inference model.
Our approach is superior if the backward mapping from real data to synthetic data is highly accurate and the inference model is well trained on the synthetic data.
We show that our approach achieves the improved performance on the gaze estimation task, outperforming the state of the art of (Shrivastava et al., 2017).
Our contributions can be summarized as follows.
Contributions
A new end-to-end S+U learning algorithm is proposed (Sec. 2).
We develop a simple method of exploiting the ﬂexibility of simulators (Sec. 2.2).
We propose a novel prediction method that ﬁrst translates real images to synthetic images and then applies a prediction algorithm trained on synthetic images (Sec. 2.4).
Our method outperforms the state of the art on the gaze estimation task (Sec. 3).
O UR A PPROACH 2. N OTATIONS
For notational simplicity, we denote the set of simulated images by a random variable X ∈ X and the real images by another random variable Y ∈ Y .
For the bidirectional mappings, we denote by G
X→Y the forward generator that maps simulated images to the real domain and by G Y→X the backward generator that maps real images to the simulation domain.
The content of an image, which we call label of the image , is denoted by Z .
For instance, Z denotes the digit contained in an image for the digit classiﬁcation problem, and it denotes the eye gaze vector of an eye image for the eye gaze estimation problem.
The label (or feature) extractor F ( · ) is a function that extracts the label (or feature) of a given image.
Given an unlabeled real image of Y , we call W = F ( Y ) the pseudo-label of the image.
2. A DAPTIVE DATA GENERATION BASED ON PSEUDO - LABELS
Let us ﬁrst consider how one would build a simulator that can generate an arbitrary amount of synthetic data.
For ease of explanation, imagine a simulator that generates synthetic images for the digit classiﬁcation problem.
A typical process of building a simulator is as follows.
First, one chooses P Z , the (synthetic) label distribution, arbitrarily or possibly aided with the prior knowledge on the target dataset.
Then, the simulator speciﬁes P X | Z , the image distribution conditioned on labels.
For instance, given Z = 0 , a simulator might ﬁrst draw a perfect ellipsoid and then add some pixel noise to reﬂect the diversity of images.
As illustrated above, a simulator is usually fully speciﬁed with the label distribution and the data (image) distribution conditioned on each label.
A na ¨ ıve choice of taking arbitrary distributions may result in a distributional gap between the synthetic dataset and real dataset, i.e., sample selection bias (Huang et al., 2007).
Fortunately, for most existing simulators, it is easy to adjust the label distribution or P Z .
For instance, UnityEyes is a high-resolution 3D simulator that can render realistic human eye regions (Wood et al., 2016).
The label is a 4 -dimensional vector, and the marginal distribution of each element can be specifeid by the user.
See Sec.
3 for more details.
On the other hand, the way an image is rendered given a label or P X | Z is not modiﬁable.
Motivated by this observation, we propose a simple iterative algorithm that can be used to ﬁnd a good set of distribution parameters for the data generator.
We note that the approach of (Shrivastava et al., 2017) does not adjust the data generation process, and stick with an arbitrary label distribution, say P Z (0) .
Our algorithm is based on the novel use of pseudo-labels (Lee, 2013).
Pseudo-label is originally proposed to tackle semi-supervised learning problems in which a large number of unlabeled data is provided with a small number of labeled data: Once a coarse predictor is trained with the small labeled dataset, one can assign pseudo-labels to the large unlabeled dataset using the prediction results of the coarse predictor.
We now describe our algorithm, visualized in Fig.
It ﬁrst starts with an arbitrary label distribution
P Z (0) (and hence an arbitrary data distribution P X (0) ), and then trains a predictor F (0) using the generated data.
Once the predictor F (0) is trained, it computes the empirical distribution of the pseudo-labels of the unlabeled real dataset, i.e., W (0) =
F (0) ( Y ) .
Finally, it ﬁnds the new label distribution P Z (1) for the simulator by minimizing the distance to the pseudo-label distribution P W (0)
( ✓, )=(12 .
5 ,  7 .
8) ( ✓, )=(  37 .
9 ,  7 . 1) ( ✓, )=(7 . 4 , 27 . 2) ( ✓, )=(6 . 93 , 12 .
8) (ˆ ✓, ˆ  )=(
0 .
5 , 1 . 2) (ˆ ✓, ˆ  )=(  10 . 6 , 1 . 0)
(ˆ ✓, ˆ  )=(  11 . 0 , 12 .
8) (ˆ ✓, ˆ  )=(  16 .
2 , 6 . 4) ( ✓, )=(12 .
5 ,  7 .
8) ( ✓, )=(  37 .
9 ,  7 . 1) ( ✓, )=(7 . 4 , 27 . 2) ( ✓, )=(6 . 93 , 12 .
F (2) Generate synthetic data ( X,Z ) (3) Train a predictor F on ( X,Z ) (4) Pseudo-label real data: ( Y,F ( Y ))
(5) Estimate distribution of F ( Y ) (1) Set distribution of Z ˆ ✓ ˆ  P Z
=
P
( ✓, ) ✓
P F ( Y ) =
P (ˆ ✓, ˆ  )  simulator parameters.
1 This procedure can be repeated a few times.
2. L ABEL - PRESERVING IMAGE - TO - IMAGE TRANSLATION
CycleGAN is an unsupervised image-to-image translation method based on GANs (Zhu et al., 2017a).
In this section, we ﬁrst observe that image content, i.e., labels of the image , may alter when translated via the original CycleGAN framework.
In order to mitigate this problem, we propose a slight modiﬁcation to the CycleGAN framework by employing the concept of “content representation” (Gatys et al., 2016).
The key idea is simple: Given a label (content) extractor F ( · ) , we simply regularize the difference between the labels of the original image and its translated image.
Note that this idea is also called perceptual loss or feature matching (Gatys et al., 2017; Bousmalis et al., 2017a), and a similar idea has been applied in other works on unsupervised image-to-image translations (Shrivastava et al., 2017; Taigman et al., 2016).
To illustrate the limitation of the CycleGAN framework, we ﬁrst formally describe the CycleGAN framework.
The goal is to ﬁnd G X→Y , the forward generator from X to Y , and G Y→X , the back- ward generator from
Y to X .
These generators, or translation mappings, take images from the source domain and convert them to the target domain without altering the image content.
Using the GAN framework of Goodfellow et al.
(2014), we train these generators jointly with discriminators: D Y ( D X ) outputs the probability that the given image is from Y ( X ).
Using these notations, the standard
GAN loss for the forward mapping can be written as
[Equation]
The backward GAN loss is similarly deﬁned with ( X, X ) and ( Y, Y ) being interchanged.
Further, the CycleGAN framework has an additional loss term, called the cycle-consistency loss:
[Equation]
where (cid:107)·(cid:107) can be an arbitrary norm, e.g., (cid:96) 1 norm.
This loss essentially imposes a restriction on the forward/backward mappings so that G Y→X ( G X→Y ( X )) (cid:39) X , and similarly,
G X→Y ( G Y→X ( Y ))
(cid:39) Y .
Combining these terms and omitting the function arguments for sim- plicity, the overall loss function is as follows:
[Equation]
[Figure]
[Figure]
where λ cyc is the regularization parameter.
Then, one can apply the standard training methodology of GANs to ﬁnd G ’s that minimize the loss function and D ’s that maximize it.
While the original CycleGAN is observed to well preserve labels under certain scenarios, its theoretical understanding is still missing.
Further, the authors of the CycleGAN paper also acknowledged the potential limitation of the original approach: they made a remark that their method sometimes permutes the labels for tree and building when applied to the cityscapes photos, and provided failure cases (Zhu et al., 2017b).
Indeed, we also observe that it frequently fails to maintain labels: See
Sec.
B for some examples.
Indeed, it can be explained via a simple example why the cycle-consistency loss alone can fail to preserve labels.
See Fig.
3 for a toy example where the goal is to learn an image-to-image translation rule between digit images in Arial font ( X ) and those in Script font ( Y ), where the label is the number present in the image.
Consider the G X→Y
and G Y→X shown as arrows in the ﬁgure.
Note that the labels, or the digits contained in images, are not preserved but permuted.
However, observe that these mappings result in perfect cycle-consistency, i.e., L cyc = 0 .
Further, if the discriminators
D X and D Y are perfect, L GAN, forward =
L GAN, backward = 0 .
This implies that the zero loss can be achieved even when labels are not preserved at all.
In general, with n samples in each domain, there are at least n !
− 1 pairs of mappings that attain zero loss without preserving labels.
As mentioned in the beginning of the section, we employ the feature-consistency loss in order to better preserve labels.
Speciﬁcally, we use the bidirectional feature-consistency loss, deﬁned as
[Equation]
where (cid:107)·(cid:107) is an appropriate norm of the feature space.
We then use the following total loss function:
[Equation]
Going back to the toy example, assume that we are given with a perfect digit classiﬁer F .
With the feature consistency loss term added to the objective function, the mappings shown in Fig.
3 will incur non-zero loss.
In order to achieve zero loss, one must ﬁnd the correct bidirectional mapping between the two domain that preserve labels.
Hence, generators will attempt at maintaining the labels between image pairs to minimize the loss.
2. P REDICTION WITH BACKWARD TRANSLATION
After the translation mappings are obtained, there are two possible approaches that one can take.
The ﬁrst approach, which is the ﬁrst simulated+unsupervised learning approach to image-to-image translations, is proposed by Shrivastava et al.
; Bousmalis et al. (2017b).
It ﬁrst translates all the labeled synthetic data using the forward mapping G X→Y , and then train a model on this real-like (labeled) dataset.
Both works show that this approach signiﬁcantly outperforms the previous state of the arts for diverse tasks.
See Fig.
4a for visual illustration.
The second approach, which we propose in this work, relies on the backward translation, i.e.,
G Y→X .
As described in the previous section, leveraging simulators, one can generate an arbitrarily large amount of synthetic data, and then train an efﬁcient model on it.
Hence, assuming perfect
[Table]
[Table]
F (
[Table]
F ( ✓, )=(12 .
5 ,  7 .
8) ( ✓, )=(
( ✓, )=(7 . 4 , 27 . 2) ( ✓, )=(
( ✓,
[Table]
[Table]
F (
[Table]
F ( [Table] F (
[Table] [Table] [Table]
F ( [Table] [Table] /
[Table] [Table] ✓ [Table]
[Table] ( ✓, ) [Table]
[Table]
[Table]
✓, )= [Table]  1 / [Table]
[Table]
=
[Table]
[Figure]
backward translation, one can ﬁrst translate real data to synthetic data, and then apply the model (trained on synthetic data) to the mapped data.
See Fig.
4b for visual illustration.
Indeed, in most practically relevant cases, the amount of available synthetic data is much larger than that of real data, and hence one can easily train a highly accurate predictor at least in the synthetic domain (but not necessarily in the real domain).
Another advantage of this approach is that one does not have to train a new model when a new real dataset arrives, while the ﬁrst approach needs to retrain the prediction model for each real dataset.
We remark that this approach requires an additional computational cost for backward translation during prediction.
However, the additional computational cost is usually not prohibitive compared to the complexity of the prediction model.
E XPERIMENTS 3. C ROSS - DOMAIN APPEARANCE - BASED GAZE ESTIMATION
In this section, we apply our methodology to tackle the cross-domain appearance-based gaze estimation problem, and evaluate its performance on the the MPIIGaze dataset (Zhang et al., 2015).
The goal of this problem is to estimate the gaze vector given an image of a human eye region using the data collected from a different domain.
To generate a synthetic dataset of labeled human eye images, we employ UnityEyes , a highresolution 3D simulator that renders realistic human eye regions (Wood et al., 2016).
For each image,
UnityEyes draws the pitch and yaw of the eyeball and camera, uniformly at random, and then renders the corresponding eye region image.
More speciﬁcally, the random distributions are speciﬁed byinput parameters ( θ p , θ y , φ p , φ y , δθ p , δθ y , δφ p , δφ y ) :
The ﬁrst two are the expected values of eyeball pitch and yaw, and the following two are the expected values of camera pitch and yaw; And the other four are the half-widths of the distributions.
As a default setting, UnityEyes uses the following distribution parameters: ( θ p , θ y , φ p , φ y , δθ p , δθ y , δφ p , δφ y ) =
(0 ◦ , 0 ◦ , 0 ◦ , 0 ◦ , 30 ◦ , 30 ◦ , 20 ◦ , 40 ◦ ) .
More details on how UnityEyes render eye regions can be found in (Wood et al., 2016).
We ﬁrst evaluate the performance of our data generation algorithm based on pseudo-labels.
Using the 380 k images generated with UnityEyes with the default parameters, we train a simple gaze estimation network, with which we annotate the real dataset with pseudo-labels.
We then obtain the horizontal and vertical statistics of the gaze vectors, and accordingly adjust the UnityEyes parameters to make the means and variances coincide.
Since there are many free parameters, we reduce the number of free parameters by considering the following speciﬁc classes of distributions:
[Equation]
We run the data adaption algorithm for 4 iterations, and report in Table 1 the sequences of means and standard deviations, relative to the ground truth.
Further, we also run our algorithm starting from each label distribution and report the validation/test errors in the table.
We observe that after the second iteration of the algorithm, the validation/test errors are minimized.
Note that in this experiment, the validation error increases after the second iteration in our experiment, implying that the ﬁrst stage of our algorithm may not converge well in practice.
Hence, one needs to use a small validation set to choose which iteration to proceed with.
In the rest of this section, we report the results based on the output of the second iteration.
Using the newly generated synthetic dataset together with the unlabeled real dataset, we then train a cyclic image-to-image translation mapping with the new loss function (5).
For the choice of the feature, we set the identity mapping as the feature extractor since two domains share the high-level structure and differ only in detailed expressions.
We use the (cid:96) 1 norm for the feature-consistency term.
For the choice of regularization parameters, we test the performance of our algorithm with λ cyc ∈ { 0 , 1 , 5 , 10 , 50 } and λ feature ∈ { 0 , 0 . 1 , 0 .
The test results are summarized in Table 3 in Sec.
C.1.
As a result, we observe that λ cyc = 10 and λ feature = 0 .
5 achieved the best performance.
Shown in Fig.
5 are some examples of the translation results.
Here, we compare the CycleGAN trained only with the cycle-consistency loss and that trained with both the cycle-consistency loss and the feature-consistency loss.
The ﬁrst column shows the input real images, the next two columns are the translated images using the output of the former approach, and the last two columns are the outputs of the latter approach.
It is clear that the the learned mappings satisfy near-perfect cycle-consistency by comparing Y ’s with G X→Y
( G Y→X ( Y )) ’
s.
However, the gaze vector is not preserved when real images are translated to synthetic images.
On the other hand, when both consistencies are enforced, the gaze vector does not alter, maintaining the label information, showing the necessity of the feature-consistency loss term.
Summarized in Table 2 are the experimental results along with the state of the arts reported in the literature.
We ﬁrst run the ﬁrst two stages of our algorithm using the optimal regularization parameters.
To see the gain of the adaptive data generation and the bidirectional mapping, we ﬁrst apply the forward mapping to the simulated data to obtain reﬁned synthetic (RS) images.
Then, we train a new predictor on these RS images, and then apply this predictor to the test images.
Note that this can be viewed as a combination of the ﬁrst two stages of our algorithm and the original
S+U learning approach.
Using this approach, we obtain the test error of 7 . 71 , outperforming the state-of-the-art performance of Shrivastava et al.
Note that this performance improvement is due to our adaptive data generation algorithm and the use of bidirectional mapping.
Further, our algorithm is trained only with 380 k images while the results of Shrivastava et al.
(2017) is trained with 1200 k synthetic images, demonstrating the sample-efﬁciency of our algorithm.
Further, instead of reﬁning synthetic images and training a new predictor on that, we apply the backward mapping to the real data and apply the predictor trained on the synthetic data.
As a result, we achieve the test error of 7 . 60 , showing that our backward mapping can further improve the prediction performance.
3. I MPLEMENTATION DETAILS
For our experiments, we slightly modify the CycleGAN framework of Zhu et al. (2017a).
The generator network, G , is modiﬁed to take input image of 36 × 60 , and the rest of the G is identical to that of the original CycleGAN architecture with 6 blocks.
The discriminator network, D is identical [Table] =(cid:96) =(cid:96) =(cid:96) =(cid:96) =| E [Θ ( (cid:96) ) ] − E [ ˆΘ (cid:63) ] | 8.
| σ (Θ ( (cid:96) ) )
− σ ( ˆΘ (cid:63) ) | 14.
| E [Φ ( (cid:96) ) ] − E [ ˆΦ (cid:63) ] | 0.
| σ (Φ ( (cid:96) ) )
− σ ( ˆΦ (cid:63) )
| 20.
Val./Test error 22 . 88 / 22 .9 .
[Figure]
G ’s are trained
’s
Method
Trained on
Tested with
Error
Support Vector Regression (SVR) (Schneider et al., 2014)
R
R 16 .
Adaptive Linear Regression (ALR) (Lu et al., 2014)
R
R 16 .kNN
w/ UT Multiview (Zhang et al., 2015)
R
R 16 .
Random Forest (RF) (Sugano et al., 2014)
R
R 15 .
CNN w/ UT Multiview (Zhang et al., 2015)
R
R 13 .
CNN w/ UnityEyes (Shrivastava et al., 2017)
S
R 11 .kNN
w/ UnityEyes
(Wood et al., 2016)
S
R 9 .
SimGAN (Shrivastava et al., 2017)
RS
R 7.
Ours (Adaptive data generation + Bidirectional + Forward)
RS
R 7 .
Ours (Adaptive data generation + Bidirectional + Backward)
S
RR 7 .to
that of the original CycleGAN framework.
For detailed information, we refer the readers to (Zhu et al., 2017a).
The CycleGAN was trained with batch size of 64 and learning of 2 × 10 − 4 .
The eye gaze prediction network is designed based on the architecture proposed in (Shrivastava et al., 2017).
The input is a 36 × 60 gray-scale image that is passed through 5 convolutional layers followed by 3 fully connected layers, the last one encoding the 3 -dimensional (normalized) gaze vector: (1) Conv3x3, 32 feature maps, (2) Conv3x3, 32 feature maps, (3) Conv3x3, 64 feature maps, (4) MaxPool3x3, stride = 2 , (5) Conv3x3, 80 feature maps, (6) Conv3x3, 192 feature maps, (7) [Table] [Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] loss.
The predictor network is trained with batches of size 512 , until the validation error converges.
C ONCLUSION
In this work, we propose a new S+U learning algorithm, which fully exploits the ﬂexibility of simulators and the recent advances in learning bidirectional mappings, and show that it outperforms the state-of-the-art performance on the gaze estimation task.
We conclude the paper by enumerating a few open problems.
In our experiments, we arbitrarily choose the feature for the consistency term.
We, however, observe that the choice of feature mapping signiﬁcantly affects the performance of the algorithm.
Thus, one needs to study how the optimal feature mapping can be designed for different tasks.
In this work, we have separated learning of cross-domain translation mappings from learning of a prediction algorithm.
One interesting open question is whether one can jointly train these two components and potentially outperform the current separation-based approach.
Another interesting topic to study is the regularization techniques for the adaptive data generation method.
For instance, if the real data provided in the training set is not representative enough, our approach, which tries to match the synthetic label distribution with the real one, may not be able to generalize well on unseen data.
One may address this limitation by incorporating prior knowledge about the label distributions or manually tune the simulation parameters.
A thorough study is needed to understand how one could obtain diverse synthetic images via such methods in a systematic way.
Further, our adaptive data generation method assumes that the predictor trained on simulated data works quite well on real data.
In other words, if the predictor trained solely on simulated data provide completely wrong pseudo-labels, matching the synthetic label distribution with the pseudolabel distribution may not be helpful at all.
For instance, when we pseudo-label the images in the
Street View House Numbers (SVHN) dataset using a digit classiﬁer that is trained on the MNIST dataset, the resultant pseudo-label distribution is observed to be useless to reﬁne the synthetic label distribution (Netzer et al., 2011; LeCun et al., 1998).
It is an interesting open question whether or not one can devise a similar adaptive data generation method for such cases.
Building a differentiable data generator is also an interesting topic (Graves et al., 2014; Gaunt et al., 2017; Feser et al., 2016).
It is well known that neural networks with external memory resources is a differentiable Turing Machine or differentiable Von Neumann architecture (Graves et al., 2014).
Further, researchers have proposed the use of differentiable programming language with neural networks Gaunt et al.
; Feser et al.
Indeed, the adaptive data generation algorithm proposed in this work can be viewed as an extremely limited way of adjusting the way synthetic data is generated.
If the data generator can be written in a differentiable language, one could possibly jointly optimize the synthetic data generator together with the other components such as translation networks and prediction networks, potentially achieving an improved performance
A A DDITIONAL E XPERIMENTAL R ESULTS ON E YE G AZE E STIMATION
In this section, we provide additional qualitative results on the eye gaze estimation task.
See Fig. 6.
[Figure]
B A DDITIONAL E XPERIMENTAL R ESULTS ON GTA D ATA
We learn a bidirectional mapping between the driving images collected from the video game GTA
V and the KITTI driving dataset (Geiger et al., 2012).
Here, we use the last fully connected layer of resnet18 as the feature mapping (He et al., 2016).
Shown in Fig.
7 and Fig.
8 are some qualitative mapping results, proving the efﬁcacy of the feature-consistency loss.
[Figure]
[Figure]
C A DDITIONAL E XPERIMENTS C. O PTIMAL C HOICE OF R EGULARIZATION P ARAMETERS
When training the bidirectional mappings, one needs to choose the values of λ cyc and λ feature .
In this section, we report the test errors measured with different choices of these regularization parameters in Table 3.
As a result, we observe that the choice ( λ cyc = 10 , λ feature = 0 . 5) obtains the best performance.
[Table]
C. P ARAMETER R EDUCTION
Since the number of observed variables is larger than the number of simulation parameters
[Table]
we reduced the number of free parameters by using following reduction: θ p = φ p , θ y = φ y , δθ p = δφ p , δθ y = δφ y .
Here, we conduct an additional experiment with a different reduction method: 2 θ p = φ p , 2 θ y =
φ y , 2 δθ p = δφ p , 2 δθ y
=
δφ y .
We adapt the data distribution with this new reduction method for 4 times.
As a result, we obtain the same test error performance with this different reduction method, implying that the reduction method does not much affect the overall performance.
[Table]