Original: M ETA -FAVAE: T OWARD F AST AND D IVERSE F EWSHOT I MAGE G ENERATION VIA M ETA -L EARNING AND F EEDBACK A UGMENTED A DVERSARIAL VAE
Revised: M ETA -FAVAE: T OWARD F AST AND D IVERSE F EWSHOT I MAGE G ENERATION VIA M ETA -L EARNING AND F EEDBACK A UGMENTED A DVERSARIAL VAE

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning to synthesis realistic images of new categories based on just one or a few examples is a challenge task for deep generative models, which usually require to train with a large amount of data.
Revised: Learning to synthesis realistic images of new categories based on just one or a few examples is a challenge task for deep generative models, which usually require to train with a large amount of data.

Original: In this work, we propose a data efﬁcient meta-learning framework for fast adapting to few-shot image generation task with an adversarial variational auto-encoder and feedback augmentation strategy.
Revised: In this work, we propose a data efficient meta-learning framework for fast adapting to few-shot image generation task with an adversarial variational auto -encoder and feedback augmentation strategy.

Original: By training the model as a meta-learner, our method can adapt faster to the new task with signiﬁcant reduction of model parameters.
Revised: By training the model as a meta-learner, our method can adapt faster to the new task with significant reduction of model parameters.

Original: We designed a novel feedback augmented adversarial variational auto-encoder.
Revised: We designed a novel feedback augmented adversarial variational auto-encoder.

Original: This model learns to synthesize new samples for an unseen category just by seeing few examples from it and the generated interpolated samples are then used in feedback loop to expand the inputs for encoder to train the model, which can effectively increase the diversity of decoder output and prevent the model overﬁtting with insufﬁcient samples of unseen category.
Revised: This model learns to synthesize new samples for an unseen category just by seeing few examples from it and the generated interpolated samples are then used in feedback loop to expand the in- puts for encoder to train the model which can effectively increase the diversity of decoder output and prevent the model overfitting with insufficient samples of unseen category.

Original: Additionally, with the dual concatenation of latent code and random noise vectors, this method can be generalized to more complex color images compared to existing meta-learning based methods.
Revised: Additionally, with the dual concatenation of latent code and random noise vectors this method can be generalized to more complex color images compared to existing meta learning based methods

Original: Experimental results show that our model can have much faster adaption to new generation tasks of unseen categories while generating high-quality and diverse images on three datasets
Revised: Experimental results show that our model can have much faster adaption to new generation tasks of unseen categories while generating high-quality and diverse images on three datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Following the great success of deep learning, deep generative models, such as Generative Adversarial Network (GAN)( Goodfellow et al.
Revised: Following the great success of deep learning, deep generative models, such as Generative Adversarial Network (GAN)( Goodfellow et al.

Original: and Variational Autoencoder (VAE)( Kingma &
Revised: and Variational Autoencoder (VAE)( Kingma &

Original: Welling (2014)), achieved tremendous progress in the task of generating realistic image by training with large-scale datasets.
Revised: Welling (2014)), achieved tremendous progress in the task of generating realistic image by training with large-scale datasets.

Original: However, due to the high expense of collecting and annotating a large amount of data, given only one or a few images from a category, it is still a challenging task for the typical deep generative models to learn fast and adapt to synthesis realistic and diverse images.
Revised: However, due to the high expense of collecting and annotating a large amount of data, given only one or a few images from a category, it is still a challenging task for the typical deep generative models to learn fast and adapt to synthesis realistic and diverse images.

Original: This task is referred to as few-shot image generation(FSIG).
Revised: This task is referred to as few-shot image generation(FSIG).

Original: Several recent efforts have been devoted to improve the learning efﬁciency of models for the FSIG task.
Revised: Several recent efforts have been devoted to improve the learning efficiency of models for the FSIG task.

Original: These approaches can roughly fallen into three categories, that are transformation-based methods, fusion-based methods and optimization-based methods.
Revised: These approaches can roughly fallen into three categories, that are transformation-based methods, fusion-based methods and optimization-based methods.

Original: For the transformation-based methods, DAGAN( Antoniou et al.
Revised: For the transformation-based methods, DAGAN( Antoniou et al.

Original: (2018)) is an GAN-based approach that captures cross-class transformations on one conditional image and injects random vectors into the generator for generating different images.
Revised: (2018)) is an GAN-based approach that captures cross-class transformations on one conditional image and injects random vectors into the generator for generating different images.

Original: However, the diversity of generated images from this method is quite limited and it fails to capture the information from multiple images in the same category.
Revised: However, the diversity of generated images from this method is quite limited and it fails to capture the information from multiple images in the same category.

Original: Among the fusion-based methods, GMN( Bartunov & Vetrov (2018)) is inspired by the matching network( Vinyals et al.
Revised: Among the fusion-based methods, GMN( Bartunov & Vetrov (2018)) is inspired by the matching network( Vinyals et al.

Original: and combines a matching procedure with VAE.
Revised: and combines a matching procedure with VAE.

Original: Due to the limited generation capacity of VAE, this method can only generate28x28 binary digits.
Revised: Due to the limited generation capacity of GMN, this method can only generate binary digits.

Original: For generating more diversiﬁed and realistic images, MatchingGAN( Hong et al.
Revised: For generating more diversified and realistic images, MatchingGAN( Hong et al.

Original: (2020)) combines adversarial learning and matching generators and is capable to fully exploit several conditional images from the same class.
Revised: (2020)) combines adversarial learning and matching generators and is capable to fully exploit several conditional images from the same class.

Original: Nevertheless, this method still struggles with more complex natural color image generation and training these models is time-consuming due to complexity of the networks.
Revised: Nevertheless, this method still struggles with more complex natural color image generation and training these models is time-consuming due to complexity of the networks.

Original: Optimization-based methods are proposed based on the concept of meta-learning and initialize a generator with images from seen categories and ﬁne-tuning the trained model with images from each unseen category.
Revised: Optimization-based methods are proposed based on the concept of meta-learning and initialize a generator with images from seen categories and fine-tuning the trained model with images from each unseen category.

Original: Clouˆatre & Demers (2019) proposed a FIGR by applying meta-learning algorithm Reptile( Nichol et al.
Revised: Clouˆatre & Demers (2019) proposed a FIGR by applying meta-learning algorithm Reptile( Nichol et al.

Original: (2018)) to GAN.
Revised: (2018)) to GAN.

Original: It can signiﬁcantly reduce the 
Revised: It can significantly reduce the trainable parameters of the model and fast adapt to the new generation task in the few-shot setting.

Original: DAWSON Liang et al.
Revised: DAWSON Liang et al.

Original: (2020) is another method incorporating meta-learning model of MAML Finn et al.
Revised: (2020) is another method incorporating meta-learning model of MAML Finn et al.

Original: (2017) with GAN for the few-shot generation task.
Revised: (2017) with GAN for the few-shot generation task.

Original: Nevertheless, the DAWSON generates images without diversity and is inapplicable for color images.
Revised: Nevertheless, the DAWSON generates images without diversity and is inapplicable for color images.

Original: Both models adopted a similar idea of meta-learning method, but can hardly produce sharp and realistic images or adapt to complex natural color images.
Revised: Both models adopted a similar idea of meta-learning method, but can hardly produce sharp and realistic images or adapt to complex natural color images.

Original: FAML Phaphuangwittayakul et al.
Revised: FAML Phaphuangwittayakul et al.

Original: (2021) is another meta-learning inspired method that can signiﬁcantly reduce the model parameters and fast adapt to generate images from color image dataset.
Revised: (2021) is another meta-learning inspired method that can significantly reduce the model parameters and fast adapt to generate images from color image dataset.

Original: This simple and effective model increased the number of generator iterations and utilized conditional feature vectors.
Revised: This simple and effective model increased the number of generator iterations and utilized conditional feature vectors.

Original: However, given the insufﬁcient input of unseen category, only conditioning the feature vectors from the encoder can not guarantee the exploitation of the key input features.
Revised: However, given the insufficient input of unseen category, only conditioning the feature vectors from the encoder can not guarantee the exploitation of the key input features.

Original: Moreover, training GAN-based model is time-consuming and mode collapse issues are remained, while VAEs have limited capacity of generating high-quality images.
Revised: Moreover, training GAN-based model is time-consuming and mode collapse issues are remained, while VAEs have limited capacity of generating high-quality images.

Original: To tackle the above problems, we propose a Meta-learning-based Feedback augmented Adversarial
Revised: To tackle the above problems, we propose a Meta-learning-based Feedback augmented Adversarial

Original: Variational Auto-Encoder(Meta-FAVAE),that learns optimal model parameters to fast adapt to fewshot image generation task and increases the diversity by a feedback augmentation strategy.
Revised: Variational Auto-Encoder (Meta-FAVAE),that learns optimal model parameters to fast adapt to fewshot image generation task and increases the diversity by a feedback augmentation strategy.

Original: This model is trained in a meta-learning mechanism with an inner-loop and outer-loop.
Revised: This model is trained in a meta-learning mechanism with an inner-loop and outer-loop.

Original: By integrating meta-learning, this method has less than 10M parameters and thus leads to a faster adaption to the new generation tasks of unseen categories.
Revised: By integrating meta-learning, this method has less than 10M parameters and thus leads to a faster adaption to the new generation tasks of unseen categories.

Original: In order to combine the advances from VAE and GAN ( Plumerault et al.
Revised: In order to combine the advances from VAE and GAN ( Plumerault et al.

Original: , we introduced an optimal adversarial variational auto-encoder to generate high-quality images while retrieving key latent features.
Revised: , we introduced an optimal adversarial variational auto-encoder to generate high-quality images while retrieving key latent features.

Original: More speciﬁcally, the model utilized
Revised: More specifically, the model utilized

Original: Relativistic average GAN ( Jolicoeur-Martineau (2018)) and beta-VAE ( Higgins et al.
Revised: Relativistic average GAN ( Jolicoeur-Martineau (2018)) and beta-VAE ( Higgins et al.

Original: to signiﬁcantly improve data quality and stability of the generated images with 64*64 size outputs.
Revised: to significantly improve data quality and stability of the generated images with 64 × 64 size outputs.

Original: By introducing a novel feedback augmentation technique, we can signiﬁcantly increase the diversity of the generated images by adding interpolated samples to the inputs, which can also prevent model collapse.
Revised: By introducing a novel feedback augmentation technique, we can significantly increase the diversity of the generated images by adding interpolated samples to the inputs, which can also prevent model collapse.

Original: Additionally, we utilize the dual fusion of latent code and random noise vectors and mode-seeking regularization term from FAML for improving the generalization of few-shot image generation task to natural color images.
Revised: Additionally, we utilize the dual fusion of latent code and random noise vectors and mode-seeking regularization term from FAML for improving the generalization of few-shot image generation task to natural color images.

Original: Extensive experiments validate the superiority of our Meta-FVAE in terms of fast adaptation and diverse image generation over other state-of-the-art methods. 
Revised: Extensive experiments validate the superiority of our Meta-FAVAE in terms of fast adaptation and diverse image generation over other state-of-the-art methods. also Dec a

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: Figure 1 illustrates the pipeline of proposed model in this study.
Revised: Figure 1 illustrates the pipeline of proposed model in this study.

Original: Given a set of tasks { T i } mi =1 containing various tasks where each task τ is an image generation for one class of images ( X τ ) .
Revised: Given a set of tasks { T i } mi =1 containing various tasks where each task τ is an image generation for one class of images ( X τ ) .

Original: The model can generate a set of images for the unseen classes.
Revised: The model can generate a set of images for the unseen classes.

Original: This framework is composed of two main phase, which are training phase and generation phase.
Revised: This framework is composed of two main phase, which are training phase and generation phase.

Original: During the inner loop of the training phase, the encoder Enc 1 of VAE extracts mean µ and variance σ from images x of the training task to form a latent code c .
Revised: During the inner loop of the training phase, the encoder Enc 1 of VAE extracts mean µ and variance σ from images x of the training task to form a latent code c .

Original: Then, the decoder Dec 1 of VAE reproduces images ˆ x from the latent code.
Revised: Then, the decoder Dec 1 of VAE reproduces images ˆ x from the latent code.

Original: The generator G 1 takes the latent code concatenated with two random noise vectors ( z 1 and z 2 ) as the input and generates output images, y 1 =
Revised: The generator G 1 takes the latent code concatenated with two random noise vectors ( z 1 and z 2 ) as the input and generates output images, y 1 =

Original: G 1 ( z 1 , c ) and y 2 =
Revised: G 1 ( z 1 , c ) and y 2 =

Original: G 1 ( z 2 , c ) .
Revised: G 1 ( z 2 , c ) .

Original: The discriminator network D 1 distinguishes the real images (input x ) and fake images (output samples y from generator network G 1 ).
Revised: The discriminator network D 1 distinguishes the real images (input x ) and fake images (output samples y from generator network G 1 ).

Original: The Adam optimizer Kingma & Ba (2014) is used to optimize the encoder Enc 1 , decoder Dec 1 , generator G 1 , and discriminator D 1 networks in the loss computation.
Revised: The Adam optimizer Kingma & Ba (2014) is used to optimize the encoder Enc 1 , decoder Dec 1 , generator G 1 , and discriminator D 1 networks in the loss computation.

Original: The objective functions to calculate the loss of Enc 1 , Dec 1 , G 1 , and D 1 are deﬁned as follows.
Revised: The objective functions to calculate the loss of Enc 1 , Dec 1 , G 1 , and D 1 are defined as follows.

Original: Instead of training the model with only input images as traditional method, we increase the number of training images by fusing the interpolated samples.
Revised: Instead of training the model with only input images as traditional method, we increase the number of training images by fusing the interpolated samples.

Original: The interpolated samples are produced based on the images generated by our model.
Revised: The interpolated samples are produced based on the images generated by our model.

Original: The linear interpolation is performed based on the four generated images in each episode.
Revised: The linear interpolation is performed based on the four generated images in each episode.

Original: The linear interpolation is a technique to visualize the data in a subspace.
Revised: The linear interpolation is a technique to visualize the data in a subspace.

Original: With two images multiplied by the interpolation coefﬁcients ranging in [0.1,0.9], the new examples between these two images can be explored in the space of generated images.
Revised: With two images multiplied by the interpolation coefficients ranging in [0.1,0.9], the new examples between these two images can be explored in the space of generated images.

Original: The total number of interpolated samples are equal to loop n 2 , where n > 0 .
Revised: The total number of interpolated samples are equal to loop n 2 , where n > 0 .

Original: For example, a total of 16 interpolated images with n = 4 are produced by random images generated by the latent code of Meta-AVAE and four random noise vectors.
Revised: For example, a total of 16 interpolated images with n = 4 are produced by random images generated by the latent code of Meta-AVAE and four random noise vectors.

Original: Once the interpolated images are generated, these images are used to combine with the input images and step the model with the inner loop gradient one more episode.
Revised: Once the interpolated images are generated, these images are used to combine with the input images and step the model with the inner loop gradient one more episode.

Original: To train VAE for extracting the latent code and producing reconstructed images, a reconstruction loss and a Kullback-Leibler (KL) divergence Larsen et al.
Revised: To train VAE for extracting the latent code and producing reconstructed images, a reconstruction loss and a Kullback-Leibler (KL) divergence Larsen et al.

Original: (2016) loss are computed as equation (1).
Revised: (2016) loss are computed as equation (1).

Original: [Equation]
Revised: [Equation]

Original: where g denotes the number of images sampled from a class of images X τ .
Revised: where g denotes the number of images sampled from a class of images X τ .

Original: β indicates adjustable hyperparameter, set to 10.
Revised: β indicates adjustable hyperparameter, set to 10.

Original: The loss functions of both discriminator and generator followed the two random noise vectors z 1 and z 2 as follows.
Revised: The loss functions of both discriminator and generator followed the two random noise vectors z 1 and z 2 as follows.

Original: The Discriminator objective function is:
Revised: The Discriminator objective function is:

Original: [Equation]
Revised: [Equation]

Original: Thus, L disc is the average of L disc, 1 and L disc, 2 and the Generator objective function is as below and L gen is the average of L gen, 1 and L gen, 2 :
Revised: Thus, L disc is the average of L disc, 1 and L disc, 2 and the Generator objective function is as below and L gen is the average of L gen, 1 and L gen, 2 :

Original: [Equation]
Revised: [Equation]

Original: For each sampled task τ including training and test tasks, Meta-FAVAE ﬁrst initializes and then minimizes the parameters of the encoder Φ 1 , enc , decoder Φ 1 , dec , discriminator Φ 1 , disc , and generator Φ 1 , gen in the K inner loop through objective functions L V AE , L disc , and L gen , respectively.
Revised: For each sampled task τ including training and test tasks, Meta-FAVAE first initializes and then minimizes the parameters of the encoder Φ 1 , enc , decoder Φ 1 , dec , discriminator Φ 1 , disc , and generator Φ 1 , gen in the K inner loop through objective functions L V AE , L disc , and L gen , respectively.

Original: The local minimum parameters of the encoder, decoder, discriminator, and generator are achieved by minimizing loss L V AE , L disc , and L gen .
Revised: The local minimum parameters of the encoder, decoder, discriminator, and generator are achieved by minimizing loss L V AE , L disc , and L gen .

Original: The total loss function of our Meta-FAVAE method can be written as
Revised: The total loss function of our Meta-FAVAE method can be written as

Original: [Equation]
Revised: [Equation]

Original: Where L ms represents a mode-seeking regularization termMao et al.
Revised: Where L ms represents a mode-seeking regularization termMao et al.

Original: (2019) that maximizes the ratio of the distance between G 1 ( z 1 , c )
Revised: (2019) that maximizes the ratio of the distance between G 1 ( z 1 , c )

Original: and G 1 ( z 2 , c ) corresponding to the distance between z 1 and z 2 , as expressed in equation (7).
Revised: and G 1 ( z 2 , c ) corresponding to the distance between z 1 and z 2 , as expressed in equation (7).

Original: [Equation]
Revised: [Equation]

Original: where d ( · , · ) indicates the L 1 norm distance metric.
Revised: where d ( · , · ) indicates the L 1 norm distance metric.

Original: Following the same settingMao et al.
Revised: Following the same settingMao et al.

Original: (2019), we set the hyper-parameters λ ms = 1 .
Revised: (2019), we set the hyper-parameters λ ms = 1 .

Original: The outer loop is performed after the K steps of the inner loop operation.
Revised: The outer loop is performed after the K steps of the inner loop operation.

Original: The global parameters Φ 2 of the encoder Enc 2 , decoder Dec 2 , generator G 2 , and discriminator D 2 are updated with parameters Φ 1 by setting it to Φ 2 − Φ 1 .
Revised: The global parameters Φ 2 of the encoder Enc 2 , decoder Dec 2 , generator G 2 , and discriminator D 2 are updated with parameters Φ 1 by setting it to Φ 2 − Φ 1 .

Original: The pseudo-code of
Revised: The pseudo-code of

Original: Meta-FAVAE for the training process and generation process is described in supplementary
Revised: Meta-FAVAE for the training process and generation process is described in supplementary

Original:  E VALUATIONS AND C ONCLUSION
Revised:  E VALUATIONS AND C ONCLUSION

Original: The experiments are conducted on three few-shot image datasets: MNIST( LeCun & Cortes (2010)),
Revised: The experiments are conducted on three few-shot image datasets: MNIST( LeCun & Cortes (2010)),

Original: Omniglot( Lake et al.
Revised: Omniglot( Lake et al.

Original: and VGG-Faces( Cao et al.
Revised: and VGG-Faces( Cao et al.

Original: (2018)).The quality of generated images are evaluated by FID ( Heusel et al.
Revised: (2018)).The quality of generated images are evaluated by FID ( Heusel et al.

Original: (2017)), while The diversities of output are evaluated through
Revised: (2017)), while The diversities of output are evaluated through

Original: LPIPS( Zhang et al.
Revised: LPIPS( Zhang et al.

Original: and IS( Xu et al.
Revised: and IS( Xu et al.

Original: (2018)).Based on the 1000 generated images of the unseen classes, we compare our model with baseline approaches including GMN( Bartunov & Vetrov (2018)), DAGAN( Antoniou et al.
Revised: (2018)).Based on the 1000 generated images of the unseen classes, we compare our model with baseline approaches including GMN( Bartunov & Vetrov (2018)), DAGAN( Antoniou et al.

Original: , FIGR( Clouˆatre & Demers (2019)), DAWSON( Liang et al. (2020)) and FAML( Phaphuangwittayakul et al.
Revised: , FIGR( Clouˆatre & Demers (2019)), DAWSON( Liang et al. (2020)) and FAML( Phaphuangwittayakul et al.

Original: The results presented in Table 1 show our method achieves the best performance in the evaluations on all three datasets.
Revised: The results presented in Table 1 show our method achieves the best performance in the evaluations on all three datasets.

Original: Additionally, the feedback operation is leveraged to increase the diversity of output images and generalize the model across different categories.
Revised: Additionally, the feedback operation is leveraged to increase the diversity of output images and generalize the model across different categories.

Original: The model with feedback operation not only augments the data to increase the size of input but also avoids the overﬁtting occurred by a few training samples.
Revised: The model with feedback operation not only augments the data to increase the size of input but also avoids the overfitting occurred by a few training samples.

Original: Although a variational auto-encoder has been added to the network, our model can signiﬁcantly reduce the number of parameters to 8.3M, which is merely one-quarter of FIGR.
Revised: Although a variational auto-encoder has been added to the network, our model can significantly reduce the number of parameters to 8.3M, which is merely one-quarter of FIGR.

Revised: Furthermore, it converges over ten times faster than the baseline models.

Original: Thus, our model can fast adapt to new generation tasks with limited steps as shown in Figure 2.
Revised: Thus, our model can fast adapt to new generation tasks with limited steps as shown in Figure 2.

Original: Comparing with the generated images from other methods, the Meta-FAVAE can not only retain the compact features due to the latent representations but also generate more diverse images.
Revised: Comparing with the generated images from other methods, the Meta-FAVAE can not only retain the compact features due to the latent representations but also generate more diverse images.

Original: In this work, we have introduced a novel FSIG approach based on meta-learning with a feedback augmented variational auto-encoder.
Revised: In this work, we have introduced a novel FSIG approach based on meta-learning with a feedback augmented variational auto-encoder.

Original: This model can outperform the existing approaches in term of fast adaptation and the diversity of outputs.(More results can be seen in supplementary)
Revised: This model can outperform the existing approaches in term of fast adaptation and the diversity of outputs. (More results can be seen in supplementary)

Original: [Table]
Revised: [Table]

Original: FIGR 36.

Original: [Table]