R OBUST
C ROSS -M ODAL
S EMI -
SUPERVISED
F EW
S HOT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Semi-supervised learning has been successfully applied to few-shot learning (FSL) due to its capability of leveraging the information of limited labeled data and massive unlabeled data.
However, in many realistic applications, the query and support sets provided for FSL are potentially noisy or unreadable where the noise exists in both corrupted labels and outliers.
Motivated by that, we propose to employ a robust image-text multi-modal semi-supervised few-shot learning (RCFSL) based on Bayesian deep learning.
By placing the uncertainty prior on top of the parameters of inﬁnite Gaussian mixture model for noisy input, multimodality information from image and text data are integrated into a robust heterogenous variational autoencoder.
Subsequently, a robust divergence measure is employed to further enhance the robustness, where a novel variational lower bound is derived and optimized to infer the network parameters.
Finally, a robust semi-supervised generative adversarial network is employed to generate robust features to compensate data sparsity in few shot learning and a joint optimization is applied for training and inference.
Our approach is more parameter-efﬁcient, scalable and adaptable compared to previous approaches.
Superior performances over the state-of-the-art approaches on multiple benchmark multi-modal dataset are demonstrated given complicated noise for semi-supervised few-shot learning.
I NTRODUCTION
Despite the impressive success of deep learning models, frequently it requires massive amount of training data to fully demonstrate the potential of the model.
In contrast, human is capable of learning new concepts given limited data.
Consequently, few-shot learning gathers extensive research interest due to the capabilities of learning new concepts from limited training data.
Nevertheless, the success of few-shot learning requires careful handling to robustness and generalization as it is extremely susceptible to noisy labels, outliers as well as adversarial attack Lu et al.
(2020a).
For instance, in order to automatically recognize several kinds of uncommon animals, only a few annotated images for them are available due to their rarity.
Moreover, the images could potentially be corrupted due to an uncontrollable shooting environment or an instrumental malfunction.
To mitigate this, one common approach to few shot learning is meta-learning Ren et al.
(2018), where the goal is to learn a classiﬁer to distinguish between previously unseen classes, given labeled classes and a larger pool of unseen examples, some of which may belong to the classes of interest, namely semi-supervised few shot learning (SFSL).
Despite the impressive capabilities equipped the ability to leverage unlabeled examples for SFSL, the challenge of lacking novel samples remains to be a bottleneck.
Besides visual information, textual data frequently contains rich information and more descriptive concepts for learning.
Incorporating image-text multi-modal learning into the framework by training on image-text pairs provides an efﬁcient tool to inject the diversity to the generation process Pahde
et al.
(2021)Pahde1 et al.
The work in Pahde1 et al.
(2018) provides a benchmark for multimodal few-shot learning relying on a class-discriminative text conditional generative adversarial network.
Later on, Pahde1 et al.
(2021) tackles the multimodal few shot learning problem by employing a cross-modal feature generation network to infer the class membership of unseen samples with a simple nearest neighbor approach.
Despite of the success of these methods with clean features and perfect labels, the important case that features and labels are contaminated due to out-of-distribution samples, adversarial attack and human fatigue is rarely studied.
In parallel, Bayesian deep learning (BDL) has served as a powerful tool in terms of transforming the problem of posterior inference of a BDL model into the optimization of an objective function based on latent variables.
Now the question then is: how to design a
Bayesian deep learning which counters the noisy labels and outliers jointly in the multimodal semisupervised few-shot learning.
Accordingly, this paper tackles this challenging problem in robust cross-modal few-shot learning by integrating a deep generative heterogenous model that generalizes well to multi-modality (e.g. image-text modeling) in order to counter noisy labels and outliers .
Speciﬁcally, a robust heterogeneous variational auto-encoder is ﬁrst proposed to encode the noisy visual features and labels in order to jointly learn the information from both modalities by placing the uncertainty prior on the top of the inﬁnite Gaussian mixture models.
Subsequently, a robust variational lower bound based on β -divergence is derived to infer the network parameters.
Finally, a robust semi-supervised GAN is integrated with the heterogenous variational auto-encoder by collapsing the generator and the decoder into one to further boost the learning capabilities.
RCFSL is built on top of Bayesian deep learning by fusing cross-modal information via the approximation of the joint posterior distributions.
In contrast to modality-alignment methods Xing et al.
(2019a)Tsai et al.
(2017) for robust few-shot learning, the robustness of RCFSL is achieved by accurate modeling of the complicated joint distribution of multi-modality data and robust variational inference by the derived lower bound.
Distinct from the work in Xing et al.
(2019b)
calculating linear combinations in the prototypical representation space, our fusion of multimodality features in the probability distribution is completely data-driven, yielding more robust classiﬁcation performance in few-shot learning.
Major contributions of this paper are: (1) RCFSL harnesses three levels of denoising to ensure the robustness of cross-modal semi-supervised few-shot learning: Firstly, motivated by Hou & Galata (2008), it places the uncertainty prior of the parameters of an inﬁnite Gaussian mixture distribution of image data to avoid mixture components collapsing onto a point or a hyperplane due to outliers.
(2) Subsequently, the robust β -divergence is employed to replace Kullback-Leibler divergence used for data ﬁtting to infer the network parameters and a novel evidence lower bound for semi-supervised few shot learning is derived.
(3) Noise-transition layers are applied to both the heterogenous variational encoder and the robust discriminator in semi-supervised learning with an end-to-end training.
The performance of RCFSL is further boosted with robust feature generation yielding 7% to 10% absolute accuracy improvement over STOA approaches.
Related Work Previous work in multimodal few shot learning frequently tends to ﬁrst learn text to image mapping to generate additional visual features and then calculate the joint prototype using a weighted average from two representations.
Two recent approaches have attracted significant attention in the few-shot learning domain: Matching Networks Vinyals et al.
(2016), and
Prototypical Networks Snell et al.
(2017) where the sample set and the query set are embedded with a neural network, and nearest neighbor classiﬁcation is exploited relying on a metric in the embedded space.
In Oreshkin et al.
(2018), metric scaling and metric task conditioning are utilized to improve the performance of few-shot learning algorithms.
Kim et al.
and Finn et al.
(2018) employ a probabilistic extension to model-agnostic meta-learning (MAML) framework trained with variational approximation so that the model can generalize well to a new task with a few ﬁne-tuning updates.
In Zhang et al.
(2020), a bidirectional joint image-text modeling was proposed and VHE-raster-scan-GAN was applied.
RCFSL advances the work from Zhang et al.
(2020) by improving the robustness of the multi-modal heterogenous encoder and extend the solution to semi-supervised few shot learning.
In Tseng et al.
(2020), feature-wise transformation layers are utilized for augmenting the image features relying on afﬁne transforms to simulate various feature distributions under different domains for few-shot learning.
Different from Tseng et al.
(2020), RCFSL augments the robust feature generation from BDL perspective relying on robust semi-supervised GAN.
Moreover, our method advances from other robust few shot learning such as
Rapnets Lu et al.
(2020b)
and
Adversarial Query Goldblum et al.
by providing mathmatically rigoriously denoising schemes via uncertainty priors and robust divergence in variational inference.
O UR M ETHOD
Our approach is to focus on ﬁrst constructing a semi-supervised robust heterogeneous variational autoencoder leveraging a mixture model to encode both image and text data in few shot learning insensitive to both noisy labels and outliers.
Subsequently, a novel robust variational lower bound is derived to facilitate the inference of network parameters relying on β -divergence for both labeled and unlabeled data.
Finally, a robust generative adversairial network is integrated with denoising layers to strengthen the denoising performance togehter with the end-to-end optimization to generate additional visual features to alleviate the sparsity in the semi-supervised few shot learning.
Let Ω denote image space, Υ denote text space and C = { 1 , . .
, R } be the discrete label space.
Further let x i ∈ Ω as the i -th input image observation, t i ∈ Υ as its corresponding textual description and y i ∈ C as its label.
Denote C base as base classes where we have both labeled and unlabeled samples and denote C novel novel classes, which are underrepresented in the data.
Inspired by the fact that the student-t distribution is more robust to the outliers than Gaussian distribution by constraining the shapes of the mixture components from collapsing Hou & Galata (2008), we propose to place uncertainty priors (e.g. Gaussian priors) on parameters of inﬁnite Gaussian mixture models to characterize the inﬂuence of outliers on image data and constrain the shape of the components to prevent them from collapsing .
The Heterogeneous Mixture Model: Variational autoencoder Diederik & Welling (2013) has been recently proposed as a powerful solution for semi-supervised learning.
To the best of our knowledge, this is the ﬁrst time that a robust heterogenous VAE model has been applied which naturally integrates noisy images and text together which cohesively fuse continuous and discrete multi-modality features.
Variational inference are applied to ﬁt the heterogenous model on both of image and text features Ψ
I from base classes C base , where the embedding is obtained from the last dense layer right before the softmax output in the discriminator.
Once a good mapping Ψ
I based on heterogenous image and text data is learnt, given a test sample, the class membership is given by assigning the class label of the closest prototype to an unseen test sample.
In particular, the heterogenous features are ﬁrst fed into the Dirichlet process mixture and clustered based on their similarity measures, where here image features are modeled as inﬁnite Gaussian mixture distribution Allen et al.
(2019) to characterize the samples from the unseen classes by computing the posterior distribution for unrepresented clusters more accurately and inferring the number of classes automatically and text features are characterized by Poisson distribution.
Speciﬁcally, x n is a noisy measurement of its true position and is a draw from the Gaussian mixture model, where the mean of each
Gaussian component T k is unknown and the variance C k is known.
In order to characterize the uncertainty and the outliers from the input, the Gaussian prior is placed on the top of the mean for each Gaussian component.
Namely, T k satisﬁes the normal distribution with the mean µ k and the precision matrix Λ
k . ω
i is the latent variable for the i th data point specifying which Gaussian it came from and π is the mixing weight for the Gaussian mixture model.
Speciﬁcally, a NormalWishart prior Murphy (2007) is placed on the mean and precision of the Gaussian components: p ( µ, Λ) =
(cid:81)
Kk =1 N ( µ k | m 0 , ( β 0 Λ k ) − 1 ) W (Λ k | W 0 , ν 0 ) , where m 0 is set to be zero and β 0 is set to be a very small value.
W (Λ | W, ν ) is a Wishart distribution with scale matrix W and ν degrees of freedom.
For text features after word embeddings, (cid:15) l and ζ l are the prior shape and rate parameters respectively in the Gamma distribution that generates the average rate parameter for the l -th Poisson feature.
In particular, the cluster assignments for each observation are drawn from multinomial distributions where the prior parameters represent the mixing weights of the corresponding clusters.
Speciﬁcally, the truncated stick-breaking process is employed to construct mixing weights.
By construction, a single parameter α controls the portion of the K − 1 major clusters, and χ separately controls for the portion of the remainder via a stick-breaking procedure Beta (( K − 1) α, χ ) Blei & Jordan (2017).
Namely, where ω =
[ ω 1 , . .
, ω K ] belongs to the ( K − 1) -dimensional simplex and is generated from a Dirichlet prior with two parameters p ( ω )
˜ Dir ( α, . . .
, α, χ ) .
Typically, given the image data x and the textual features t , variational inference with deep learning from powerful probabilistic models are constructed by an inference neural network q ( z | x, t ) and a generative neural network p ( x, t | z ) .
The generative model and the inference model are parameterized by θ and φ respectively.
Subsequently, the network parameters are estimated by optimizing the evidence lower bound (ELBO) in the variational inference.
For unsupervised learning of multimodality data, the vanilla VAEs are optimized by maximizing ELBO:
[Equation]
The Generative and Inference Model: We shall now present the proposed generative and inference model for few-shot learning.
Given a semi-supervised learning setting, the labels y are either unknown (for unlabeled data) or noisy (for labeled data).
The generative model is then deﬁned as: p θ ( x, t | ω, T, C ) p θ ( T | Z, µ, Λ , λ ) p θ ( a | z, ˜ y, x, t ) p θ ( x, t | ˜
y, z ) p ( z ) p (˜ y )
p ( ω ) p ( t | λ ) .
Deﬁne p θ as the deep neural network with parameters θ
and y as the ground truth of class label.
For unlabeled data, y is considered as a latent variable.
Further denote Cat( . ) as a multinomial distribution and in this paper we reasonably assume that the labels are of categorical distribution and the proposed model applies to other distributions for the latent variable y .
In order to ﬁt more complicated posteriors for the marginal distribution p ( z | x, t ) and bridge the gap between two modalities, motivated by Maaløe et al.
(2017), we extend the variational distribution with auxiliary variables a , so that the generative model is invariant to marginalization over a p ( x, z, a, ω, T, C, µ, Λ) =
p θ ( x | ω, T, C ) p θ ( T | Z, µ, Λ) p θ ( a | z, x ) p θ ( x, z ) .
To attenuate the inﬂuence with the noisy labels, further denote ˜ y as the corrupted labels and ˆ y as the corrected label after denoising layer.
Deﬁne K × K noise transition matrix M to associate ˜ y with ˆ y and estimation of M has been addressed in previous methods Sukhbaatar et al.
(2015)Patrini et al.
In particular,
M =
( M i,j ) ∈ [0 , 1] c × c ( (cid:80) i
M i,j = 1) .
The proposed generative model can then be expressed as:
[Equation]
if the j th cluster is represented
[Equation]
if the j th cluster is unrepresented
The inference model can be represented as:
[Equation]
where q ( λ l | t ) characterizes posterior variational density of Poisson parameters given the discrete embedded feature vectors t converted from text data.
ψ ik stands for the probability of the i th observation belongs to the cluster k .
t il represents the l -th Poisson feature of the i th observation.
Denote
N − n,j as the number of data points, excluding x n which belongs to the mixture component j .
To compute q ( T, µ, Λ , λ | x, t ) , mean-ﬁeld approximation is applied Bishop (2006) to factorize all the latent variables and parameters for the represented and unrepresented j th cluster respectively:
[Equation]
The above equation indicates that given the multimodality data, the unlabeled samples has a certain probability of being classiﬁed as unrepresented mixtures (e.g. unseen new classes), which facilitates the learning capabilities of the proposed robust semi-supervised few shot learning.
Within an iteration, some unlabeled samples are associated with unrepresented mixtures, a new represented mixtures will emerge which successfully addressed the challenges in semi-supervised few shot learning when the unlabeled data contains unseen new classes which doesnot exist in labeled data.
Robust Variational Lower Bound: To further alleviate the impact from outliers, the robust divergence is employed to infer network parameters more accurately.
The theoretical foundation of β -divergence has been initially deﬁned at Basu et al.
(1998), where the β -divergence between two functions g and f are deﬁned as
[Equation]
When β → 0 , the β -divergence converges to KL-divergence, lim β
→ 0 D β ( g (cid:107) f )
=
D KL ( g (cid:107) f ) .
As described in Futami et al.
(2018), minimizing the β -divergence from the empirical distribution ˆ p ( x ) to p ( x ; θ ) arg min θ D β
(ˆ p ( x ) (cid:107) p ( x ; θ )) , it is easy to show 1 N (cid:80)
Ni =1 p ( x i ; θ ) β
∂ ln p ( x i ; θ )
∂θ −
E p ( x ; θ ) [ p ( x ; θ ) β ∂ ln p ( x ; θ ) ∂θ ] .
As the probability densities of outliers are usually much smaller than those of inliers, the ﬁrst term of the above equation is the likelihood weighted according to the power of the probability density for each sample, which effectively suppress the likelihood of outliers.
This estimator is also called as M -estimator Huber & Ronchetti (2011), which provides provably superior performance in various machine applications Li & Gal (2017).
The variational lower bound for the proposed RCFSL model for labeled data can be represented as
[Equation]
The above inequality can be rewritten as
[Equation]
To mitigate the inﬂuence of outliers
[Equation]
let H =
{ a, z, T, ω, µ, Λ , λ } represent the set of all the latent variables and leverage the technique from Futami et al.
[Equation]
we can replace KL-divergence with β -Divergence and cast the β -ELBO for labeled data L β as:
[Equation]
For unlabeled data, by introducing the variational distribution for ˜ y as q φ ( a, x, t | ˜ y ) , the variational lower bound for the proposed RCFSL can be represented as
[Equation]
Similarly, replacing the KL-divergence with β -Divergence and augmenting the latent variable H u
= { a, z, y, T, ω, µ, Λ , λ } , the β -ELBO for unlabeled data in RCFSL is:
[Equation]
Practically, L β and U β are calculated via Monte Carlo sampling.
The robustness of our proposed ELBO can be guaranteed leveraging the inﬂuence function (IF)
Futami et al.
(2018)Huber &
Ronchetti (2011).
As IF is widely used to analyze how much contamination affects estimated statistics, it is straightforward to show that given the perturbation on the empirical cumulative distribution caused by outliers, it is straightforward to show that the IF of our posterior distribution is bounded.
The objective function for robust cross-modal variational autoencoder is: L RCV AE =
L β + λ 1 U β , where λ 1 represents the weight to control the trade-off between the labeled data and unlabeled data.
Robust Feature Generation:
The construction of the proposed uncertainty priors and the robust divergence measure in our framework aims at better approximation of the posterior distribution under noisy labels and outliers.
This is also related to employing generative adversarial learning of z and x, t deﬁned by the data, the encoder, the prior and decoder.
Different from VAEs that assumes parametric data distribution and perform posterior inference, GANs in general utilize implicit data distribution and do not provide meaningful latent representations.
By learning both a generator G and a discriminator D , a min-max objective is optimized:
[Equation]
In Kaneko et al.
(2019), a noise transition model is incorporated to learn a clean label conditional generative distribution.
But their model only considered noisy labels without outliers and is limited to supervised learning.
Recent work on semi-supervised GAN with k classes Kumar et al.
(2017)Salimans et al.
(2016) modify the discriminator with k + 1 outputs on the discriminator by considering the fake images as the k + 1 th class.
Hence, the loss for the training can be computed with a supervised loss and an unsupervised loss respectively:
[Equation]
where L sup represents the negative log probability of the label given the data is real.
Denote D ( x, t ) = 1 − p f ( y = k + 1 | x, t ) , the loss for semi-supervised GAN on multimodality data L sGAN can be written as: L = − E x, t ∼ p
d ( x, t ) log D ( x, t ) − E z ∼ noise (1 − D ( G ( z ))) −
E x, t ,y ∼ p
d ( x, t ,y ) log p f
( y | x, t , y ≤ k ) .
More recently, the label noise robust GAN (rGAN) Kaneko et al.
(2019) has achieved promising performance in classifying images with noisy labels by incorporating a noise transition model to learn a clean label conditional generative distribution under noisy labels.
Thus, given the noise samples (˜ x r , ˜ y r ) ∼ ˜ p
d ( x, ˜ y ) , to construct a label-noise robust conditional generator and discriminator, the objective function of the robust semi-supervised GAN in the proposed robust cross-modal semi-supervised few-shot learning is expressed as: L RSGAN =
[Equation]
1 | x, t ))
− E z ∼ noise (1 − D ( G ( z ))) −
E (˜ x r , t , ˜ y r ) ∼ ˜ p
d ( x, t , ˜ y ) log (cid:80) ˆ y r M ˆ y r , ˜ y
r ˆ C (ˆ y =
ˆ y r
| x, t , ˆ y ≤ k ) .
Without modeling the data distribution explicitly and representing the latent space in a meaningful manner, it does not provide the functionality to counter the outliers in the generator and discriminator.
Motivated by the thought of bridging the gap between robust VAE and robust GAN, we feed the robust variational posterior p ( z | x, t ) instead of the random noise p
( z ) into the label noise robust semi-supervised GAN as the source of randomness as both of the decoder and the generator of RSGAN share the mapping from z to x and t .
The full optimization function for the proposed RCFSL framework can be represented as:
[Equation]
where λ 1 is set to be the ratio of unlabled samples verus labeled samples and the discriminator loss is characterized by E[ln D ( x, t )]
= − E x, t ∼ p
d ( x, t ) log(1 − M ˆ y r , ˜ y
r ˆ C (ˆ y = k
+ 1 | x, t ))
−
E (˜ x r , t , ˜ y r ) ∼ ˜ p
d ( x, t , ˜ y ) log (cid:80) ˆ y r M ˆ y r , ˜ y
r ˆ C (ˆ y =
ˆ y r
| x, t , ˆ y ≤ k ) .
This new architecture is expected to better predict the class labels under the compound noise.
We then train RCFSL including the robust heterogenous encoder, the generator and the robust discriminator in an end-to-end manner using adaptive moment estimation (Adam) Kingma & Ba (2015).
Each multimodal embedding prototype p c (of category c ) is computed by averaging the embeddings of all support samples of class c .
Once the robust embedding is obtained, the distance between the embedding of the query q t and the multimodal prototype p c is calculated by p ( y = c
| p c ) = exp
[Equation] [Equation]
k exp( − d ( f ( q t ) , p k )) where d refers to Euclidean distance and the query is classiﬁed as the class with the minimum distance.
[Table]
E XPERIMENTS
Dataset and Competing Methods: We extensively evaluate the proposed robust cross-modal semisupervised few shot learning (RCFSL) algorithm on multiple benchmark datasets including miniImageNet(mIN), tieredImageNet (tIN), Fewshot-CIFAR100 (FC100) and Caltech-UCSD Birds 2002011 (CUB).
The miniImageNet dataset is a subset of ILSCRS-12 which includes 100 classes where each class contains 600 RGB images with size 84 by 84.
We split the datasets by using 64 classes for training, 16 classes for validation and 20 classes for testing where 16 classes are utilized to monitor the model’s generalization performance.
The CUB Bird dataset contains 11788 images ofdifferent bird species, where the data is split equally in training and testing data.
Namely, for each category, 30 training image and 30 testing images are included, where in the training dataset, 10% images are labeled images and the rest of the images are unlabeled.
10 short textual descriptions per image are provided by Reed et al.
Following the work Zhang et al.
(2017), the text encoder pretrained by Reed et al.
(2016) is employed and the data is split such as | C Base | = 150 , | C Novel | = 50 .
Speciﬁcally, n = { 1 , 2 , 5 , 10 , 15 , 20 } images are selected from novel classes in order to perform few shot learning.
We evaluate the two types of label noise including symmetric noise and asymmetric noise.
For symmetric noise, we inject the label noise by randomly ﬂipping the labels of the labeled data into a different label in the novel classes.
For asymmetric noise, the corrupted labels are generated by replacing the correct labels with their most similar classes (using nearest neighbor measurement).
Outliers are mimicked by including samples from data distributions far away from the training data distributions, namely out-of-distribution (OOD) samples.
GloVe Pennington et al.
(2014) is utilized to extract the word embeddings for the category labels, where the embeddings are trained with large unsupervised text corpora.
RCFSL is compared with multiple benchmark methods and state-of-the-art approaches.
Typically, similarly as Xing et al.
(2019b), three families of methods are evaluated: (1) uni-modality few-shot learning methods such as MAMLFinn et al.
(2017), LEORusu et al.
and CAMLJiang et al.
In particular, latent embedding optimization (LEO) Rusu et al.
(2019) addresses MAML problem by relying on a few updates on a low data regime to train models in a high dimensional parameter space.
(2) cross modality few shot learning methods ReViSE Tsai et al.
CADA-VAE Sch¨onfeld et al.
Xing et al.
(2019b),
VHE-raster-scan-GANZhang et al.
and Feature transformTseng et al.
Among them, ReViSE Tsai et al.
(2017) minimizes maximum mean discrepancy (MMD) of the distributions from two representation spaces for better alignment.
CADA-VAE Sch¨onfeld et al.
[Table]
(2019) leverages two VAEs to embed information for both modalities and align the distribution of the two latent spaces.
(3) Robust few shot learning methods including robust attentive proﬁle networks (Rapnets) Lu et al.
(2020b)
and
Adversarial Querying (AQ) Goldblum et al.
RapnetsLu et al.
(2020b) consists of an embedding module, a correlation module, and an attentive module to suppress the outliers in few-shot learning.
Adversarial Querying Goldblum et al.
(2020) produces adversarially robust meta-learners and achieves efﬁcient classiﬁcation for few-shot learning tasks.
Implementation Details: For miniImagenet, tiredImagenet and FC100 datasets, we parameterize the heterogeneous VAE with three sets of 5 by 5 fully convolutional, ReLU and pooling layers followed by two fully connected hidden layers where each pair of layers contains the hidden units as dim(h)
= 1000.
We further set the dimension of the auxiliary variable a and the latent variable z to be 200 respectively.
For the CUB dataset, we leverage Resnet-18 as backbone and incorporate the uncertainty prior for the input, the noise transition model along with the auxiliary variables as our encoder.
The network is trained with SGD using a batch size of 128.
A momentum of 0.9 is set with a weight decay of 0.0005.
The network is trained for 30000 iterations.
The initial learning rate is set as 0.1, and reduce it by a factor of 10 after iterations 15,000, 17,500 and 19,000.
For the conﬁgurations of discriminator in GAN, we follow the same conﬁguration of DC-GAN and follow the similar discriminator Mirza & Osindero (2014) for DC-GAN.
β is varied from 0.1 to 0.
where the best performance is reported.
The similar setup in Xing et al. (2019b) is utilized relying on ACC ( N, K, C ) to evaluate the performance of classiﬁcation accuracy in a N -way and K -shot
SFSL settings, where C represents the total number of outliers and noisy labels in each class.
We use 10 textual descriptions per image to generate multimodal prototypes.
The semi-supervised fewshot classiﬁcation accuracy on test split of miniImageNet is shown in Table 1.
As shown in Table 1, RCFSL achieves the best classiﬁcation accuracy compared to all of the competing methods in different cases.
The large performance margin suggests the proposed mechanism not only fuses two modalities of features cohesively and but also effectively suppresses the noisy labels and outliers in semi-supervised few-shot learning.
Compared to the adaptive modality mixture mechanism (AM3) with TADAM as backbone Xing et al.
(2019b), RCFSL has the advantage of fusing multimodal distributions in the heterogeneous encoder via the latent space to fully utilize the multimodal feature representation instead of computing multimodal prototypes with a convex combination afterwards.
In contrast to robust FSL methods like Rapnets Lu et al. (2020b) and AQ Goldblum et al.
(2020), the performance gain can be mainly attributed to the fact that RCFSL employs a mathematically rigorous variational inference insensitive to noisy labels and outliers to optimize network parameters, yielding more accurate classiﬁcation performance.
The margin in performance is particularly
[Table]
The embeddings from 10 distinct clusters using our method corresponds to true class labels, which validates the robustness of our method to label noise and outliers.
Fig 3(c)- (d) illustrate the classiﬁcation accuracy vs the number of shots for FC100 and CUB-200 datasets respectively.
remarkable in the fewer shots scenario, supporting the robustness of RCFSL.
Similar performance evaluations are conducted on tieredImageNet dataset in Table 2 and RCFSL again is the top performer under various noise settings.
Fig.3 demonstrates the comparison of the t-SNE visualization with two dimensional multimodal prototype space for the FewShot CIFAR-100 dataset using RapNetsLu et al.
(2020b) and our RCFSL ( β =0.2) on a 10-Way 8-shot semi-supervised learning (Fig 3.(a)- (b)), where it clearly shows our method has a better capability in terms of separating noisy labels and outliers in semi-supervised few-shot learning.
We also investigate the classiﬁcation accuracy of different methods versus the number of shots in Fig 3.(c)- (d) for FC100 and CUB datasets respectively with 25% noisy labels and outliers respectively, RCFSL yields considerable performance gains over competing methods especially with smaller number of shots.
Ablation Study:
Described in the ablation study in Table 3, RCFSL w/o uncertainty prior excludes the uncertainty prior from the model.
Therefore, the performance degradation suggests the importance of the proposed hierarchical structure for variational inference by placing the uncertainty prior on the inﬁnite GMMs to counter the detrimental effects of outliers.
Secondly, our method w/o robust divergence replaces β -divergence which places small weights on noisy labels and outliers with the regular KL-divergence.
Moreover, RCFSL w/o cross modality relies on only image data for SFSL where multi-modality further improves the classiﬁcation accuracy by at least 4%.
C ONCLUSION
By integrating uncertainty prior of an inﬁnite Gaussian mixture model into the heterogenous encoder and the robust lower bound based on β -divergence for variational inference, our RCFSL is capable of tackling the outliers and noisy labels simultaneously in semi-supervised few shot learning.
Moreover, by integrating the robust semi-supervised GAN for feature generation, experimental evaluations on multiple benchmark datasets have demonstrate the superiority of the proposed method with large margins using multimodality data.
A D ERIVATION OF I NFLUENCE F UNCTION FOR L OWER B OUND OF RCFSL
Deﬁne G ( x ) as a empirical cumulative distribution given by { x i } ni =1 and denote the perturbed version of G at z as G ε,z ( x ) =
(1 − ε ) G ( x ) + ∆ z ( x ) , where ε is the contamination portion and ∆ z ( x ) is the point mass at x .
Given a statistic T .
the inﬂuence function (IF) is deﬁned as Futami et al.
[Equation]
Thus, the IF of the heterogenous variational autoencoder for L RCV AE is given by
[Equation]
It is straightforward to show that the above result is always bounded, namely RCFSL is robust to the compound noise (the outliers on the data x and the labels y ).
B V ISUAL C OMPARISON WITH R OBUST D IVERGENCE
Fig.1 demonstrates the comparison of the t-SNE visualization with two dimensional multimodal prototype space for the FewShot CIFAR-100 dataset using the same setting as our method except replacing robust divergence with KL-divergence and our RCFSL ( β =0.2) on a 10-Way 8-shot semi-supervised learning, where it clearly shows the efﬁcacy of the robust divergence by offering a stronger capability in separating noisy labels and outliers in semi-supervised few-shot learning.
[Table]
C D ETAILS ON N ETWORK S TRUCTURE , T RAINING AND I NFERENCE T IME C OMPARISON
The discriminator of RCFSL following deep convolutional GAN is structured as follows: the ﬁrst convolutional layer is of size 32 × 32 × 64 , the second convolutional layer is of dimension 16 × 16 × 128 , the third layer is 8 × 8 × 256 and the last convolutional layer is of the dimension 4 × 4 × 512 with the convolutional ﬁlter size 5 × 5 × 3 .
Each convolutional layer is followed by a batch normalization layer and then a ReLU layer.
We analyze the training time (hours) and inference time (seconds) of our RCFSL to demonstrate its computational efﬁciency in Table 4 and Table 5.
The training and the inference time of RCFSL is reported on Fewshot-CIFAR100 with several state-of-the-art approaches, evaluated on a single
Nvidia V100 GPU.
RCFSL is slower than AQ Goldblum et al.
(2020), however is faster than the rest of competing methods for both training and inference time due to the efﬁcient probabilistic graphical model and the effective novel lower bound and compact network architecture.
D D ISCUSSION ON DIFFERENCES FROM OTHER MULTIMODAL VAE S Prior approaches
Pandey & Dukkipati (2017) on multimodal VAEs mostly focus on cross-modal generation.
In particular, given two modalities x 1 and x 2 (e.g. image and text), these approaches
[Table]
learn the conditional generative model p ( x 1 | x 2 ) , where the conditioning modality x 2 and the generation modality x 1 are usually not interchangebale.
Compared to Pandey & Dukkipati (2017)
[Table]
our heterogeneous robust VAE targets joint image-text modeling by explicitly modelling the joint distribution over latent variables from both image and textual data.
Moreover, our multimodal VAEs integrates the uncertainty priors, denoising layers and robust variational lower bound to ensure the robustness in the presence of noisy labels and outliers.
The work Suzuki et al.
(2017) introduced joint multimodal VAE (JMVAE) which aims at learning shared representation with joint encoder q ψ ( z | x 1 , x 2 ) .
Distinct from Suzuki et al.
(2017) which does not consider the robust modeling to tackle with outliers, our approach leverages inﬁnite Gaussian mixture models equipped with uncertainty prior which not only characterizes the distributions for image-text modeling more accurately but also facilitates the learning of new unseen classes by inferring the number of classes automatically in few-shot learning.
Recently, Zhang et al.
(2020) utilized a VAE that encodes an image into a deterministic-upward–stochastic-downward ladder-structured latent representation, and then is applied to decode the corresponding text.
While our work is relying on joint encoding and decoding schemes on image-text modelling and focusing on tackling the noisy-labels and outliers by enhancing the robustness of multimodal modeling using the various proposed techniques.
More recently,
Shi et al.
(2019) leverages a mixture-of-experts multimodal variational autoencoder (MMVAE) to learn generative models on different sets of modalities, indicating the suitability of mixture models for learning from multi-modal data.
Our work (RCFSL) advances MMVAE Shi et al.
(2019) in three aspects: (1) We places uncertainty prior on the parameters of mixture models to avoid the collapsing of models in the presence of outliers and derived novel inference solution given the uncertainty prior (shown in our novel generative and inference models in Eqs (2)-(4)); (2) instead of using importance weighted autoencoder (IWAE) estimator Burda et al.
(2015)Shi et al.
(2019), RCFSL takes advantage of robust divergence to derive the robust evidence lower bound for unlabeled data and labeled data respectively which are unique and mathematically rigorously (e.g. Eq(7)), yielding superior performance for classiﬁcation of noisy dataset in semi-supervised few-shot learning.
(3) RCFSL integrates the denoising layers in both multimodal VAE and robust GAN to mitigate the inﬂuence of noisy labels, which further enhances the robustness of the model.
E E VALUATION ON S EMANTIC L ABEL N OISE
In order to evaluate our algorithm under real-world noise (semantic label noise), RCFSL is further evaluated on Clothing 1M dataset with the network architecture Resnet-18.
For Clothing1M dataset, it includes 1 million training images with 14 classes obtained from online shopping websites and labels are generated from surrounding texts.
Since the data is collected from multiple online shopping websites and include many mislabelled samples, therefore the labels are contaminated by real-world noise (semantic label noise).
The data is split such as | C Base | = 10 , | C Novel | = 4 .
Tabledemonstrates the comparison of classiﬁcation accuracy using different learning algorithms on the
Clothing1M datasets (real-world noise with 10% and 20% outliers respectively), where n = 5 images are selected from novel classes in order to perform few shot learning.
As it can be seen from
Table 6, our RCFSL continues to serve as the best performing method in the presence of semantic label noise compared to other competing methods under different noise statistics, which further conﬁrms the outstanding learning capability of the proposed approach in robust semi-supervised few-shot learning.
Moreover, Figure 5 illustrates exemplary images with compound noise detected
[Table]
F E VALUATION ON O MNIGLOT D ATASET
We further evaluate the proposed algorithm on Omniglot dataset.
The Omniglot dataset includes 1623 different handwritten characters from 50 different alphabets where each of the 1623 characters was drawn online via Amazon’s Mechanical Turk by 20 different people and each image is paired with stroke data, a sequences of spatial and temporal coordinates with in milliseconds.
Table 7 demonstrates the comparison of classiﬁcation accuracy using different learning algorithms on the Omniglot dataset (20% symmetric label noise with 10% outliers) for semi-supervised few shot learning on 1000 test episodes, where it can be seen from Table 7 that our RCFSL provides the best performance against other competing methods in the presence of noisy labels and outliers in semi-supervised few-shot learning.
[Table] [Table]
IWAE, L=2, IW=1 Burda et al.
IWAE, L=2, IW=50 Burda et al.
VAE+VGP, L=2 Tran et al.
LVAE, L=5, IW=
Sønderby et al.
ADGM, L=1, IW=
Maaløe et al.
ADGM, L=1, IW=2 Maaløe et al.
RCFSL+ KL divergence, L=1, IW= -83.
RCFSL (robust divergence), L=1, IW= -81.
G C OMPARISON OF L OG - LIKELIHOOD WITH R OBUST D IVERGENCE
We report the roust lower bound for the unlabeled data with 5000 importance weighted samples where the similar setting as Rasmus et al.
(2015) with warm up, batch normalization and 1 Monte
Carlo and IW sample for training.
The percentage of outliers is set to be 20%.
Table 8 demonstrates the log-likelihood scores for the permutation invariant MNIST dataset.
The results shown in Table 8 indicates the the proposed method has strong expressive power by performing better than other methods in terms of log-likelihood due to the utilization of the robust divergence in the inference, especially by comparing with the same RCFSL method but with KL-divergence.
H A NALYSIS FOR D IFFERENT T ERMS IN THE O PTIMIZATION F UNCTION
In the above optimization cost function of RCFSL in the equation (10), the ﬁrst term represents the supervised and unsupervised loss from the discriminator, the second and the third terms tend to minimize the approximation error by ”regularizing”
q ( H | x, t , ˜ y ) and q ( H u | x, t ) to the prior p ( H ) and p ( H u ) for labeled data and unlabeled data respectively.
The fourth term characterizes the generator loss from the robust semi-supervised GAN (RSGAN).
The ﬁfth to the eighth terms are employed to enhance the robustness in the presence of data outliers with β -divergence for the ELBOs on the