L EARNING SHARED MANIFOLD REPRESENTATION OF IMAGES AND ATTRIBUTES FOR GENERALIZED ZERO - SHOT LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The most prior methods of zero-shot learning have realized predicting labels of unseen images by learning a mapping from images to pre-deﬁned class-attributes.
However, recent studies show that these approaches severely suffers from the issue of biased prediction under the more realistic generalized zero-shot learning (GZSL) scenarios, i.e., their classiﬁer tends to predict all the examples from both seen and unseen class as one of the seen classes.
The cause of this problem is that we can not obtain training data of the unseen class and that the representation of attributes is poor.
To solve this, we propose a concept to learn a mapping that embeds both images and attributes to a space that is robust to such representations and generalized even for unseen data, which we refer to shared manifold learning .
Furthermore, we propose modality invariant variational autoencoders , which can perform shared manifold learning by training variational autoencoders with both images and attributes as inputs.
The empirical validation of well-known datasets in GZSL shows that our method achieves the signiﬁcantly superior performances to the existing relation-based works.
I NTRODUCTION
The recent high performance of deep neural networks on image classiﬁcation and object recognition depends greatly on whether we can obtain sufﬁciently labeled images of classes to predict.
However, it is difﬁcult to operate such a AI system as-is in the real world because the number of existing classes is huge and, as long as human beings create or develop new objects, their number can be expected to continue to increase day by day, resulting in difﬁculty obtaining labeled data of all these classes.
Human beings have the excellent ability to infer objects that have hardly seen by exploiting and transferring the knowledge learned in the past so as to survive in the harsh real world.
In the machine learning context, this problem is formulated as few-shot learning.
Zero-shot learning (ZSL) is a more extreme problem setting of few-shot learning proposed by Lampert et al.
(2009), which is to train by labeled dataset from certain classes called seen classes, and then to predict completely unseen classes not included in the trained dataset.
It is usually accomplished by preparing pre-deﬁned semantic representations of all classes such as attributes, and learning the mapping from images to class-attributes (Lampert et al., 2009; 2014; Frome et al., 2013;
Akata et al., 2013; 2015; Romera-Paredes & Torr, 2015; Kodirov et al., 2017).
However, this conventional setting has a strong premise that the test dataset does not include the seen classes.
That setting is unnatural in application to the real world because the objects we encounter are not always new but are rather likely to have been experienced in the past.
Therefore, to apply the ZSL system to the real world, one should change to a more natural setting that the test dataset can include examples of both seen and unseen classes, which is known as generalized zero-shot learning (GZSL)
(Chao et al., 2016).
Although the setting of GZSL is more natural, it has been conﬁrmed that existing relation-based approaches suffer greatly in terms of performance (Chao et al., 2016; Xian et al., 2018).
The main reason for this difﬁculty is that the training dataset does not include examples of the unseen class, leading to biased prediction of attributes.
Therefore, to archive GZSL, it is necessary to consider how to generalize to unseen data without training them.
Moreover, Chao et al.
(2016) found that improvement of the representation of attributes contributes to the higher performance of GZSL.
[Figure]
However, because we can not always obtain such an appropriate representation, it seems better to learn more robust representation from them.
In light of them, we propose a new solution of GZSL called shared manifold learning .
In shared manifold learning, we learn a mapping that embeds each image and attribute in the same space and representation so that it is robust against representation of inputs,
i
[Figure]
e., shared representation learning of images and attributes.
In addition, we make sure that this mapping is also generalized, which means the inputs of the unseen class are embedded in separate places from examples of other classes.
That is, it can be regarded as manifold learning of shared representation.
If such shared manifold learning can be done properly, we can estimate the corresponding class-attributes from the test image by embedding it in the shared manifold representation whether the class of this image is seen or unseen, that is, the bias problem can be solved.
To realize this, we propose modality invariant variational autoencoders (MIVAE), which are based on variational autoencoders (VAE) (Kingma & Welling, 2013; Rezende et al., 2014).
In this method, ﬁrst of all, to obtain shared representations from inputs of images x and class-attributes a y with different dimensions and distributions, we deﬁne mappings from each input to the latent space z as the deep probabilistic distributions, q λ ( z | x ) and q λ ( z | a y ) , which enable nonlinear embedding considering different distributions of the respective inputs.
Next, to learn manifold representation from these two inputs, we prepare a VAE with both images and attributes as inputs.
In learning of
MIVAE, we not only maximize the lower bound of this VAE but also learn so that each mapping approximates the encoder of VAE, q φ ( z | x , a ) , and so that each mapping also approaches another, which enables us to capture the mappings that embed inputs in the shared manifold representations.
See Figure 1 as the outline of shared manifold learning with MIVAE.
If we consider images and attributes as different modalities , then we can regard MIVAE as an extension of works examining multimodal VAE (Suzuki et al., 2016; Vedantam et al., 2017).
As a method for solving the bias-related shortcoming of GZSL, a synthesis-based method has been proposed recently (Mishra et al., 2017; Verma et al., 2018), which is to generate images from classattributes with the conditional deep generative model and to learn the classiﬁer from the generated image to the labels of seen and unseen classes.
Although this method is known to give high performance to GZSL, it is necessary to generate a plurality of images for each class.
Moreover, it is difﬁcult to extend because the representation of the input image is restricted to make it easier to generate.
Therefore, we believe that our proposed method is more widely available.
The contribution of this research is as follows.
• We discuss the reasons why existing methods do not work in GZSL and propose shared manifold learning as a concept to solve the problem.
• We propose MIVAE, which extended VAE as a method to realize shared manifold learning.
• We conﬁrm that MIVAE can perform shared manifold learning properly and that it can contribute to the higher performance of GZSL compared to existing ZSL methods based on shared representation learning.
• We demonstrate that MIVAE has markedly higher performance than state-of-the-art relation-based ZSL methods on several benchmark datasets.
P ROBLEM FORMULATION
We assume that the dataset D tr = { x i , y i } N tr i =1 is given as the training set, where x i ∈
X is the input data, e.g. an image, and y i ∈
Y s = { 1 ..., S } is the corresponding label data.
The objective of ZSL is to train the classiﬁer using D tr and to predict labels ˆ y j ∈ Y from the example x j ∈ X in the test set D ts = { x j } N ts j =1 .
In the original ZSL, it is assumed that the classes of the test set are completely unseen in the training set, which means Y =
Y u
= { S + 1 , ..., U } .
However, our goal is to train the setting of GZSL, which includes both seen and unseen classes in the test set ( Y = Y s ∩ Y u ).
In the context of ZSL, we also assume that we have the class-attribute matrix A ∈ R M × ( S + U ) as the semantic information of classes, where each column represents the M -dimensional attribute vector
a c ∈ A = R M of each class c = 1 , ..., S + U .
Using this attribute vector, the training set D tr can be replaced as { x i , a y i } N tr i =1 .
P ROPOSED METHOD 3. S HARED MANIFOLD LEARNING
GZSL is expected to be a more difﬁcult problem than conventional ZSL because the number of candidate classes to predict becomes much greater.
In fact, it is conﬁrmed that the usual method of learning the relation between images and attributes severely fails to predict unseen classes (Chao et al., 2016; Xian et al., 2018).
The direct reason for this difﬁculty is that the training dataset does not include examples of the unseen class, leading to biased map that classiﬁes all images into only seen classes.
Moreover, Chao et al.
(2016) argues that improvement of the representation of attributes is vital for GZSL, because replacing the attributes with the image features as ideal semantic representation improves the performance of GZSL.
However, we can not always obtain such an appropriate semantic representation.
In this paper, we consider a learning method of another space that is robust to its representation and also generalizes to unseen data.
First, we consider mapping from each image and attribute to the same space, not mapping from images to attributes, so that the relation between images and attributes is robust against the representation of the attributes.
At this time, we enforce the images and attributes to be the same representations in the embedding space (shared representation learning).
This representation not only becomes robust against attribute representation but also becomes more informative by integrated with the images.
Next, we make sure that the shared representation also generalizes to unseen data (manifold learning of shared representation).
For example, given the image and attributes of the unseen class, we make sure that they retain the same representation in the embedding space, and that they are embedded in separate places from examples of other classes.
For this study, we designate these series of learning as shared manifold learning of images and attributes.
We claim that we can deal with the bias-related shortcoming of GZSL by learning shared manifold representation.
3. M ULTIMODAL INVARIANT VARIATIONAL AUTOENCODER
Next, we propose modality invariant VAE (MIVAE), which is a method of learning mapping from two inputs to a space to be shared and manifold representations.
Here, we consider that the dataset of ZSL { x i , a y i } N tr i =1 has different modalities , which means that x and a y are different in their spaces and distributions, i.e., X (cid:54) =
A and p ( X ) (cid:54) =
p ( A ) ,where
X = { x 1 , ..., x N tr } and A = { a y 1 , ..., a y Ntr } .
At ﬁrst, we consider two Gaussian distributions parameterized with deep neural networks q λ ( z | x ) and q λ ( z | a y ) as mappings from each of X and A to Z , where λ is the parameter of the distribution.
See Appendix A for the way of parameterization with deep neural networks.
In MIVAE, we aim to learn so that these mappings embed inputs in shared manifold representation.
Note that it is important to set these mappings to the deep probabilistic distributions in obtaining shared representations.
For a detailed discussion on this, see Appendix C.
Next, we use variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014), which has been widely used as a model of representation learning and manifold learning.
Moreover, we consider VAE with two modalities as input, which is called joint VAE (Vedantam et al., 2017).
Joint VAE has a latent concept z that is common to the two modalities x , a y , each of which is generated conditionally and independently from the concept.
That is, their generative processes are z ∼ p ( z ) and x , a y ∼ p θ ( x , a y | z )
= p θ ( x | z ) p θ ( a y | z ) , where θ corresponds to model parameters.
These processes indicate that the latent variable z can be regarded as a shared representation containing different modality information.
To train this, we should maximize the joint distribution p ( x , a y )
= (cid:82) p θ ( x , a y | z ) p ( z ) d z over the training set.
However, it is difﬁcult to perform this maximization directly because this distribution is intractable.
Therefore, we instead maximize the following evidence lower bound (ELBO).
[Equation]
where q φ ( z | x , a y ) is an approximate distribution of posterior and φ represents parameters.
We call q φ ( z | x , a y ) as encoder and p θ ( x | z ) and p θ ( a y | z ) as decoder.
These models are parameterized by deep neural networks.
The distribution of the encoder is Gaussian, and that of the decoder depends on the distribution of the input data.
From the encoder of trained VAE, we can extract shared representations that integrate the information of different modalities.
In addition, the VAE can obtain the manifold representation of the input in the latent space and appropriately place the unseen input on the latent space.
Because we use images and attributes as inputs to joint VAE here, we can obtain the encoder that integrates and maps them into manifold representation, i.e., we can perform shared manifold learning .
Therefore, if we train each modality’s distribution to minimize the distance between the encoder, then we can obtain such representations from the distribution of each modality.
We deﬁne this distance using Kullback-Liebler (KL) divergence as
[Equation]
Intuitively, because the encoder has more information to be conditioned, it seems that the mapping distributions, especially q λ ( z | a )) , might degenerate if each is brought closer to the encoder.
Actually, Vedantam et al.
(2017) showed that minimization of E p data ( x , a ) [ D KL ( q φ ( z | x , a ) ||
q λ ( z | a ))] is equivalent to minimization of E p data ( x , a )
[ D KL ( q avgφ ( z | a ) ||
q λ ( z | a ))] , where q avgφ ( z | a ) =
E p data ( x | a ) [ q φ ( z | x , a )] , i.e., this is the average of the encoder over all x corresponding to a .
Therefore, the minimization of equation 2 encourages each mapping to close to the distribution complementing the other modality of the encoder, thereby resulting in stabilization.
When equation 2 becomes minimum, q λ ( z | x )
=
q φ ( z | x , a ) =
[Equation]
λ ( z | a ) is satisﬁed.
However, if the objective function of VAE is also optimized simultaneously, then two mapping distributions might not be brought close to each other explicitly during training.
Therefore, to force the two distributions to be explicitly close at the training time, we also minimize the following equation:
[Equation]
Therefore, the objective function of MIVAE is
[Equation]
where λ dist and λ map are hyperparameters.
When λ map = 0 , this objective function is the same as JMVAE (Suzuki et al., 2016), which means that this model can be regarded as an extension of
JMVAE.
After training MIVAE, we can predict the labels of test images simply as ˆ y j
[Equation]
R ELATED WORKS
ZSL is one of the greatest attention problems among image recognition tasks, and various methods have been proposed.
Here, we explain them separately on relation-based and synthetic-based approaches.
[Table]
4. R ELATION - BASED
Relation-based methods are intended to train relations between images and class-attributes F ( x, a y ) from training data and to predict a class whose relation is strong in test data, i.e. ˆ y = F ( x, a y ) .
This approach can be classiﬁed further into the following two methods: training of mapping from image space to attribute space (image-attribute projection) and training of shared representation that embed both image and attribute space (shared representation learning).
Image-attribute projection In ZSL, the simplest and most widely adopted approach is to obtain mapping from image space to class-attribute space with training data.
Some early studies used classiﬁers such as SVM to learn mapping from images to class attributes (Lampert et al., 2009; 2014), but many other methods use linear embedding into attributes (or semantic representations) and typically set a ranking loss as the objective function to train such embedding (Frome et al., 2013;
Akata et al., 2013; 2015; Romera-Paredes & Torr, 2015; Kodirov et al., 2017).
Actually, RomeraParedes & Torr (2015) adds a regularization term of the mapping to this objective function.
Also,
Kodirov et al.
(2017) minimizes the embedding error in both directions with the objective.
Recently, nonlinear models are also used(Socher et al., 2013) to consider nonlinearity between images and attributes.
However, classiﬁers tend to predict only one class from many images at testing, which is known as the hubness problem because of the difference in the distribution of the two modalities.
To avoid this difﬁculty, some works have proposed training of reverse mapping from attribute to image space with such as deep neural networks (Zhang et al., 2017).
Verma & Rai (2017) suggests that mapping from attributes to an image are a conditional distribution by exponential family rather than deterministic, which can incorporate consideration of uncertainty among modalities.
Therefore the hubness problem can be relaxed further.
Shared representation learning
Another means of obtaining relations is to learn mapping that embeds images and attributes into a shared latent representation.
This learning can be realized by shared representation learning in which images and attributes are regarded as different modalities.
In
ZSL, several methods of learning shared embedding spaces have been proposed Zhang & Saligrama (2015; 2016), but few methods use deep neural networks as embedding maps.
VZSL (Wang et al., 2017) trains shared representations of images and attributes using deep generative models, which most closely approximates our method.
As with Verma & Rai (2017), uncertainty can be taken into consideration, and furthermore, deep neural networks are used.
Although these relation-based methods work well in conventional ZSL settings, in GZSL, their performance is much worse, as shown in Chao et al.
As described in Chapter 1, one of the reason is that the representation of attributes is poor.
The approach of shared representation learning is robust to the representation of attributes, but it does not perform manifold learning at two inputs, which means that it does not perform shared manifold learning.
Indeed, VZSL performs manifold learning only with image input, which results in poor generalization performance in examples of unseen classes, as shown later in an experiment.
4. S YNTHESIS - BASED
Recently, several synthesis-based methods have been proposed (Verma et al., 2018; Mishra et al., 2017).
These approaches are attempts to generate images from class-attributes and to train a classiﬁer of unseen classes by these synthesis samples.
To generate samples from attributes properly, they train conditional deep generative models of images given class-attributes in training data.
It is apparently difﬁcult to generate realistic test images that are useful to train the classiﬁer.
However, in many ZSL settings, it is common to use feature extraction representations from images as input so that they can generate these inputs with high quality.
The greatest advantage of the synthesis-based approaches is that they can achieve high performance in a GZSL setting because we train classiﬁcation with a new classiﬁer after generating all samples of seen or unseen classes from the trained deep generative model.
For that reason, we no longer need [Table] to consider differences between seen and unseen classes.
However, training the classiﬁer requires generation of multiple images of all classes.
Therefore, if the image size increases or the number of classes increases, then it might be costly to generate these synthetic samples.
Moreover, this method works well only if the input representations of images are easy to generate.
In recent years, several deep generative models have been proposed that can produce high-quality images (Kingma & Dhariwal, 2018; Karras et al., 2017), but it still might be difﬁcult to generate arbitrary types of inputs to be useful for classiﬁcation of unseen classes.
Therefore, in this paper, we take the relation-based approach and propose the concept of shared manifold learning to deal with the issue of GZSL.
Table 1 presents a brief comparison of our method with others.
E XPERIMENTS
For experiment, we use the following four datasets commonly used for ZSL: Animals with Attributes (AWA1) (Lampert et al., 2014), CUB-200-2011 Bird (CUB)
(Wah et al., 2011), SUN Attribute (SUN) (Patterson & Hays, 2012), and Attribute Pascal and Yahoo (aPY) (Farhadi et al., 2009).
In addition, because the images of AWA1 were not public available, Xian et al.
(2018) additionally introduced Animals with Attributes 2 (AWA2) composed of images collected from public sites.
Therefore, in this experiment, we will use ﬁve datasets including AWA2.
Each dataset contains an image of each class, where each class is represented by semantic attributes.
Table 2 presents the numbers of images, attributes, and classes in respective datasets.
For fair comparisons, we use 2048-dim top-layer embedding of the 101-layered ResNet (He et al., 2016) provided by Xian et al.
(2018) as the image vector.
Furthermore, for the class-attribute representation we use the continuous valued attributes between 0 and 1 provided with each dataset.
We followed Xian et al. (2018) for the splits of images of each class at training and test times.
For the metric of evaluation at the test time, we use the average per-class top-1 accuracy on both the seen and unseen classes (referred as acc Y s and acc Y u ).
In addition, to evaluate the performance on GZSL, we calculate the harmonic mean of acc Y s and acc Y u , which is H =
(2 · acc Y s · acc Y u ) / ( acc Y s + acc Y u ) .
We used the Adam optimization algorithm (Kingma & Ba, 2014) with a learning rate of 10 − 3 .
Also, in order to suppress over-ﬁtting, we applied batch normalization to each layer and added L2 regularization to the parameters, where the coefﬁcient of this term is 10 − 4 .
For details of network architectures, see the appendix.
In all experiments, we set λ dist = 1 , which works well in multimodal learning according to reports of related studies (Suzuki et al., 2016; Vedantam et al., 2017).
For another parameter λ map , we will verify the effect by changing this value in the experiments.
All models in this paper were implemented with PyTorch (Paszke et al., 2017).
We conduct two experiments.
First, compared with shared representation learning, we demonstrate that MIVAE with shared manifold learning can learn with high performance qualitatively and quantitatively in GZSL.
The second experiment compares our method with the state-of-the-art relationbased methods and conﬁrms that our method has much higher performance than those methods.
5. C OMPARISON WITH SHARED REPRESENTATION LEARNING METHODS
To the best of our knowledge, VZSL (Wang et al., 2017) is the only existing method of ZSL used to train shared representation learning with deep probabilistic embedding maps, q λ ( z | x ) and q λ ( z | a ) .
Furthermore, we consider the following three methods including VZSL as shared learning compare.
Minimization of embedded divergence (MED)
This simply minimizes KL divergence of the two distributions, i.e. the loss function of MED is D KL ( q λ ( z | x ) ||
q λ ( z | a )) .
MED-VAE To train the manifold representation in the latent space, we add the reconstruction error of the VAE whose input is the image vector to that of the MED, i.e.,
[Equation]
VZSL (Wang et al., 2017)
Wang et al.
(2017) proposed margin regularization that encourages separation of the true class from the next class.
The loss function of VZSL is the addition of the margin regularization to that of MED-VAE.
See Wang et al.
(2017) for more mathematical details.
This experiment used AWA2 for the dataset.
The distribution of the embedding maps was
Gaussian.
The networks structures were set the same in all methods.
Furthermore, the prediction methods of the classes of each example in the test data are all the same: ˆ y j =
[Equation]
Figure 2 shows learning curves obtained with these three methods and MIVAE in GZSL.
First, for seen classes, all models except MED have nearly equal accuracy.
However, because the accuracy of MED is markedly low, it is apparent that manifold learning on latent representation by VAEs is important.
In MIVAE, we conﬁrm that the accuracy score is higher when λ map is set to 1.
Next, the result of the unknown classes shows that both the MED and the MED-VAE deteriorate considerably during early training.
Regarding MED, failure to train generalized representations during training caused bias to known classes, leading to bad prediction in unknown classes.
However, in
MED-VAE, accuracy becomes very poor despite manifold training of images, probably because we were unable to train the connection between attributes and images on the embedding space.
VZSL tries to link images and corresponding attributes more closely during training, which contributes to making them better than MED-VAE.
However, as training progresses, the accuracy drops, i.e., the training becomes unstable.
Finally, it is apparent that MIVAE is markedly more accurate than these methods.
From these results, we conﬁrmed that, to predict the unseen class in the GZSL setting it might be important to train a shared manifold representation that integrates images and attributes.
Moreover, we found that MIVAE can properly obtain such a representation.
The result of the harmonic mean is almost identical to that of the accuracy of unseen classes.
In addition, in these AWA2 experiments, the MIVAE’s results on accuracy of the unseen classes and the harmonic mean are apparently less dependent on the λ map value.
Figure 3 shows representations mapped respectively by the embedding distribution trained with
MED-VAE, VZSL, and MIVAE.
From this result as well, it is apparent that MIVAE can obtain representations of seen and unseen classes in the latent space.
These results clarify that shared representation learning is insufﬁcient in GZSL and clarify that shared manifold learning is important for high performance.
5. C OMPARISON WITH STATE - OF - THE - ART METHODS
Next, we compare MIVAE with state-of-the-art relation-based ZSL.
All results of these methods were reprinted from Xian et al.
[Figure]
[Table] As Table 3 shows [Table] our proposed method largely outperforms all existing methods of relation-based.
Particularly, Table 3 shows that performance is poor in datasets with few seen classes such as AWA (both 1 and 2) and aPY.
By contrast MIVAE, despite being a relation-based method, can properly g eneralize unseen classes even in these difﬁcult datasets.
Relation-based methods are known to tend to be biased to seen classes, so that the results of unseen classes become considerably worse (Chao et al., 2016). However, since our results are much greater than those, we can see that our approach solves the bias problem of GZSL.
Furthermore, for seen classes, the performance of our method is equal to or higher than that of other methods. For harmonic mean, the proposed method is far superior to the others. Also, for seen classes, our method is equal to or higher than other methods, so that for harmonic mean, the proposed method is much superior to the others. For the results of accuracy on both seen and unseen classes, see Appendix D.
In MIVAE, the result of λ map = 1 is better than that of λ map = 0 in SUN and CUB, but there is not much difference between them in terms of AWA and aPY; rather it is better when λ map = 1 .
This result is probably related to the number of seen classes in the dataset, i [Table] e., if the number of seen classes is large, a positive inﬂuence on prediction of the seen class, e.g. minimizing equation 3 at training time, will also have a positive inﬂuence on that of the unseen classes.
Note that we do not reprint the results of VZSL in Wang et al.
(2017) to Table 3 because the training/test split and the evaluation metric in Wang et al.
(2017) is different from here.
In our reproduction of VZSL in the same split and metric here, the harmonic means in each dataset and their average (SUN, CUB, AWA1, AWA2, aPY, Average) are (20.9, 30.7, 16.7, 18.7, 11.6, 19.7) respectively, which are all inferior to our method.
Finally, we evaluated the performance in the conventional ZSL setting.
As Table 4 shows, our proposed method has nearly the same performance as that of existing methods in most datasets.
Therefore, our method, shared manifold learning, deals directly with the bias-related shortcoming of a relation-based approach on GZSL. [Table]
C ONCLUSION
In this paper, we addressed the setting of generalized zero-shot learning (GZSL) and speciﬁcally demonstrated that shared manifold learning is important to resolve the bias-related shortcoming.
Furthermore, to demonstrate that point, we proposed modality invariant variational autoencoders (MIVAE), which realizes shared manifold learning using variational autoencoders that take both images and attributes as inputs.
We ﬁrst compared MIVAE with some ZSL methods that perform only shared representation learning.
Thereby, we conﬁrmed that performing manifold learning with both images and attributes is important for good performance of GZSL.
Next, we experimented with several benchmark datasets.
Results show that MIVAE has markedly higher performance than state-of-the-art relation-based ZSL methods, which suggests that the bias difﬁculty can be mitigated.
Furthermore, the fact that the performance in the conventional ZSL is not much different from that of the existing method means that MIVAE can directly address difﬁculties posed by GZSL.
Actually,
MIVAE is a robust and extensible method for the input information format such that we can address more realistic and complicated problems.
For example, it might be possible to accommodate text information as more sophisticated side information of classes with an encoder-decoder RNN model.
A P ARAMETERIZATION OF DISTRIBUTIONS WITH DEEP NEURAL NETWORKS
The Gaussian distribution can be parameterized with deep neural networks as
[Equation]
where f µ and f σ 2 are respectively denote linear single layer neural networks and where f MLP represents a deep neural network with an arbitrary number of layers.
Moreover, applying the softplus function for each element of a vector is denoted as Softplus .
The Bernoulli distribution is parameterized as
[Equation]
where Sigmoid is represents the sigmoid function.
B N ETWORK ARCHITECTURES
For the notation of model structures, we denote a linear fully-connected layer with k units, batch normalization, and ReLU as DkBR .
Also, we denote DkBR without batch normalization and ReLU as
Dk .
In addition, the process of applying J after I is denoted as I-J , and the process of concatenating the last layers of the two networks I, J into one layer is denoted as (I,J) .
Therefore, the network structures of distributions of MIVAE are as follows:
• p ( x | z ) (Gaussian, where σ 2 = 1 ) – f µ : D – f MLP : z-D1000BR • p ( a | z ) (Bernoulli) [Table] – f µ : DdimA – f MLP : z-D1000BR • q ( z | x , a ) (Gaussian) – f µ and f σ 2 : D – f MLP : (x,a)-D1000BR-D1000BR • q ( z | x ) (Gaussian) – f µ and f σ 2 : D – f MLP : x-D1000BR-D1000BR • q ( z | a ) (Gaussian) – f µ and f σ 2 : D – f MLP : a-D1000BR-D1000BR
C D ISCUSSION ON PROBABILISTIC MAPPING TO OBTAIN SHARED REPRESENTATION
In shared representation learning on ZSL, we consider nonlinear functions that map each modality as input into the same space Z , i.e. f x : X → Z and f a : A → Z .
If each mapping can be trained so that the same representation can be extracted from each modality of the same example, i.e. ∀ i z xi =
z ay i , where z xi =
f
x ( x i ) and z ay i =
f a ( a y i ) , we should be able to predict labels of the images in the test set by selecting the class that corresponds to the closest representation, i.e. ˆ y j = arg min y ∈Y dis ( z xj , z ay ) , where dis ( a, b ) is an arbitrary distance function between a and b .
Therefore, the simplest approach is to train the mappings so that the distance of the representations of the two modalities is small on the training set, i.e. min f x ,f a (cid:80) N tr i =1 dis
( f x ( x i ) , f a ( a y i )) .
However, this method is highly likely to fail for the following two reasons.
First, although only one example of attributes corresponding to a class exists, there are countless examples of images corresponding to that class.
Therefore, if we attempt to make representations of attributes and images contained in one class closer, then these representations will degenerate to one point.
Another reason is that if one can train to correspond well between each modality of training data, then that representation might not generalize to test data.
To solve these problems, we consider the uncertainty of each example of each modality.
To account for this uncertainty, we replace the deterministic embedding functions, f x and f a with the deep probabilistic distributions, p ( z i | x i ) and p ( z i | a y i ) .
The beneﬁts of considering distributions for each example are twofold.
First, we can prevent the degeneracy of representation which occurs when making examples of modalities that do not closely correspond one-to-one.
This is true because we can consider the distance between distributions of each example, not between each example.
Second, minimizing the distance between distributions of examples of each modality implies that the distribution of each modality is closer, i.e.,, ∀ i ∈ { 1 ..., N tr } [ p ( z i | x i ) =
p ( z i | a y i )] ⇒ p
( Z x ) =
p ( Z a ) , where Z x = { z x 1 ..., z xN tr } and Z a = { z ay 1 ..., z ay Ntr } .
This point might be readily apparent from (cid:81)
N tr i =1 p ( z i
| x i ) =
p ( Z | X ) = p ( Z x ) .
Therefore, if we can train to bring each distribution closer, then we can obtain shared representation of them.
D C OMPARISON WITH STATE - OF - THE - ART METHODS IN ALL METRICS
Table 5 shows that results evaluated with all metrics.
Relation-based methods are known to tend to be biased to seen classes, so that the results of unseen classes become considerably worse (Chao et al., 2016).
However, since our results are much greater than those, we can see that our approach solves the bias problem of GZSL.
Furthermore, for seen classes, the performance of our method is equal to or higher than that of other methods.
For harmonic mean, the proposed method is far superior to the others.
Also, for seen classes, our method is equal to or higher than other methods, so that for harmonic mean, the proposed method is much superior to the others.