Original: R ELATION - BASED G ENERALIZED Z ERO - SHOT C LASSIFICATION WITH THE D OMAIN D ISCRIMINATOR ON THE SHARED REPRESENTATION
Revised: R ELATION - BASED G ENERALIZED Z ERO - SHOT C LASSIFICATION WITH THE D OMAIN D ISCRIMINATOR ON THE SHARED REPRESENTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generalized zero-shot learning (GZSL) is the task of predicting a test image from seen or unseen classes using pre-deﬁned class-attributes and images from the seen classes.
Revised: Generalized zero-shot learning (GZSL) is the task of predicting a test image from seen or unseen classes using pre-deﬁned class-attributes and images from the seen classes.

Original: Typical ZSL models assign the class corresponding to the most relevant attribute as the predicted label of the test image based on the learned relation between the attribute and the image.
Revised: Typical ZSL models assign the class corresponding to the most relevant attribute as the predicted label of the test image based on the learned relation between the attribute and the image.

Original: However, this relation-based approach presents a difﬁculty: many of the test images are predicted as biased to the seen domain, i.e., the domain bias problem .
Revised: However, this relation-based approach presents a difﬁculty: many of the test images are predicted as biased to the seen domain, i.e., the domain bias problem .

Original: Recently, many methods have addressed this difﬁculty using a synthesis-based approach that, however, requires generation of large amounts of high-quality unseen images after training and the additional training of classiﬁer given them.
Revised: Recently, many methods have addressed this difﬁculty using a synthesis-based approach that, however, requires generation of large amounts of high-quality unseen images after training and the additional training of classiﬁer given them.

Original: Therefore, for this study, we aim at alleviating this difﬁculty in the manner of the relation-based approach.
Revised: Therefore, for this study, we aim at alleviating this difﬁculty in the manner of the relation-based approach.

Original: First, we consider the requirements for good performance in a ZSL setting and introduce a new model based on a variational autoencoder that learns to embed attributes and images into the shared representation space which satisﬁes those requirements.
Revised: First, we consider the requirements for good performance in a ZSL setting and introduce a new model based on a variational autoencoder that learns to embed attributes and images into the shared representation space which satisﬁes those requirements.

Original: Next, we assume that the domain bias problem in GZSL derives from a situation in which embedding of the unseen domain overlaps that of the seen one.
Revised: Next, we assume that the domain bias problem in GZSL derives from a situation in which embedding of the unseen domain overlaps that of the seen one.

Original: We introduce a discriminator that distinguishes domains in a shared space and learns jointly with the above embedding model to prevent this situation.
Revised: We introduce a discriminator that distinguishes domains in a shared space and learns jointly with the above embedding model to prevent this situation.

Original: After training, we can obtain prior knowledge from the discriminator of which domain is more likely to be embedded anywhere in the shared space.
Revised: After training, we can obtain prior knowledge from the discriminator of which domain is more likely to be embedded anywhere in the shared space.

Original: We propose combination of this knowledge and the relationbased classiﬁcation on the embedded shared space as a mixture model to compensate class prediction.
Revised: We propose combination of this knowledge and the relationbased classiﬁcation on the embedded shared space as a mixture model to compensate class prediction.

Original: Experimentally obtained results conﬁrm that the proposed method signiﬁcantly improves the domain bias problem in relation-based settings and achieves almost equal accuracy to that of high-cost synthesis-based methods.
Revised: Experimentally obtained results conﬁrm that the proposed method signiﬁcantly improves the domain bias problem in relation-based settings and achieves almost equal accuracy to that of high-cost synthesis-based methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The recent high performance of deep neural networks on image classiﬁcation and object recognition depends greatly on whether one can obtain sufﬁciently labeled images of classes to predict.
Revised: The recent high performance of deep neural networks on image classiﬁcation and object recognition depends greatly on whether one can obtain sufﬁciently labeled images of classes to predict.

Original: Nevertheless, it is difﬁcult to do this in the real world because the number of existing classes is enormous.
Revised: Nevertheless, it is difﬁcult to do this in the real world because the number of existing classes is enormous.

Original: As long as human beings create or develop new objects, their number might continue to increase daily, thereby creating difﬁculty in obtaining labeled data of all classes to predict.
Revised: As long as human beings create or develop new objects, their number might continue to increase daily, thereby creating difﬁculty in obtaining labeled data of all classes to predict.

Original: In recent years, this difﬁculty led to great interest in zero-shot learning (ZSL) (Farhadi et al., 2009;
Revised: In recent years, this difﬁculty led to great interest in zero-shot learning (ZSL) (Farhadi et al., 2009;

Original: Frome et al., 2013; Lampert et al., 2014; Xian et al., 2018a), which is training by a labeled set from certain classes called seen classes and then predicting completely unseen classes that are not included in the training set.
Revised: Frome et al., 2013; Lampert et al., 2014; Xian et al., 2018a), which is training by a labeled set from certain classes called seen classes and then predicting completely unseen classes that are not included in the training set.

Original: Usually, ZSL is accomplished by preparing pre-deﬁned semantic representations of all classes, such as attributes, and learning the relation between images and the class-attributes ( relation-based approach).
Revised: Usually, ZSL is accomplished by preparing pre-deﬁned semantic representations of all classes, such as attributes, and learning the relation between images and the class-attributes ( relation-based approach).

Original: Once it is learned from the training set, we can predict the labels of test examples from unseen classes by selecting the most relevant attributes on this relationship.
Revised: Once it is learned from the training set, we can predict the labels of test examples from unseen classes by selecting the most relevant attributes on this relationship.

Original: However, it has been pointed out that this approach does not work for generalized zero-shot learning (GZSL), which is a more general setting where test samples can be from both seen and unseen classes (Chao et al., 2016).
Revised: However, it has been pointed out that this approach does not work for generalized zero-shot learning (GZSL), which is a more general setting where test samples can be from both seen and unseen classes (Chao et al., 2016).

Original: This is because all examples of the training set are obtained from the seen classes, so
Revised: This is because all examples of the training set are obtained from the seen classes, so

Original: the
Revised: the

Original: [Figure]
Revised: [Figure]

Original: data to be predicted as one of the unseen classes also has a strong relationship with the seen class attributes, resulting most of them assigned to one of the seen classes.
Revised: data to be predicted as one of the unseen classes also has a strong relationship with the seen class attributes, resulting most of them assigned to one of the seen classes.

Original: In this paper, we refer to a domain as belonging to either seen or unseen, and call the problem that class prediction is biased to the seen domain as the domain bias problem .
Revised: In this paper, we refer to a domain as belonging to either seen or unseen, and call the problem that class prediction is biased to the seen domain as the domain bias problem .

Original: To address this difﬁculty, recent works have taken an approach of learning a generative model that generates images from corresponding attributes, and of then training a classiﬁer to predict classes from the generated synthesis images (Mishra et al., 2017; Verma & Rai, 2017; Xian et al., 2018b;
Revised: To address this difﬁculty, recent works have taken an approach of learning a generative model that generates images from corresponding attributes, and of then training a classiﬁer to predict classes from the generated synthesis images (Mishra et al., 2017; Verma & Rai, 2017; Xian et al., 2018b;

Original: Felix et al., 2018).
Revised: Felix et al., 2018).

Original: The advantage of this synthesis-based approach is that samples of both domains are obtainable by generation, which contributes to alleviation of the domain bias problem.
Revised: The advantage of this synthesis-based approach is that samples of both domains are obtainable by generation, which contributes to alleviation of the domain bias problem.

Original: However, these synthesis-based methods require the generative model to generate numerous diverse and highquality images for each class, including unseen ones, sufﬁcient to classify with high-performance, which can be difﬁcult and costly.
Revised: However, these synthesis-based methods require the generative model to generate numerous diverse and highquality images for each class, including unseen ones, sufﬁcient to classify with high-performance, which can be difﬁcult and costly.

Original: In addition, this requires a classiﬁer that classiﬁes all classes from the generated images.
Revised: In addition, this requires a classiﬁer that classiﬁes all classes from the generated images.

Original: On the other hand, conventional relation-based methods can make class predictions using only the learned image–attribute relation, not requiring enormous image generation and additional classiﬁer training after training the model.
Revised: On the other hand, conventional relation-based methods can make class predictions using only the learned image–attribute relation, not requiring enormous image generation and additional classiﬁer training after training the model.

Original: Therefore, we address the following question in this paper:
Revised: Therefore, we address the following question in this paper:

Original: Can we mitigate the domain bias problem of GZSL in the relation-based manner and achieve high performance?
Revised: Can we mitigate the domain bias problem of GZSL in the relation-based manner and achieve high performance?

Original: We ﬁrst discuss the importance of the following requirements for good relation-based ZSL performance when embedding images and attributes in a shared representation space that satisﬁes the following requirements: images and attributes belonging to the same class must be in the same place ( modality invariance ); and different classes of samples must be separated in the shared space ( class separability ).
Revised: We ﬁrst discuss the importance of the following requirements for good relation-based ZSL performance when embedding images and attributes in a shared representation space that satisﬁes the following requirements: images and attributes belonging to the same class must be in the same place ( modality invariance ); and different classes of samples must be separated in the shared space ( class separability ).

Original: To achieve such embedding, we propose Modality-invariant and Classseparable Multimodal AutoEncoder (MCMAE) , which is an extension of variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014).
Revised: To achieve such embedding, we propose Modality-invariant and Classseparable Multimodal AutoEncoder (MCMAE) , which is an extension of variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014).

Original: The objective of MCMAE is designed based on the two requirements presented above.
Revised: The objective of MCMAE is designed based on the two requirements presented above.

Original: Next, we hypothesize that the domain bias problem results from a situation in which the unseen domain overlaps that of the seen one in the shared space.
Revised: Next, we hypothesize that the domain bias problem results from a situation in which the unseen domain overlaps that of the seen one in the shared space.

Original: To address this point, we explicitly introduce a discriminator for separation of these two domains.
Revised: To address this point, we explicitly introduce a discriminator for separation of these two domains.

Original: This discriminator is trained jointly with
Revised: This discriminator is trained jointly with

Original: MCMAE.
Revised: MCMAE.

Original: After training, it gives the probability of a test image being in a given domain.
Revised: After training, it gives the probability of a test image being in a given domain.

Original: In other words, it gives prior knowledge of the domain in the shared space.
Revised: In other words, it gives prior knowledge of the domain in the shared space.

Original: Based on this insight, we consider the class prediction probability as a soft combination of MCMAE classiﬁcation and consider the domain discriminator as a mixture model (see Figure 1).
Revised: Based on this insight, we consider the class prediction probability as a soft combination of MCMAE classiﬁcation and consider the domain discriminator as a mixture model (see Figure 1).

Original: Such combination-based classiﬁcation has been proposed as a “gating” approach (Atzmon & Chechik, 2019).
Revised: Such combination-based classiﬁcation has been proposed as a “gating” approach (Atzmon & Chechik, 2019).

Original: However, unlike this work, our method is able to train the entire model while retaining an end-to-end manner.
Revised: However, unlike this work, our method is able to train the entire model while retaining an end-to-end manner.

Original: We call this proposed approach as MCMAE with the Domain discriminator (MCMAE-D) .
Revised: We call this proposed approach as MCMAE with the Domain discriminator (MCMAE-D) .

Original: The contribution of this research is the following.
Revised: The contribution of this research is the following.

Original: (1) We consider the requirements of the shared representation space to perform in the ZSL setting and introduce MCMAE as a model to learn the embedding of images and attributes into that space.
Revised: (1) We consider the requirements of the shared representation space to perform in the ZSL setting and introduce MCMAE as a model to learn the embedding of images and attributes into that space.

Original: (2) We also propose MCMAE-D, which combines MCMAE with the domain discriminator.
Revised: (2) We also propose MCMAE-D, which combines MCMAE with the domain discriminator.

Original: The experiment results demonstrate that it greatly reduces the domain bias problem, thereby contributing to exceed the performances of the existing relation-based models greatly and to be equivalent to the state-of-the-art synthesis-based method.
Revised: The experiment results demonstrate that it greatly reduces the domain bias problem, thereby contributing to exceed the performances of the existing relation-based models greatly and to be equivalent to the state-of-the-art synthesis-based method.

Original: [Figure]
Revised: [Figure]

Original:  P ROBLEM F ORMULATION : G ENERALIZED Z ERO -S HOT L EARNING
Revised:  P ROBLEM F ORMULATION : G ENERALIZED Z ERO -S HOT L EARNING

Original: We assume that the dataset D tr = { x i , y i } N tr i =1 is given as the training set, where x i ∈
Revised: We assume that the dataset D tr = { x i , y i } N tr i =1 is given as the training set, where x i ∈

Original: X is the input data, e.g. an image, and where y i ∈
Revised: X is the input data, e.g. an image, and where y i ∈

Original: Y s = { 1 ..., S } is the corresponding label data.
Revised: Y s = { 1 ..., S } is the corresponding label data.

Original: The objective of ZSL is to learn the classiﬁer using D tr and to predict labels ˆ y j ∈ Y from the example x j ∈ X in the test set D ts = { x j , y j } N ts j =1 .
Revised: The objective of ZSL is to learn the classiﬁer using D tr and to predict labels ˆ y j ∈ Y from the example x j ∈ X in the test set D ts = { x j , y j } N ts j =1 .

Original: For the standard ZSL, it is assumed that the classes of the test set are completely unseen in the training set, which means that Y =
Revised: For the standard ZSL, it is assumed that the classes of the test set are completely unseen in the training set, which means that Y =

Original: Y u
Revised: Y u

Original: = {
Revised: = {

Original: S +1 , ..., S + U } .
Revised: S +1 , ..., S + U } .

Original: Our goal is to train in the setting of GZSL, which includes both seen and unseen classes in the test set (
Revised: Our goal is to train in the setting of GZSL, which includes both seen and unseen classes in the test set (

Original: Y = Y s ∪ Y u ).
Revised: Y = Y s ∪ Y u ).

Original: Furthermore, we assume that we have the class-attribute matrix A ∈ R M × ( S + U ) as the semantic information of classes, where each column represents the M -dimensional attribute vector
Revised: Furthermore, we assume that we have the class-attribute matrix A ∈ R M × ( S + U ) as the semantic information of classes, where each column represents the M -dimensional attribute vector

Original: a c ∈ A = R M of each class c = 1 , ..., S + U .
Revised: a c ∈ A = R M of each class c = 1 , ..., S + U .

Original: Using this attribute vector, the training set D tr can be replaced as { x i , a y i } N tr i =1 .
Revised: Using this attribute vector, the training set D tr can be replaced as { x i , a y i } N tr i =1 .

Original: Using the relation-based approach, the objective is changed to train the compatibility function of the input and attribute F ( x , a y ) , which represents how the input and attribute are related: stronger relations have greater values.
Revised: Using the relation-based approach, the objective is changed to train the compatibility function of the input and attribute F ( x , a y ) , which represents how the input and attribute are related: stronger relations have greater values.

Original: Once this function is learned from the training set, the classiﬁcation probability can be expressed as
Revised: Once this function is learned from the training set, the classiﬁcation probability can be expressed as

Original: [Equation]
Revised: [Equation]

Original: One can predict the class labels by choosing the one that maximizes this probability, i.e., ˆ y = arg max y ∈Y p ( y | x ) .
Revised: One can predict the class labels by choosing the one that maximizes this probability, i.e., ˆ y = arg max y ∈Y p ( y | x ) .

Original: Additionally, we call seen and unseen ones as different domains and express it as a binary variable d ∈ { 0 , 1 } , where d = 0 represents the seen domain and d = 1 represents the unseen one.
Revised: Additionally, we call seen and unseen ones as different domains and express it as a binary variable d ∈ { 0 , 1 } , where d = 0 represents the seen domain and d = 1 represents the unseen one.

Original:  P ROPOSED METHOD 3. R ELATION - BASED CLASSIFICATION OF THE SHARED REPRESENTATION
Revised:  P ROPOSED METHOD 3. R ELATION - BASED CLASSIFICATION OF THE SHARED REPRESENTATION

Original: In this study, the image x and the attribute a y are regarded as different modalities.
Revised: In this study, the image x and the attribute a y are regarded as different modalities.

Original: Also, the mapping q ϕ x ( z | x ) and q ϕ a ( z | a y ) embed them into the same space, i.e., the shared representation space z .
Revised: Also, the mapping q ϕ x ( z | x ) and q ϕ a ( z | a y ) embed them into the same space, i.e., the shared representation space z .

Original: In this approach, the compatibility function can be expressed with the Kullback–Leibler (KL) divergence as
Revised: In this approach, the compatibility function can be expressed with the Kullback–Leibler (KL) divergence as

Original: [Equation]
Revised: [Equation]

Original: where ϕ x,a is a shorthand notation for ϕ x and ϕ a .
Revised: where ϕ x,a is a shorthand notation for ϕ x and ϕ a .

Original: Therefore, learning the compatibility function corresponds to learning embeddings into the shared representation.
Revised: Therefore, learning the compatibility function corresponds to learning embeddings into the shared representation.

Original: Then, what kind of space embedding engenders good performance of ZSL and GZSL?
Revised: Then, what kind of space embedding engenders good performance of ZSL and GZSL?

Original: First, images and attributes belonging to the same class must be embedded in a nearby place in terms of KL divergence.
Revised: First, images and attributes belonging to the same class must be embedded in a nearby place in terms of KL divergence.

Original: It is clear that this embedding is a requirement for ZSL because, if not satisﬁed, the assumption on the relation-based methods is violated, rendering it impossible to predict corresponding classes from an input.
Revised: It is clear that this embedding is a requirement for ZSL because, if not satisﬁed, the assumption on the relation-based methods is violated, rendering it impossible to predict corresponding classes from an input.

Original: Note that this requirement must also be generalized to unknown data, i.e., examples of unseen classes.
Revised: Note that this requirement must also be generalized to unknown data, i.e., examples of unseen classes.

Original: That is, the shared representation must be modality invariant (see Figure 2(a)).
Revised: That is, the shared representation must be modality invariant (see Figure 2(a)).

Original: Second, examples embedded in the shared space by the mapping must be grouped by the same class and separated from other classes.
Revised: Second, examples embedded in the shared space by the mapping must be grouped by the same class and separated from other classes.

Original: If not satisﬁed (e.g., an area of one class overlaps with that of other classes), they may be misclassiﬁed as other classes.
Revised: If not satisﬁed (e.g., an area of one class overlaps with that of other classes), they may be misclassiﬁed as other classes.

Original: This can be rephrased that the embedded representation needs to be easily separated by some classiﬁer, i.e., the shared representation must have class separability (Figure 2(a)).
Revised: This can be rephrased that the embedded representation needs to be easily separated by some classiﬁer, i.e., the shared representation must have class separability (Figure 2(a)).

Original: Class separability between domains is particularly important for GZSL performance.
Revised: Class separability between domains is particularly important for GZSL performance.

Original: Because only seen data are given during GZSL training, there is no clue to properly embed the unseen classes.
Revised: Because only seen data are given during GZSL training, there is no clue to properly embed the unseen classes.

Original: Consequently, embedding of the unseen domain might overlap with those of the seen domain, and any input might be only to assigned to one of the more relevant seen classes (Figure 2(b)).
Revised: Consequently, embedding of the unseen domain might overlap with those of the seen domain, and any input might be only to assigned to one of the more relevant seen classes (Figure 2(b)).

Original: We hypothesize that the failure of class separability between domains is the underlying cause of the domain bias problem .
Revised: We hypothesize that the failure of class separability between domains is the underlying cause of the domain bias problem .

Original: 3. M ODALITY - INVARIANT AND C LASS - SEPARABLE M ULTIMODAL A UTO E NCODER
Revised: 3. M ODALITY - INVARIANT AND C LASS - SEPARABLE M ULTIMODAL A UTO E NCODER

Original: Given training data ( x , a y ) , the simplest way to satisfy modality invariant is to maximize
Revised: Given training data ( x , a y ) , the simplest way to satisfy modality invariant is to maximize

Original: Eq.
Revised: Eq.

Original: 2 over these data directly, but it is difﬁcult to generalize to test data.
Revised: 2 over these data directly, but it is difﬁcult to generalize to test data.

Original: For this study, we ﬁrst introduce a shared representation learning method based on variational autoencoders (VAE) (Kingma & Welling, 2013; Rezende et al., 2014).
Revised: For this study, we ﬁrst introduce a shared representation learning method based on variational autoencoders (VAE) (Kingma & Welling, 2013; Rezende et al., 2014).

Original: In VAE, the data x is assumed to be generated from a generative model p θ x ( x ) = ∫
Revised: In VAE, the data x is assumed to be generated from a generative model p θ x ( x ) = ∫

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( θ x is a learnable parameter), and learning is performed by maximizing the lower bound of the log marginal likelihood over the given data:
Revised: ( θ x is a learnable parameter), and learning is performed by maximizing the lower bound of the log marginal likelihood over the given data:

Original: [Equation]
Revised: [Equation]

Original: where q ϕ x ( z | x ) represents an approximate distribution of the true posterior p θ x ( z | x ) and ϕ x is a parameter.
Revised: where q ϕ x ( z | x ) represents an approximate distribution of the true posterior p θ x ( z | x ) and ϕ x is a parameter.

Original: This approximate distribution, also called the inference model, can be regarded as an embedding from x into z .
Revised: This approximate distribution, also called the inference model, can be regarded as an embedding from x into z .

Original: Although VAE learns to maximize Eq.
Revised: Although VAE learns to maximize Eq.

Original: 3 not under an exact generative distribution but under a ﬁnite training set, its representation is known to generalize well to unseen inputs.
Revised: 3 not under an exact generative distribution but under a ﬁnite training set, its representation is known to generalize well to unseen inputs.

Original: This capability is suitable for shared representation learning in ZSL, which needs to be generalized to the examples of the unseen classes.
Revised: This capability is suitable for shared representation learning in ZSL, which needs to be generalized to the examples of the unseen classes.

Original: To extend Eq.
Revised: To extend Eq.

Original: 3 to multimodal input of images and attributes, we ﬁrst replace the prior in the regularization term from p ( z ) to q ϕ a ( z | a y ) :
Revised: 3 to multimodal input of images and attributes, we ﬁrst replace the prior in the regularization term from p ( z ) to q ϕ a ( z | a y ) :

Original: [Equation]
Revised: [Equation]

Original: This makes regularization of the shared representation more relaxed because the replaced prior is learnable.
Revised: This makes regularization of the shared representation more relaxed because the replaced prior is learnable.

Original: In addition, maximizing this lower bound engenders learning to bring the two embeddings closer together explicitly.
Revised: In addition, maximizing this lower bound engenders learning to bring the two embeddings closer together explicitly.

Original: Next, we introduce a model p θ a ( a y | z ) that discriminates attributes from the representation.
Revised: Next, we introduce a model p θ a ( a y | z ) that discriminates attributes from the representation.

Original: By learning this model together with Eq. 4, one can add a constraint to the representation from which attributes can be successfully predicted:
Revised: By learning this model together with Eq. 4, one can add a constraint to the representation from which attributes can be successfully predicted:

Original: [Equation]
Revised: [Equation]

Original: Attributes are semantically distributed representations of classes, which engenders the classseparable representation.
Revised: Attributes are semantically distributed representations of classes, which engenders the classseparable representation.

Original: In addition, this equation can be interpreted as introducing an attribute generative model p θ ( a y | z ) .
Revised: In addition, this equation can be interpreted as introducing an attribute generative model p θ ( a y | z ) .

Original: From such a perspective, this equation considers not only the reconstruction of an image, but also the “cross” reconstruction of the corresponding attribute from the image.
Revised: From such a perspective, this equation considers not only the reconstruction of an image, but also the “cross” reconstruction of the corresponding attribute from the image.

Original: This model is actually the same as the multimodal model known as PSE (Jiao et al., 2019), as described in Sec 4.2.
Revised: This model is actually the same as the multimodal model known as PSE (Jiao et al., 2019), as described in Sec 4.2.

Original: Furthermore, as discussed in Sec. 3.1, the shared representation must be modality-invariant.
Revised: Furthermore, as discussed in Sec. 3.1, the shared representation must be modality-invariant.

Original: Therefore, we add an expected term at q ϕ a ( z | a y ) in Eq.
Revised: Therefore, we add an expected term at q ϕ a ( z | a y ) in Eq.

Original: 5, which is a constraint by which the embedded representation is the same as long as it represents the same thing, irrespective of which modality inference model is used.
Revised: 5, which is a constraint by which the embedded representation is the same as long as it represents the same thing, irrespective of which modality inference model is used.

Original: Therefore, our objective is ﬁnally
Revised: Therefore, our objective is ﬁnally

Original: [Equation]
Revised: [Equation]

Original: where log p θ x,a ( x , a y | z )
Revised: where log p θ x,a ( x , a y | z )

Original: = log p θ x
Revised: = log p θ x

Original: ( x | z ) + log p θ a ( a y | z ) .
Revised: ( x | z ) + log p θ a ( a y | z ) .

Original: All distributions are parameterized by deep neural networks.
Revised: All distributions are parameterized by deep neural networks.

Original: Moreover, we regard that the inference models as Gaussian and generative models as a Laplace distribution with a constant scale parameter.
Revised: Moreover, we regard that the inference models as Gaussian and generative models as a Laplace distribution with a constant scale parameter.

Original: Therefore, the log-likelihood of the generative models is obtained by the negative absolute-difference loss.
Revised: Therefore, the log-likelihood of the generative models is obtained by the negative absolute-difference loss.

Original: This model considers all the requirements that the shared representation should satisfy.
Revised: This model considers all the requirements that the shared representation should satisfy.

Original: For this study, we call this Modality-invariant and Class-separable Multimodal AutoEncoder (MCMAE) .
Revised: For this study, we call this Modality-invariant and Class-separable Multimodal AutoEncoder (MCMAE) .

Original: 3. D OMAIN DISCRIMINATOR
Revised: 3. D OMAIN DISCRIMINATOR

Original: MCMAE includes domain invariance and class separability.
Revised: MCMAE includes domain invariance and class separability.

Original: However, class separability between domains is not fully considered because no knowledge related to the unseen domain is given during training, meaning that the domain bias problem cannot be avoided.
Revised: However, class separability between domains is not fully considered because no knowledge related to the unseen domain is given during training, meaning that the domain bias problem cannot be avoided.

Original: Therefore, we propose addition of another approach to address this problem.
Revised: Therefore, we propose addition of another approach to address this problem.

Original: We introduce a model p β ( d | z ) that discriminates a domain from the shared representation.
Revised: We introduce a model p β ( d | z ) that discriminates a domain from the shared representation.

Original: If this discriminator can be trained together with MCMAE, then the representation is separable between domains, leading to alleviation of the domain bias problem.
Revised: If this discriminator can be trained together with MCMAE, then the representation is separable between domains, leading to alleviation of the domain bias problem.

Original: For training p β ( d | z ) , we create a dataset with all class-attributes as inputs and corresponding domain variables as labels D tr a = { ( a y c , d c ) } S
Revised: For training p β ( d | z ) , we create a dataset with all class-attributes as inputs and corresponding domain variables as labels D tr a = { ( a y c , d c ) } S

Original: + U c
Revised: + U c

Original: =1 .
Revised: =1 .

Original: Since the input of this discriminator is z , we should sample it from attributes a y using the inference model.
Revised: Since the input of this discriminator is z , we should sample it from attributes a y using the inference model.

Original: Here, we use a reconstructed representation not only via the inference model of attributes but also via the generative and inference model of images.
Revised: Here, we use a reconstructed representation not only via the inference model of attributes but also via the generative and inference model of images.

Original: Therefore, the objective of the domain discriminator given ( a y , d ) is
Revised: Therefore, the objective of the domain discriminator given ( a y , d ) is

Original: [Equation]
Revised: [Equation]

Original: We will explain the reason for using the reconstructed representation and the details of Eq.
Revised: We will explain the reason for using the reconstructed representation and the details of Eq.

Original: 7 in
Revised: 7 in

Original: Appendix B. When testing, the domain discriminator must be able to correctly discriminate the domain given an image.
Revised: Appendix B. When testing, the domain discriminator must be able to correctly discriminate the domain given an image.

Original: It is noteworthy that, for this to work properly, different modalities need to be embedded in the same place in the shared space by each inference model.
Revised: It is noteworthy that, for this to work properly, different modalities need to be embedded in the same place in the shared space by each inference model.

Original: By training the objective of the domain discriminator together with that of MCMAE, the embedded shared representation is prompted to be separated by domain.
Revised: By training the objective of the domain discriminator together with that of MCMAE, the embedded shared representation is prompted to be separated by domain.

Original: The resulting objective, given D tr and D tr a , becomes
Revised: The resulting objective, given D tr and D tr a , becomes

Original: [Equation]
Revised: [Equation]

Original: where α is a hyper-parameter representing the degree to which domain separation is enforced in the representation.
Revised: where α is a hyper-parameter representing the degree to which domain separation is enforced in the representation.

Original: We learn the model end-to-end by maximizing Eq.8 in terms of all parameters.
Revised: We learn the model end-to-end by maximizing Eq.8 in terms of all parameters.

Original: After training, this discriminator can predict the possibility of assigning a domain from any input z .
Revised: After training, this discriminator can predict the possibility of assigning a domain from any input z .

Original: In addition, if modality invariant embedding is possible, then one can predict the domain for unseen images and can thereby obtain “prior knowledge of the domain” from this classiﬁer.
Revised: In addition, if modality invariant embedding is possible, then one can predict the domain for unseen images and can thereby obtain “prior knowledge of the domain” from this classiﬁer.

Original: We propose combination of this with an attribute-based classiﬁer (Eq. 1) to compensate the class prediction as a mixture model:
Revised: We propose combination of this with an attribute-based classiﬁer (Eq. 1) to compensate the class prediction as a mixture model:

Original: [Equation]
Revised: [Equation]

Original: where p ( y | x , d = 0) =
Revised: where p ( y | x , d = 0) =

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We expect that the domain bias problem can be alleviated by selecting a class that maximizes this probability as the prediction result.
Revised: We expect that the domain bias problem can be alleviated by selecting a class that maximizes this probability as the prediction result.

Original: We call this approach as MCMAE with the Domain discriminator (MCMAE-D) .
Revised: We call this approach as MCMAE with the Domain discriminator (MCMAE-D) .

Original: Figure 2(c) shows the overall architecture of MCMAE-D.
Revised: Figure 2(c) shows the overall architecture of MCMAE-D.

Original:  R ELATED W ORK 4. Z ERO - SHOT LEARNING / G ENERALIZED ZERO - SHOT LEARNING
Revised:  R ELATED W ORK 4. Z ERO - SHOT LEARNING / G ENERALIZED ZERO - SHOT LEARNING

Original: The main approach for tackling ZSL is to learn a compatibility function that represents the relation between images and class-attributes from the seen data 1 .
Revised: The main approach for tackling ZSL is to learn a compatibility function that represents the relation between images and class-attributes from the seen data 1 .

Original: Some early studies used classiﬁers such as SVM to learn a mapping from images to class-attributes (Lampert et al., 2009; 2014), but many other methods use linear embedding into attributes (or semantic representations) and typically set a ranking loss as the objective function to train such embedding (Frome et al., 2013; Akata et al., 2015; Romera-Paredes & Torr, 2015; Kodirov et al., 2017).
Revised: Some early studies used classiﬁers such as SVM to learn a mapping from images to class-attributes (Lampert et al., 2009; 2014), but many other methods use linear embedding into attributes (or semantic representations) and typically set a ranking loss as the objective function to train such embedding (Frome et al., 2013; Akata et al., 2015; Romera-Paredes & Torr, 2015; Kodirov et al., 2017).

Original: Romera-Paredes & Torr (2015) add the regularization term of the mapping to this objective function, and Kodirov et al.
Revised: Romera-Paredes & Torr (2015) add the regularization term of the mapping to this objective function, and Kodirov et al.

Original: (2017) minimizes the embedding error in both directions with the objective.
Revised: (2017) minimizes the embedding error in both directions with the objective.

Original: The nonlinear model is also used (Socher et al., 2013) to take non-linearity between images and attributes into consideration.
Revised: The nonlinear model is also used (Socher et al., 2013) to take non-linearity between images and attributes into consideration.

Original: However, with this mapping direction, a hubness problem occurs in which all images are assigned to one type of attributes.
Revised: However, with this mapping direction, a hubness problem occurs in which all images are assigned to one type of attributes.

Original: To avoid this, some methods measure the similarity in the image space by obtaining the reverse mapping of attributes to images (Zhang et al., 2017; Verma & Rai, 2017).
Revised: To avoid this, some methods measure the similarity in the image space by obtaining the reverse mapping of attributes to images (Zhang et al., 2017; Verma & Rai, 2017).

Original: Another approach is to map images and attributes to another shared representation space and measure the similarity in that space (Zhang & Saligrama, 2015; 2016), which is adopted in our paper.
Revised: Another approach is to map images and attributes to another shared representation space and measure the similarity in that space (Zhang & Saligrama, 2015; 2016), which is adopted in our paper.

Original: The advantage of this is that it can obtain a representation that does not depend on the dimension or representation of the input.
Revised: The advantage of this is that it can obtain a representation that does not depend on the dimension or representation of the input.

Original: Moreover, arbitrary representation can be learned by adding constraints during training.
Revised: Moreover, arbitrary representation can be learned by adding constraints during training.

Original: Recently, several methods use VAEs to learn this, which we will discuss in Sec. 4.2.
Revised: Recently, several methods use VAEs to learn this, which we will discuss in Sec. 4.2.

Original: These relation-based methods have the advantage of being able to perform classiﬁcation just by learning the compatibility function, but they do not work well in GZSL due to the domain bias problem (Chao et al., 2016).
Revised: These relation-based methods have the advantage of being able to perform classiﬁcation just by learning the compatibility function, but they do not work well in GZSL due to the domain bias problem (Chao et al., 2016).

Original: One way to alleviate this problem is to calibrate the compatibility function (Chao et al., 2016; Liu et al., 2018).
Revised: One way to alleviate this problem is to calibrate the compatibility function (Chao et al., 2016; Liu et al., 2018).

Original: In recent years, synthesis-based methods 2 have become mainstream because of their signiﬁcant improvement in GZSL performance (Mishra et al., 2017;
Revised: In recent years, synthesis-based methods 2 have become mainstream because of their signiﬁcant improvement in GZSL performance (Mishra et al., 2017;

Original: Verma et al., 2018; Xian et al., 2018b; Felix et al., 2018).
Revised: Verma et al., 2018; Xian et al., 2018b; Felix et al., 2018).

Original: In particular, Zhang & Koniusz (2018a) propose a model selection mechanism that improves synthesis-based performance by distinguishing seen and unseen domains from the generated images.
Revised: In particular, Zhang & Koniusz (2018a) propose a model selection mechanism that improves synthesis-based performance by distinguishing seen and unseen domains from the generated images.

Original: However, these synthesis-based methods require costly image generation and additional classiﬁer learning.
Revised: However, these synthesis-based methods require costly image generation and additional classiﬁer learning.

Original: In this study, we aim to achieve high performance with the end-to-end relation-based method.
Revised: In this study, we aim to achieve high performance with the end-to-end relation-based method.

Original: 4. M ULTIMODAL DEEP GENERATIVE MODELS FOR SHARED REPRESENTATION LEARNING
Revised: 4. M ULTIMODAL DEEP GENERATIVE MODELS FOR SHARED REPRESENTATION LEARNING

Original: It has been discussed that the importance of learning appropriate representation in ZSL.
Revised: It has been discussed that the importance of learning appropriate representation in ZSL.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2017) propose to learn an attribute representation that is discriminative between classes while maintaining their semantic information.
Revised: (2017) propose to learn an attribute representation that is discriminative between classes while maintaining their semantic information.

Original: In this study, we discuss how to learn a good shared representation that integrates different modalities of images and attributes in ZSL.
Revised: In this study, we discuss how to learn a good shared representation that integrates different modalities of images and attributes in ZSL.

Original: In recent years, several methods have been proposed to learn the shared representation with
Revised: In recent years, several methods have been proposed to learn the shared representation with

Original: VAEs.
Revised: VAEs.

Original: JMVAE (Suzuki et al., 2016) and TrELBO (Vedantam et al., 2017) are aimed at obtaining a modality-invariant representation by learning the inference model of each modality close to that of taking two modality inputs.
Revised: JMVAE (Suzuki et al., 2016) and TrELBO (Vedantam et al., 2017) are aimed at obtaining a modality-invariant representation by learning the inference model of each modality close to that of taking two modality inputs.

Original: However, because an additional inference model is required, it is expensive in terms of the number of parameters.
Revised: However, because an additional inference model is required, it is expensive in terms of the number of parameters.

Original: In addition, regularization that forces the representation to be close to the standard Gaussian prior makes it difﬁcult to obtain a class-separable representation (Jiao et al., 2019).
Revised: In addition, regularization that forces the representation to be close to the standard Gaussian prior makes it difﬁcult to obtain a class-separable representation (Jiao et al., 2019).

Original: PSE (Jiao et al., 2019) and CADA-VAE (Schonfeld et al., 2019) are very similar to MCMAE in terms of shared representation learning.
Revised: PSE (Jiao et al., 2019) and CADA-VAE (Schonfeld et al., 2019) are very similar to MCMAE in terms of shared representation learning.

Original: PSE is not for ZSL but is equivalent to Eq.
Revised: PSE is not for ZSL but is equivalent to Eq.

Original: 5 by considering label information as attributes a 3 .
Revised: 5 by considering label information as attributes a 3 .

Original: Their mutual difference is therefore whether they contain the reconstruction term given attributes, imposing a modality invariant representation.
Revised: Their mutual difference is therefore whether they contain the reconstruction term given attributes, imposing a modality invariant representation.

Original: CADA-VAE, a
Revised: CADA-VAE, a

Original: 2018a) categorizes these methods into four groups, we call all of them relation-based approaches, because all methods learn the relation between images and classes (or attributes) and use it directly to evaluate test data.
Revised: 2018a) categorizes these methods into four groups, we call all of them relation-based approaches, because all methods learn the relation between images and classes (or attributes) and use it directly to evaluate test data.

Original: synthesis-based model on the shared representation, has an objective that consists of three parts 4 :
Revised: synthesis-based model on the shared representation, has an objective that consists of three parts 4 :

Original: VAE losses for each modality, cross-alignment losses that reconstruct one modality from the other, and distribution-alignment loss that brings the inference models of each modality closer.
Revised: VAE losses for each modality, cross-alignment losses that reconstruct one modality from the other, and distribution-alignment loss that brings the inference models of each modality closer.

Original: The main difference between MCMAE is that the inference model is forced to stay close to the prior.
Revised: The main difference between MCMAE is that the inference model is forced to stay close to the prior.

Original: Similarly to JMVAE and TrELBO, this might be too restrictive to obtain a good shared representation.
Revised: Similarly to JMVAE and TrELBO, this might be too restrictive to obtain a good shared representation.

Original:  E XPERIMENTS 5. D ATASET AND S ETTING
Revised:  E XPERIMENTS 5. D ATASET AND S ETTING

Original: For experimentation, we use the following four datasets, which are commonly used for ZSL:
Revised: For experimentation, we use the following four datasets, which are commonly used for ZSL:

Original: Animals with Attributes (AWA) (Lampert et al., 2014), CUB-200-2011 Bird (CUB)
Revised: Animals with Attributes (AWA) (Lampert et al., 2014), CUB-200-2011 Bird (CUB)

Original: (Wah et al., 2011), SUN Attribute (SUN) (Patterson & Hays, 2012), and Attribute Pascal and Yahoo (aPY) (Farhadi et al., 2009).
Revised: (Wah et al., 2011), SUN Attribute (SUN) (Patterson & Hays, 2012), and Attribute Pascal and Yahoo (aPY) (Farhadi et al., 2009).

Original: Each dataset includes images of each class, where each class is represented by semantic attributes.
Revised: Each dataset includes images of each class, where each class is represented by semantic attributes.

Original: For fair comparison, we use a 2048-dim top-layer embedding of the 101-layered ResNet (He et al., 2016) provided by Xian et al.
Revised: For fair comparison, we use a 2048-dim top-layer embedding of the 101-layered ResNet (He et al., 2016) provided by Xian et al.

Original: (2018a)
Revised: (2018a)

Original: as the image vector.
Revised: as the image vector.

Original: Furthermore, for the class-attribute representation, we use the attributes valued continuously betweenand 1 provided with each dataset.
Revised: Furthermore, for the class-attribute representation, we use the attributes valued continuously betweenand 1 provided with each dataset.

Original: We followed the split proposed in Xian et al.
Revised: We followed the split proposed in Xian et al.

Original: (2018a) for splitting each dataset into train, validation, and test.
Revised: (2018a) for splitting each dataset into train, validation, and test.

Original: The hyper-parameter selection of our model was based on this train–validation split.
Revised: The hyper-parameter selection of our model was based on this train–validation split.

Original: In training of GZSL, we used both as training data.
Revised: In training of GZSL, we used both as training data.

Original: As the metric of evaluation, we use the average per-class top-1 accuracy on both the seen and unseen classes (referred as acc s and acc u ).
Revised: As the metric of evaluation, we use the average per-class top-1 accuracy on both the seen and unseen classes (referred as acc s and acc u ).

Original: In addition, to evaluate the performance on GZSL, we calculate the harmonic mean of acc s and acc u , which is acc H
Revised: In addition, to evaluate the performance on GZSL, we calculate the harmonic mean of acc s and acc u , which is acc H

Original: =
Revised: =

Original: (2 · acc s · acc u ) /
Revised: (2 · acc s · acc u ) /

Original: ( acc s + acc u ) .
Revised: ( acc s + acc u ) .

Original: The architecture of each deep probabilistic model is listed in Appendix A. We set the dimension of the latent variable to 512 and α = 0 .
Revised: The architecture of each deep probabilistic model is listed in Appendix A. We set the dimension of the latent variable to 512 and α = 0 .

Original: We used the Adam optimization algorithm (Kingma & Ba, 2014) with a learning rate of 10 − 3 .
Revised: We used the Adam optimization algorithm (Kingma & Ba, 2014) with a learning rate of 10 − 3 .

Original: In all experiments, we trained for 100 epochs 5 .
Revised: In all experiments, we trained for 100 epochs 5 .

Original: All models in this paper were implemented using PyTorch (Paszke et al., 2017).
Revised: All models in this paper were implemented using PyTorch (Paszke et al., 2017).

Original: 5. A NALYSIS OF OUR PROPOSED METHOD
Revised: 5. A NALYSIS OF OUR PROPOSED METHOD

Original: This section presents analyses of the proposed method from various perspectives.
Revised: This section presents analyses of the proposed method from various perspectives.

Original: We use AWA for this analysis because the number of data per class is greater than that for other datasets.
Revised: We use AWA for this analysis because the number of data per class is greater than that for other datasets.

Original: First, we conﬁrm the effectiveness of the proposed method by comparing it with a similar shared representation learning model: PSE and CADA-VAE.
Revised: First, we conﬁrm the effectiveness of the proposed method by comparing it with a similar shared representation learning model: PSE and CADA-VAE.

Original: To align the conditions, we used the same network structure for the inference and generative models of each modality throughout all methods.
Revised: To align the conditions, we used the same network structure for the inference and generative models of each modality throughout all methods.

Original: The original CADA-VAE paper takes a synthesis-based approach in which samples are generated from the shared space and use for learning a classiﬁer.
Revised: The original CADA-VAE paper takes a synthesis-based approach in which samples are generated from the shared space and use for learning a classiﬁer.

Original: Here, to compare the effectiveness of shared representation learning, we evaluate CADA-VAE with a relation-based approach similar to MCMAE.
Revised: Here, to compare the effectiveness of shared representation learning, we evaluate CADA-VAE with a relation-based approach similar to MCMAE.

Original: In all methods, the classiﬁcation probability was obtained by Eq. 1.
Revised: In all methods, the classiﬁcation probability was obtained by Eq. 1.

Original: Table 1 presents results in GZSL.
Revised: Table 1 presents results in GZSL.

Original: First, compared with PSE and MCMAE, acc s is higher in the
Revised: First, compared with PSE and MCMAE, acc s is higher in the

Original: MCMAE, but acc u is higher in the PSE.
Revised: MCMAE, but acc u is higher in the PSE.

Original: One might infer that this result means that MCMAE obtains a shared representation that is not generalized.
Revised: One might infer that this result means that MCMAE obtains a shared representation that is not generalized.

Original: However, when learning these models and domain classiﬁers simultaneously and when classifying them with the mixture prediction model (shown as PSE-D and MCMAE-D), MCMAE-D has markedly higher performance.
Revised: However, when learning these models and domain classiﬁers simultaneously and when classifying them with the mixture prediction model (shown as PSE-D and MCMAE-D), MCMAE-D has markedly higher performance.

Original: Furthermore, the performance of the domain discriminator by AUROC is the highest in MCMAE-D. As described in Sec.
Revised: Furthermore, the performance of the domain discriminator by AUROC is the highest in MCMAE-D. As described in Sec.

Original: 3.3, this domain discriminator approach does not work properly unless the modality invariance is generalized.
Revised: 3.3, this domain discriminator approach does not work properly unless the modality invariance is generalized.

Original: This suggests that MCMAE can obtain a modality-invariant representation.
Revised: This suggests that MCMAE can obtain a modality-invariant representation.

Original: On the other hand, because PSE does not consider modality invariance, the domain discriminator does not generalize well to the image modality.
Revised: On the other hand, because PSE does not consider modality invariance, the domain discriminator does not generalize well to the image modality.

Original: In addition, the result that acc u is greatly improved by
Revised: In addition, the result that acc u is greatly improved by

Original: MCMAE-D implies that MCMAE can acquire the class separability representation but its performance is degraded due to the domain bias problem.
Revised: MCMAE-D implies that MCMAE can acquire the class separability representation but its performance is degraded due to the domain bias problem.

Original: [Table]
Revised: [Table]

Original: To conﬁrm this consideration qualitatively we visualize the shared representation of PSE, MCMAE, and MCMAE-D in two dimensions.
Revised: To conﬁrm this consideration qualitatively we visualize the shared representation of PSE, MCMAE, and MCMAE-D in two dimensions.

Original: Figure 3 portrays the visualization results.
Revised: Figure 3 portrays the visualization results.

Original: In PSE, the test images of the seen classes are well embedded around the corresponding attribute.
Revised: In PSE, the test images of the seen classes are well embedded around the corresponding attribute.

Original: However, for the unseen class, the data of some classes such as “collie”, are embedded in a location that differs from the corresponding attribute which indicates that the modality invariance is generalized insufﬁciently.
Revised: However, for the unseen class, the data of some classes such as “collie”, are embedded in a location that differs from the corresponding attribute which indicates that the modality invariance is generalized insufﬁciently.

Original: By contrast, in MCMAE, we can conﬁrm that the unseen images are embedded almost appropriately, i.e., we can obtain a generalized modality-invariant representation.
Revised: By contrast, in MCMAE, we can conﬁrm that the unseen images are embedded almost appropriately, i.e., we can obtain a generalized modality-invariant representation.

Original: However, it is apparent that some class attributes overlap, which can cause the domain bias problem.
Revised: However, it is apparent that some class attributes overlap, which can cause the domain bias problem.

Original: Then, moving to the MCMAED representation, we conﬁrm that embedding of seen and unseen is separated slightly because of learn embedding with the domain discriminator jointly.
Revised: Then, moving to the MCMAED representation, we conﬁrm that embedding of seen and unseen is separated slightly because of learn embedding with the domain discriminator jointly.

Original: Furthermore, the domain classiﬁer has prior knowledge to predict which domain is the location before the test images are given.
Revised: Furthermore, the domain classiﬁer has prior knowledge to predict which domain is the location before the test images are given.

Original: Then we conﬁrm that the test images are actually embedded almost in accordance with it.
Revised: Then we conﬁrm that the test images are actually embedded almost in accordance with it.

Original: This demonstrates that the domain discriminator has the ability to alleviate the domain bias problem.
Revised: This demonstrates that the domain discriminator has the ability to alleviate the domain bias problem.

Original: Let us go back to Table 1 and see the CADA-VAE results.
Revised: Let us go back to Table 1 and see the CADA-VAE results.

Original: Schonfeld et al.
Revised: Schonfeld et al.

Original: (2019) reports that this model performs well with a synthesis-based approach in latent space.
Revised: (2019) reports that this model performs well with a synthesis-based approach in latent space.

Original: However, this result demonstrates that it is worse than PSE and MCMAE in the relation-based case.
Revised: However, this result demonstrates that it is worse than PSE and MCMAE in the relation-based case.

Original: This is probably true because the representation is over-constrained by the regularization term of the inference model included in CADA-VAE.
Revised: This is probably true because the representation is over-constrained by the regularization term of the inference model included in CADA-VAE.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: Next

Original: [Table]
Revised: [Table]

Original: Next, we analyze the parameter sensitivity of our proposed models: The dimension of the shared representation (in both MCMAE and MCMAE-D) and the coefﬁcient of domain discriminator α (in
Revised: we analyze the parameter sensitivity of our proposed models: The dimension of the shared representation (in both MCMAE and MCMAE-D) and the coefﬁcient of domain discriminator α (in

Original: MCMAE-D).
Revised: MCMAE-D).

Original: Figure 4 presents the results.
Revised: Figure 4 presents the results.

Original: First, from Figure 4(a), we see that MCMAE performance decreases slightly as the dimension of latent variables increases.
Revised: First, from Figure 4(a) [Table] we see that MCMAE performance decreases slightly as the dimension of latent variables increases.

Original: This seems to be because in a large dimension, embedding of the unseen domain becomes more difﬁcult.
Revised: This seems to be because in a large dimension, embedding of the unseen domain becomes more difﬁcult.

Original: On the other hand, we ﬁnd that MCMAE-D is robust to the dimension of latent variables.
Revised: On the other hand, we ﬁnd that MCMAE-D is robust to the dimension of latent variables.

Original: This result shows that the domain classiﬁcation probability contributes to the compensation of the performance signiﬁcantly.
Revised: This result shows that the domain classiﬁcation probability contributes to the compensation of the performance signiﬁcantly.

Original: Next, from Figure 4(b), we ﬁnd that increasing the value of the coefﬁcient α decreases the GZSL performance because over-enforced separation might cause imperfect embedding from the input.
Revised: Next, from Figure 4(b), we ﬁnd that increasing the value of the coefﬁcient α decreases the GZSL performance because over-enforced separation might cause imperfect embedding from the input.

Original: 5. C OMPARISON WITH GZSL STATE - OF - THE - ART MODELS
Revised: 5. C OMPARISON WITH GZSL STATE - OF - THE - ART MODELS

Original: Table 2 presents the respective performance results obtained for the proposed method and GZSL state-of-the-art methods.
Revised: Table 2 presents the respective performance results obtained for the proposed method and GZSL state-of-the-art methods.

Original: First, MCMAE can be seen to have the same performance as existing relation-based models.
Revised: First, MCMAE can be seen to have the same performance as existing relation-based models.

Original: These models have low accuracy for the unseen domain, indicating that the domain bias problem occurs.
Revised: These models have low accuracy for the unseen domain, indicating that the domain bias problem occurs.

Original: Next, in synthetic-based models, the accuracy of the unseen domain is almost identical to that of the seen, indicating that the domain bias problem is mitigated.
Revised: Next, in synthetic-based models, the accuracy of the unseen domain is almost identical to that of the seen, indicating that the domain bias problem is mitigated.

Original: However, as described above, this approach must generate many images for even unseen classes after training their generative model and must prepare and learn an additional classiﬁer.
Revised: However, as described above, this approach must generate many images for even unseen classes after training their generative model and must prepare and learn an additional classiﬁer.

Original: Finally, the results obtained with MCMAE-D, which learns domain discriminator jointly, clarify that the domain bias problem has been relaxed greatly and MCMAE-D achieves the best performance among the relation-based models.
Revised: Finally, the results obtained with MCMAE-D, which learns domain discriminator jointly, clarify that the domain bias problem has been relaxed greatly and MCMAE-D achieves the best performance among the relation-based models.

Original: Additionally, it is apparent that it performs as well as state-ofthe-art synthesis-based models.
Revised: Additionally, it is apparent that it performs as well as state-ofthe-art synthesis-based models.

Original: These results revealed that the domain bias problem in GZSL can be alleviated and that high performance can be achieved using the domain discriminator on the shared representation, even in relation-based approach.
Revised: These results revealed that the domain bias problem in GZSL can be alleviated and that high performance can be achieved using the domain discriminator on the shared representation, even in relation-based approach.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This study addressed the domain bias problem in GZSL.
Revised: This study addressed the domain bias problem in GZSL.

Original: First, we considered domain invariance and class separability as requirements necessary for high performance in ZSL, and introduced MCMAE that learns embedding in a shared representation that satisfy these requirements.
Revised: First, we considered domain invariance and class separability as requirements necessary for high performance in ZSL, and introduced MCMAE that learns embedding in a shared representation that satisfy these requirements.

Original: Next, we assumed that the domain bias problem occurs when the unseen domain overlaps with the seen domain and, to address this, proposed MCMAE-D to learn the discriminator that distinguishes two domains from its representation.
Revised: Next, we assumed that the domain bias problem occurs when the unseen domain overlaps with the seen domain and, to address this, proposed MCMAE-D to learn the discriminator that distinguishes two domains from its representation.

Original: This discriminator not only encourages domain embedding to be separated in the representation space during training, but it also gives prior knowledge of which domain a point in the representation belongs to after training.
Revised: This discriminator not only encourages domain embedding to be separated in the representation space during training, but it also gives prior knowledge of which domain a point in the representation belongs to after training.

Original: Therefore, we combined this discriminator and relation-based classiﬁcation on MCMAE shared representation as a mixture model to ascertain the classiﬁcation probability.
Revised: Therefore, we combined this discriminator and relation-based classiﬁcation on MCMAE shared representation as a mixture model to ascertain the classiﬁcation probability.

Original: Through experimentation, we conﬁrmed that this approach mitigates the domain bias problem considerably.
Revised: Through experimentation, we conﬁrmed that this approach mitigates the domain bias problem considerably.

Original: Future studies will assess application of our model to larger amounts of data while taking advantage of the relation-based method.
Revised: Future studies will assess application of our model to larger amounts of data while taking advantage of the relation-based method.

Original: A N ETWORK ARCHITECTURES
Revised: A N ETWORK ARCHITECTURES

Original: The Gaussian distribution is parameterized as
Revised: The Gaussian distribution is parameterized as

Original: [Equation]
Revised: [Equation]

Original: where f x represents a neural network and Softplus is a softplus function.
Revised: where f x represents a neural network and Softplus is a softplus function.

Original: Moreover, the Laplace distribution with a constant scale c is parameterized as follows.
Revised: Moreover, the Laplace distribution with a constant scale c is parameterized as follows.

Original: [Equation]
Revised: [Equation]

Original: In this paper, it is assumed that only the mean parameter µ is output deterministically when sampling from the Laplace distribution with a constant scale:
Revised: In this paper, it is assumed that only the mean parameter µ is output deterministically when sampling from the Laplace distribution with a constant scale:

Original: [Equation]
Revised: [Equation]

Original: For the notation of model structures, we denote a linear fully-connected layer with k units as k , batch normalization, and ReLU as DkBR .
Revised: For the notation of model structures, we denote a linear fully-connected layer with k units as k , batch normalization, and ReLU as DkBR .

Original: Also, we denote DkBR without batch normalization and
Revised: Also, we denote DkBR without batch normalization and

Original: ReLU as Dk .
Revised: ReLU as Dk .

Original: In addition, the process of applying J after I is denoted as I-J , and the process of concatenating the last layers of the two networks I, J into one layer is denoted as (I,J) .
Revised: In addition, the process of applying J after I is denoted as I-J , and the process of concatenating the last layers of the two networks I, J into one layer is denoted as (I,J) .

Original: The network structures of distributions of MCMAE are as follows ( DdimA is the dimension of attributes and DdimZ is that of latent variable):
Revised: The network structures of distributions of MCMAE are as follows ( DdimA is the dimension of attributes and DdimZ is that of latent variable):

Original: • p θ x ( x | z ) (Laplace) – f µ : D – f MLP : z-D1024BR-D1024BR • p θ a ( a | z ) (Laplace) – f µ : DdimA – f MLP : z-D1024BR-D1024BR • q ϕ x
Revised: • p θ x ( x | z ) (Laplace) – f µ : D – f MLP : z-D1024BR-D1024BR • p θ a ( a | z ) (Laplace) – f µ : DdimA – f MLP : z-D1024BR-D1024BR • q ϕ x

Original: ( z | x ) (Gaussian) – f µ and f σ 2 : DdimZ – f MLP : x-D1024BR-D1024BR • q ϕ a ( z | a ) (Gaussian) – f µ and f σ 2 : DdimZ – f MLP : a-D1024BR-D1024BR
Revised: ( z | x ) (Gaussian) – f µ and f σ 2 : DdimZ – f MLP : x-D1024BR-D1024BR • q ϕ a ( z | a ) (Gaussian) – f µ and f σ 2 : DdimZ – f MLP : a-D1024BR-D1024BR

Original: Moreover, the structure of the domain discriminator p λ ( d | z ) =
Revised: Moreover, the structure of the domain discriminator p λ ( d | z ) =

Original: B ( d ; µ = Sigmoid( f µ ( f MLP ( z )))
Revised: B ( d ; µ = Sigmoid( f µ ( f MLP ( z )))

Original: (where B means Bernoulli distribution and Sigmoid is a sigmoid function) is as follows.
Revised: (where B means Bernoulli distribution and Sigmoid is a sigmoid function) is as follows.

Original: • p λ
Revised: • p λ

Original: d | z ) (Bernoulli) – f µ : D – f MLP : z-D1024BR
Revised: d | z ) (Bernoulli) – f µ : D – f MLP : z-D1024BR

Original: B L EARNING THE DOMAIN DISCRIMINATOR
Revised: B L EARNING THE DOMAIN DISCRIMINATOR

Original: A straightforward way to learn the domain discriminator is to use an inference model q ϕ a ( z | a y ) to maximize the following objective function:
Revised: A straightforward way to learn the domain discriminator is to use an inference model q ϕ a ( z | a y ) to maximize the following objective function:

Original: [Equation]
Revised: [Equation]

Original: By learning both the inference model and the discriminator together, we expect to obtain a representation that can be separated by the domain.
Revised: By learning both the inference model and the discriminator together, we expect to obtain a representation that can be separated by the domain.

Original: However, only q ϕ a ( z | a y ) will be affected strongly by the learning of the discriminator and might not be stable during training.
Revised: However, only q ϕ a ( z | a y ) will be affected strongly by the learning of the discriminator and might not be stable during training.

Original: In addition, the representation that we want to discriminate in the test time is embedded from the images, not from the attributes.
Revised: In addition, the representation that we want to discriminate in the test time is embedded from the images, not from the attributes.

Original: Therefore, we take a method of generating image data in pseudo and learning the discriminator
Revised: Therefore, we take a method of generating image data in pseudo and learning the discriminator

Original: [Figure]
Revised: [Figure]

Original: on the representation inferred from them, which is why we use the reconstructed representation in discriminator learning.
Revised: on the representation inferred from them, which is why we use the reconstructed representation in discriminator learning.

Original: Because attributes have less variation than images, the variance of representation sampled from q ϕ a ( z | a y ) becomes small, and as a result, that of the reconstructed representation can also be small, which may lead to overﬁtting of the domain discriminator on the representation during training.
Revised: Because attributes have less variation than images, the variance of representation sampled from q ϕ a ( z | a y ) becomes small, and as a result, that of the reconstructed representation can also be small, which may lead to overﬁtting of the domain discriminator on the representation during training.

Original: To alleviate this, we consider increasing the variance of the inference model when calculating Eq.7, that is, we consider the inference model as q ϕ a ( z | a y ) =
Revised: To alleviate this, we consider increasing the variance of the inference model when calculating Eq.7, that is, we consider the inference model as q ϕ a ( z | a y ) =

Original: N ( z ; µ , a 2 σ 2 ) , where a > 1 .
Revised: N ( z ; µ , a 2 σ 2 ) , where a > 1 .

Original: In this study, we set a = 2 .
Revised: In this study, we set a = 2 .

Original: Note that this change is only for Eq.7 and does not apply to other calculations such as the objective function of MCMAE (Eq.6).
Revised: Note that this change is only for Eq.7 and does not apply to other calculations such as the objective function of MCMAE (Eq.6).

Original: C CADA-VAE
Revised: C CADA-VAE

Original: The objective of CADA-VAE (Schonfeld et al., 2019) is as follows:
Revised: The objective of CADA-VAE (Schonfeld et al., 2019) is as follows:

Original: [Equation]
Revised: [Equation]

Original: (11) where W 2 ( p, q ) is the 2-Wasserstein distance between p and q and where δ and γ are weighting factors.
Revised: (11) where W 2 ( p, q ) is the 2-Wasserstein distance between p and q and where δ and γ are weighting factors.

Original: Moreover, L V AE a is the objective of VAE that takes attributes as inputs and uses inference and generative models for attributes.
Revised: Moreover, L V AE a is the objective of VAE that takes attributes as inputs and uses inference and generative models for attributes.

Original: In our experiment on Table 1, all coefﬁcients in Eq.
Revised: In our experiment on Table 1, all coefﬁcients in Eq.

Original: 11 are set to 1 for a fair comparison with
Revised: 11 are set to 1 for a fair comparison with

Original: MCMAE.
Revised: MCMAE.

Original: Moreover, in the relation-based setup, we used the Wasserstein distance instead of KL divergence to calculate the compatibility function (Eq. 2).
Revised: Moreover, in the relation-based setup, we used the Wasserstein distance instead of KL divergence to calculate the compatibility function (Eq. 2).

Original: D L EARNING PROGRESS OF MCMAE-D
Revised: D L EARNING PROGRESS OF MCMAE-D

Original: One advantage of relation-based methods including MCMAE-D is that GZSL performance can be veriﬁed during model learning.
Revised: One advantage of relation-based methods including MCMAE-D is that GZSL performance can be veriﬁed during model learning.

Original: On the other hand, the synthesis-based method cannot be conﬁrmed it unless learning of the generative model is completed and the synthesis data are generated.
Revised: On the other hand, the synthesis-based method cannot be conﬁrmed it unless learning of the generative model is completed and the synthesis data are generated.

Original: Figureshows learning curves of MCMAE-D for each dataset.
Revised: Figureshows learning curves of MCMAE-D for each dataset.

Original: From this result, it can be seen that the learning progress of this model is almost stable, although the performance of aPY drops beforeepochs.
Revised: From this result, it can be seen that the learning progress of this model is almost stable, although the performance of aPY drops beforeepochs.