Original: Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classiﬁcation
Revised: Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classiﬁcation

Original: Abstract
Revised: Abstract

Original: Multi-label classiﬁcation (MLC) is a prediction task where each sample can have more than one label.
Revised: Multi-label classiﬁcation (MLC) is a prediction task where each sample can have more than one label.

Original: We propose a novel contrastive learning boosted multilabel prediction model based on a Gaussian mixture variational autoencoder (CGMVAE), which learns a multimodal prior space and employs a contrastive loss.
Revised: We propose a novel contrastive learning boosted multilabel prediction model based on a Gaussian mixture variational autoencoder (CGMVAE), which learns a multimodal prior space and employs a contrastive loss.

Original: Many existing methods introduce extra complex neural modules to capture the label correlations, in addition to the prediction modules.
Revised: Many existing methods introduce extra complex neural modules to capture the label correlations, in addition to the prediction modules.

Original: We ﬁnd that by using contrastive learning in the supervised setting, we can exploit label information effectively, and learn meaningful feature and label embeddings capturing both the label correlations and predictive power, without extra neural modules.
Revised: We ﬁnd that by using contrastive learning in the supervised setting, we can exploit label information effectively, and learn meaningful feature and label embeddings capturing both the label correlations and predictive power, without extra neural modules.

Original: Our method also adopts the idea of learning and aligning latent spaces for both features and labels.
Revised: Our method also adopts the idea of learning and aligning latent spaces for both features and labels.

Original: More speciﬁcally, C-GMVAE imposes a Gaussian mixture structure on the latent space, to alleviate posterior collapse and over-regularization issues, in contrast to previous works based on a unimodal prior.
Revised: More speciﬁcally, C-GMVAE imposes a Gaussian mixture structure on the latent space, to alleviate posterior collapse and over-regularization issues, in contrast to previous works based on a unimodal prior.

Original: C-GMVAE outperforms existing methods on multiple public datasets and can often match other models’ full performance with only 50% of the training data.
Revised: C-GMVAE outperforms existing methods on multiple public datasets and can often match other models’ full performance with only 50% of the training data.

Original: Furthermore, we show that the learnt embeddings provide insights into the interpretation of label-label interactions.
Revised: Furthermore, we show that the learnt embeddings provide insights into the interpretation of label-label interactions.

Original:  Introduction
Revised:  Introduction

Original: In many machine learning tasks, an instance can have several labels.
Revised: In many machine learning tasks, an instance can have several labels.

Original: The task of predicting multiple labels is known as multi-label classiﬁcation (MLC).
Revised: The task of predicting multiple labels is known as multi-label classiﬁcation (MLC).

Original: MLC is common in domains like vision [1], natural language [2], biology [3].
Revised: MLC is common in domains like vision [1], natural language [2], biology [3].

Original: Unlike the single-label scenario, label correlations are more important in MLC.
Revised: Unlike the single-label scenario, label correlations are more important in MLC.

Original: Early works capture the correlations through classiﬁer chains [4], bayesian inference [5] and dimensionality reduction [6].
Revised: Early works capture the correlations through classiﬁer chains [4], bayesian inference [5] and dimensionality reduction [6].

Original: Boosted by the huge capacity of neural networks (NN), many previous methods can be improved by their neural extensions.
Revised: Boosted by the huge capacity of neural networks (NN), many previous methods can be improved by their neural extensions.

Original: For example, classiﬁer chains can be naturally enhanced by RNN [1].
Revised: For example, classiﬁer chains can be naturally enhanced by RNN [1].

Original: The non-linearity of NN alleviates the design complexity of feature mapping and many deep models can therefore focus on the loss function, feature-label and label-label correlation modeling.
Revised: The non-linearity of NN alleviates the design complexity of feature mapping and many deep models can therefore focus on the loss function, feature-label and label-label correlation modeling.

Original: One trending direction is to learn a deep latent space shared by features and labels.
Revised: One trending direction is to learn a deep latent space shared by features and labels.

Original: The samples from the latent space are then decoded to targets.
Revised: The samples from the latent space are then decoded to targets.

Original: One typical example is C2AE [7], which learns latent codes for both features and labels.
Revised: One typical example is C2AE [7], which learns latent codes for both features and labels.

Original: The latent codes are passed through a decoder to obtain target labels.
Revised: The latent codes are passed through a decoder to obtain target labels.

Original: C2AE minimizes an (cid:96) 2 distance between feature and label codes, together with a relaxed orthogonality regularization.
Revised: C2AE minimizes an (cid:96) 2 distance between feature and label codes, together with a relaxed orthogonality regularization.

Original: However, the learnt deterministic latent space lacks smoothness and structures.
Revised: However, the learnt deterministic latent space lacks smoothness and structures.

Original: Small perturbations in this latent space can lead to totally different decoding results.
Revised: Small perturbations in this latent space can lead to totally different decoding results.

Original: Even if the corresponding feature and label codes are close, we cannot guarantee the decoded targets are similar.
Revised: Even if the corresponding feature and label codes are close, we cannot guarantee the decoded targets are similar.

Original: To address this concern, MPVAE [8] proposes to replace the deterministic latent space with a probabilistic subspace under a variational autoencoder (VAE) framework.
Revised: To address this concern, MPVAE [8] proposes to replace the deterministic latent space with a probabilistic subspace under a variational autoencoder (VAE) framework.

Original: The Gaussian latent spaces are aligned with KL-divergence, and the sampling process enforces smoothness.
Revised: The Gaussian latent spaces are aligned with KL-divergence, and the sampling process enforces smoothness.

Original: Similar ideas can be found in [9].
Revised: Similar ideas can be found in [9].

Original: However, these methods assume a unimodal Gaussian latent space, which is known to cause over-regularization and posterior collapse [10, 11].
Revised: However, these methods assume a unimodal Gaussian latent space, which is known to cause over-regularization and posterior collapse [10, 11].

Original: A better strategy would be to learn a multimodal latent space.
Revised: A better strategy would be to learn a multimodal latent space.

Original: It is more intuitive to assume the observed data are generated from a multimodal subspace rather than from a unimodal one.
Revised: It is more intuitive to assume the observed data are generated from a multimodal subspace rather than from a unimodal one.

Original: Another popular group of methods targets on better label correlation modeling.
Revised: Another popular group of methods targets on better label correlation modeling.

Original: Their idea is straightforward: some labels should be more correlated if they co-appear often while others should be less relevant.
Revised: Their idea is straightforward: some labels should be more correlated if they co-appear often while others should be less relevant.

Original: Existing methods adopt pairwise ranking loss, covariance matrix, conditional random ﬁeld or even graph neural nets (GNN) [12, 13, 14, 15].
Revised: Existing methods adopt pairwise ranking loss, covariance matrix, conditional random ﬁeld or even graph neural nets (GNN) [12, 13, 14, 15].

Original: These methods often either constrain the learning through a predeﬁned structure (which requires larger space), or aren’t powerful enough to capture the correlations (such as pairwise ranking loss).
Revised: These methods often either constrain the learning through a predeﬁned structure (which requires larger space), or aren’t powerful enough to capture the correlations (such as pairwise ranking loss).

Original: Our idea is simple: we learn embeddings for each label class and the inner product between embeddings should reﬂect the similarity.
Revised: Our idea is simple: we learn embeddings for each label class and the inner product between embeddings should reﬂect the similarity.

Original: We further learn feature embeddings whose inner products with label embeddings correspond to feature-label similarity and can be used for prediction.
Revised: We further learn feature embeddings whose inner products with label embeddings correspond to feature-label similarity and can be used for prediction.

Original: We assume these embeddings are generated from a probabilistic multimodal latent space shared by features and labels, where we use KL-divergence to align the feature latent distribution and label latent distribution.
Revised: We assume these embeddings are generated from a probabilistic multimodal latent space shared by features and labels, where we use KL-divergence to align the feature latent distribution and label latent distribution.

Original: On the other hand, one may concern that embeddings might not be able to capture label-label, label-feature correlations, thus requiring extra GNN and covariance matrix [15, 8].
Revised: On the other hand, one may concern that embeddings might not be able to capture label-label, label-feature correlations, thus requiring extra GNN and covariance matrix [15, 8].

Original: To this end, we use pure losses rather than extra structures to capture these correlations.
Revised: To this end, we use pure losses rather than extra structures to capture these correlations.

Original: Intuitively, if two labels co-appear often, their embeddings should be close.
Revised: Intuitively, if two labels co-appear often, their embeddings should be close.

Original: If two labels seldom co-appear, their embeddings should be distant.
Revised: If two labels seldom co-appear, their embeddings should be distant.

Original: A triplet-like loss could be naturally applied in this scenario.
Revised: A triplet-like loss could be naturally applied in this scenario.

Original: Nevertheless, more powerful contrastive loss has shown to be more effective than the triplet loss by introducing more samples rather than just one triplet.
Revised: Nevertheless, more powerful contrastive loss has shown to be more effective than the triplet loss by introducing more samples rather than just one triplet.

Original: We show that contrastive loss can pull together correlated labels, and push away unrelated labels (see Fig. 3), which performs even better than GNN-based or covariance-based methods.
Revised: We show that contrastive loss can pull together correlated labels, and push away unrelated labels (see Fig. 3), which performs even better than GNN-based or covariance-based methods.

Original: Our new method, the contrastive learning boosted Gaussian mixture variational autoencoder (CGMVAE) multi-label prediction model, alleviates the over-regularization and posterior collapse concerns, as well as learns useful feature and label embeddings.
Revised: Our new method, the contrastive learning boosted Gaussian mixture variational autoencoder (CGMVAE) multi-label prediction model, alleviates the over-regularization and posterior collapse concerns, as well as learns useful feature and label embeddings.

Original: C-GMVAE is applied to nine datasets and outperforms the existing methods on ﬁve metrics.
Revised: C-GMVAE is applied to nine datasets and outperforms the existing methods on ﬁve metrics.

Original: Furthermore, we show that often with only 50% of the data, our results can match the full performance of other state-of-the-art methods.
Revised: Furthermore, we show that often with only 50% of the data, our results can match the full performance of other state-of-the-art methods.

Original: Ablation studies and interpretability of learnt embeddings will also be illustrated in the experiments.
Revised: Ablation studies and interpretability of learnt embeddings will also be illustrated in the experiments.

Original: Our contributions can be summarized into three aspects: (i) We propose to use contrastive loss instead of triplet or ranking loss to strengthen the label embedding learning.
Revised: Our contributions can be summarized into three aspects: (i) We propose to use contrastive loss instead of triplet or ranking loss to strengthen the label embedding learning.

Original: We empirically show that by using a contrastive loss, one can get rid of heavy-duty label correlation modules (e.g. covariance matrix,
Revised: We empirically show that by using a contrastive loss, one can get rid of heavy-duty label correlation modules (e.g. covariance matrix,

Original: GNN) while achieving even better performances.
Revised: GNN) while achieving even better performances.

Original: (ii) Though contrastive learning is commonly applied in self-supervised learning, our work shows that by properly deﬁning anchor, positive and negative samples, contrastive loss can leverage label information very effectively in the supervised
Revised: (ii) Though contrastive learning is commonly applied in self-supervised learning, our work shows that by properly deﬁning anchor, positive and negative samples, contrastive loss can leverage label information very effectively in the supervised

Original: MLC scenario as well.
Revised: MLC scenario as well.

Original: (iii) Unlike prior probabilistic models, C-GMVAE learns a multimodal latent space and associates the probabilistic modeling (VAE module) with embedding learning (contrastive module) synergistically.
Revised: (iii) Unlike prior probabilistic models, C-GMVAE learns a multimodal latent space and associates the probabilistic modeling (VAE module) with embedding learning (contrastive module) synergistically.

Original:  Methods
Revised:  Methods

Original: In MLC, given the dataset containing N samples (with labels) ( x, y ) , where x ∈ R D and y ∈ { 0 , 1 } L , our goal is to ﬁnd a mapping from x to y .
Revised: In MLC, given the dataset containing N samples (with labels) ( x, y ) , where x ∈ R D and y ∈ { 0 , 1 } L , our goal is to ﬁnd a mapping from x to y .

Original: N, D, L are sample number, feature length and label set size respectively.
Revised: N, D, L are sample number, feature length and label set size respectively.

Original: The binary coding indicates the labels associated with the sample x .
Revised: The binary coding indicates the labels associated with the sample x .

Original: Labels are correlated with each other.
Revised: Labels are correlated with each other.

Original: 2.1 Preliminaries 2.1.1 Gaussian Mixture VAE
Revised: 2.1 Preliminaries 2.1.1 Gaussian Mixture VAE

Original: A standard VAE [16] pulls together the posterior distribution and a parameter-free isotropic Gaussian prior.
Revised: A standard VAE [16] pulls together the posterior distribution and a parameter-free isotropic Gaussian prior.

Original: Two losses are optimized together in training: KL-divergence between the prior and posterior, and the distance between recontructed targets and real targets.
Revised: Two losses are optimized together in training: KL-divergence between the prior and posterior, and the distance between recontructed targets and real targets.

Original: One weakness of this formulation is the unimodality of the latent space, inhibiting the learning of more complex representations.
Revised: One weakness of this formulation is the unimodality of the latent space, inhibiting the learning of more complex representations.

Original: Another concern is over-regularization.
Revised: Another concern is over-regularization.

Original: If the posterior is exactly the same as the prior, the learnt representations would be uninformative of training inputs.
Revised: If the posterior is exactly the same as the prior, the learnt representations would be uninformative of training inputs.

Original: Numerous works extend the prior to be more complex [17, 18, 10] or learn deep hierarchies of latent variables [19, 20, 21].
Revised: Numerous works extend the prior to be more complex [17, 18, 10] or learn deep hierarchies of latent variables [19, 20, 21].

Original: In our car sea bird car embedding (𝑤 #$ ) sea embedding (𝑤 &$ ) bird embedding (𝑤 ’$ ) …  (cid:22115) 𝒚 𝒙
Revised: In our car sea bird car embedding (𝑤 #$ ) sea embedding (𝑤 &$ ) bird embedding (𝑤 ’$ ) …  (cid:22115) 𝒚 𝒙

Original: Latent Space align
Revised: Latent Space align

Original: D e c o d e r f e a t u
Revised: D e c o d e r f e a t u

Original: r e e m b e dd i n g ( 𝑤 * + ) 𝒚, feature embedding sea embedding bird embedding positive negative car embedding …
Revised: r e e m b e dd i n g ( 𝑤 * + ) 𝒚, feature embedding sea embedding bird embedding positive negative car embedding …

Original: Contrastive Loss … 𝒙,
Revised: Contrastive Loss … 𝒙,

Original: Prediction ˆ y is made by passing the feature-label embedding inner products to the sigmoid functions.
Revised: Prediction ˆ y is made by passing the feature-label embedding inner products to the sigmoid functions.

Original: In the ﬁgure, a with label set {
Revised: In the ﬁgure, a with label set {

Original: } is provided.
Revised: } is provided.

Original: work, we adopt the Gausian mixture prior.
Revised: work, we adopt the Gausian mixture prior.

Original: The probability density can be depicted as p ( z ) =
Revised: The probability density can be depicted as p ( z ) =

Original: 1 k (cid:80) ki =1 N ( z | µ i , σ 2 i ) where i is the cluster index of k Gaussian clusters with mean µ i and covariance σ 2 i .
Revised: 1 k (cid:80) ki =1 N ( z | µ i , σ 2 i ) where i is the cluster index of k Gaussian clusters with mean µ i and covariance σ 2 i .

Original: Our intuition is that each label embedding could correlate to a Gaussian subspace.
Revised: Our intuition is that each label embedding could correlate to a Gaussian subspace.

Original: Given a label set, the mixture of the corresponding Gaussians forms a unique multimodal prior distribution.
Revised: Given a label set, the mixture of the corresponding Gaussians forms a unique multimodal prior distribution.

Original: The label embeddings also receive the gradients from the contrastive loss and thus combine the contrastive learning and latent space construction.
Revised: The label embeddings also receive the gradients from the contrastive loss and thus combine the contrastive learning and latent space construction.

Original: 2.1.2 Contrastive Learning
Revised: 2.1.2 Contrastive Learning

Original: We propose to use contrastive learning to capture the correlations (feature-label, label-label).
Revised: We propose to use contrastive learning to capture the correlations (feature-label, label-label).

Original: Contrastive learning [22, 23, 24] is a novel learning style.
Revised: Contrastive learning [22, 23, 24] is a novel learning style.

Original: The core idea is simple: given an anchor sample, it should be close to similar samples (positive) and far from dissimilar samples (negative) in some learnt embedding space.
Revised: The core idea is simple: given an anchor sample, it should be close to similar samples (positive) and far from dissimilar samples (negative) in some learnt embedding space.

Original: It differs from triplet loss in the number of negative samples and the way of loss estimation.
Revised: It differs from triplet loss in the number of negative samples and the way of loss estimation.

Original: Contrastive loss is largely motivated by the noise contrastive estimation (NCE) [25] and its form is generalizable.
Revised: Contrastive loss is largely motivated by the noise contrastive estimation (NCE) [25] and its form is generalizable.

Original: The raw contrastive loss formulation only considers the instance-level invariance (multiple views of one instance), but with label information, we can learn category-level invariance (multiple instances per class/category) [26].
Revised: The raw contrastive loss formulation only considers the instance-level invariance (multiple views of one instance), but with label information, we can learn category-level invariance (multiple instances per class/category) [26].

Original: In the multi-label scenario, one can regard the feature embedding as the anchor sample, positive label embeddings as positive samples and negative label embeddings as negative samples.
Revised: In the multi-label scenario, one can regard the feature embedding as the anchor sample, positive label embeddings as positive samples and negative label embeddings as negative samples.

Original: The formulation can ﬁt the contrastive learning framework naturally and is one of our major contributions.
Revised: The formulation can ﬁt the contrastive learning framework naturally and is one of our major contributions.

Original: Compared to pairwise ranking loss which focuses on the ﬁnal digits, contrastive loss is deﬁned on the embeddings and thus more expressive.
Revised: Compared to pairwise ranking loss which focuses on the ﬁnal digits, contrastive loss is deﬁned on the embeddings and thus more expressive.

Original: Contrastive loss also includes more samples in estimating the NCE and therefore outperforms triplet loss.
Revised: Contrastive loss also includes more samples in estimating the NCE and therefore outperforms triplet loss.

Original: In appendix, we show triplet loss is actually a special case of our contrastive loss.
Revised: In appendix, we show triplet loss is actually a special case of our contrastive loss.

Original: 2.2 C-GMVAE
Revised: 2.2 C-GMVAE

Original: C-GMVAE inherits the general variational autoencoder framework, but with a learnable Gaussian mixture prior.
Revised: C-GMVAE inherits the general variational autoencoder framework, but with a learnable Gaussian mixture prior.

Original: During training, the sample’s label set activates and mixes the related Gaussian clusters to derive the prior.
Revised: During training, the sample’s label set activates and mixes the related Gaussian clusters to derive the prior.

Original: Contrastive learning is applied to boost the embedding learning, using a contrastive loss between the feature and label embeddings.
Revised: Contrastive learning is applied to boost the embedding learning, using a contrastive loss between the feature and label embeddings.

Original: Fig.
Revised: Fig.

Original: 1 provides a full illustration, and the following subsections will elaborate on the details.
Revised: 1 provides a full illustration, and the following subsections will elaborate on the details.

Original: 2.2.1 Gaussian Mixture Latent Space
Revised: 2.2.1 Gaussian Mixture Latent Space

Original: Given a sample ( x, y ) where feature x ∈ R D and label y ∈ { 0 , 1 } L , many previous works take y as the input and transform it to a dense representation through a fully-connected layer [7, 8].
Revised: Given a sample ( x, y ) where feature x ∈ R D and label y ∈ { 0 , 1 } L , many previous works take y as the input and transform it to a dense representation through a fully-connected layer [7, 8].

Original: This layer essentially maps each label category to an embedding and sums up all the embeddings using label y as weights (0 or 1).
Revised: This layer essentially maps each label category to an embedding and sums up all the embeddings using label y as weights (0 or 1).

Original: The ﬁnal embedding is fed into the label encoder to produce a probabilistic space.
Revised: The ﬁnal embedding is fed into the label encoder to produce a probabilistic space.

Original: In C-GMVAE, we directly map each per-category label embedding w li ∈ R E of label class i to an individual Gaussian distribution N ( µ i , diag ( σ 2 i )) , µ i ∈ R d , σ 2 i ∈ R d . µ i , σ 2 i are derived from w li through NN.
Revised: In C-GMVAE, we directly map each per-category label embedding w li ∈ R E of label class i to an individual Gaussian distribution N ( µ i , diag ( σ 2 i )) , µ i ∈ R d , σ 2 i ∈ R d . µ i , σ 2 i are derived from w li through NN.

Original: In Fig.
Revised: In Fig.

Original: 1, the label categories car, sea,..., bird are transformed to embeddings ﬁrst.
Revised: 1, the label categories car, sea,..., bird are transformed to embeddings ﬁrst.

Original: Embeddings are then directly passed to label encoder rather than summed up.
Revised: Embeddings are then directly passed to label encoder rather than summed up.

Original: Each label category (e.g. car ) corresponds to a unimodal Gaussian in the latent space.
Revised: Each label category (e.g. car ) corresponds to a unimodal Gaussian in the latent space.

Original: y activates “positive" Gaussians and forms a Gaussian mixture subspace.
Revised: y activates “positive" Gaussians and forms a Gaussian mixture subspace.

Original: Given a random variable z ∈ R d , the probability density function (PDF) in the subspace is deﬁned as
Revised: Given a random variable z ∈ R d , the probability density function (PDF) in the subspace is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: 1 ( · ) is the indicator function and the label encoder is parameterized by ψ (NN).
Revised: 1 ( · ) is the indicator function and the label encoder is parameterized by ψ (NN).

Original: In Fig.
Revised: In Fig.

Original: 1, y activates sea and bird , p ψ ( z | y )
Revised: 1, y activates sea and bird , p ψ ( z | y )

Original: =
Revised: =

Original: 12 ( N ( z | µ sea , diag ( σ 2 sea ))
Revised: 12 ( N ( z | µ sea , diag ( σ 2 sea ))

Original: + N ( z | µ bird , diag ( σ 2 bird ))) .
Revised: + N ( z | µ bird , diag ( σ 2 bird ))) .

Original: Most VAE-based frameworks optimize over an evidence lower bound (ELBO)
Revised: Most VAE-based frameworks optimize over an evidence lower bound (ELBO)

Original: [Equation]
Revised: [Equation]

Original: The feature encoder is parameterized by φ (NN).
Revised: The feature encoder is parameterized by φ (NN).

Original: One pitfall of this objective is owing to the minimization of KL-divergence.
Revised: One pitfall of this objective is owing to the minimization of KL-divergence.

Original: If the divergence between the posterior q φ ( z | x ) and the prior p ψ ( z ) vanishes, the learnt latent codes would be non-informative.
Revised: If the divergence between the posterior q φ ( z | x ) and the prior p ψ ( z ) vanishes, the learnt latent codes would be non-informative.

Original: This is the so-called posterior collapse.
Revised: This is the so-called posterior collapse.

Original: Many recent works suggest learnable priors [28] and more sophisticated priors [29] to avoid this issue.
Revised: Many recent works suggest learnable priors [28] and more sophisticated priors [29] to avoid this issue.

Original: We adopt these ideas in our design of the prior.
Revised: We adopt these ideas in our design of the prior.

Original: Compared to a standard VAE, our prior is informative, learnable and multimodal.
Revised: Compared to a standard VAE, our prior is informative, learnable and multimodal.

Original: We form a standard posterior in our model and match it with the prior.
Revised: We form a standard posterior in our model and match it with the prior.

Original: However, unlike vanilla VAE, we cannot analytically compute the KL term.
Revised: However, unlike vanilla VAE, we cannot analytically compute the KL term.

Original: Instead, we use the following estimation:
Revised: Instead, we use the following estimation:

Original: [Equation]
Revised: [Equation]

Original: where z 0 ∼ q φ ( z | x ) denotes a single latent sample.
Revised: where z 0 ∼ q φ ( z | x ) denotes a single latent sample.

Original: The reconstruction loss remains to be a standard negative log-likelihood (we add the minus since the objective function is to be minimized.
Revised: The reconstruction loss remains to be a standard negative log-likelihood (we add the minus since the objective function is to be minimized.

Original: θ is the decoder parameters.),
Revised: θ is the decoder parameters.),

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 2.2.2 Contrastive Learning Module
Revised: 2.2.2 Contrastive Learning Module

Original: The decoder function f dθ ( · ) decodes the sample from the latent space to a feature embedding w fx ∈ R E .
Revised: The decoder function f dθ ( · ) decodes the sample from the latent space to a feature embedding w fx ∈ R E .

Original: We train w fx together with label embeddings { w l i } L i =1 .
Revised: We train w fx together with label embeddings { w l i } L i =1 .

Original: The objective function includes contrastive loss and cross-entropy loss terms.
Revised: The objective function includes contrastive loss and cross-entropy loss terms.

Original: Prior works used to explicitly capture the label-label interactions by GNN or covariance modules, which imposes the structure a priori and might not be the best way.
Revised: Prior works used to explicitly capture the label-label interactions by GNN or covariance modules, which imposes the structure a priori and might not be the best way.

Original: Our contrastive module instead captures the correlation completely driven by data.
Revised: Our contrastive module instead captures the correlation completely driven by data.

Original: For example, if in the majority of all samples, “beach" and “sunshine" appear together, the contrastive learning will implicitly pull their embeddings together [24].
Revised: For example, if in the majority of all samples, “beach" and “sunshine" appear together, the contrastive learning will implicitly pull their embeddings together [24].

Original: In other words, if two labels do co-appear often, their label embeddings become similar (Fig. 3).
Revised: In other words, if two labels do co-appear often, their label embeddings become similar (Fig. 3).

Original: On the other hand, if they only co-appear occasionally, their relations are not signiﬁcant and our module will not optimize for their similarity.
Revised: On the other hand, if they only co-appear occasionally, their relations are not signiﬁcant and our module will not optimize for their similarity.

Original: Original contrastive learning [22] augments inputs and learns instance-level invariance, but may not generalize to category-level invariance.
Revised: Original contrastive learning [22] augments inputs and learns instance-level invariance, but may not generalize to category-level invariance.

Original: In the supervised setting, however, the learning can beneﬁt from labels and discover category-level invariance [24].
Revised: In the supervised setting, however, the learning can beneﬁt from labels and discover category-level invariance [24].

Original: Let A ≡ { 1 ...L } .
Revised: Let A ≡ { 1 ...L } .

Original: We deﬁne P ( y ) ≡ { i ∈
Revised: We deﬁne P ( y ) ≡ { i ∈

Original: A : y i = 1 } for sample ( x, y ) .
Revised: A : y i = 1 } for sample ( x, y ) .

Original: Suppose we have a batch of samples, B , the contrastive loss can be written as
Revised: Suppose we have a batch of samples, B , the contrastive loss can be written as

Original: [Equation]
Revised: [Equation]

Original: sim ( · ) is a function measuring the similarity between two embeddings, and w fx , w li denote the feature and label embeddings respectively.
Revised: sim ( · ) is a function measuring the similarity between two embeddings, and w fx , w li denote the feature and label embeddings respectively.

Original: Eq.
Revised: Eq.

Original: 5 is built on top of noise-contrastive estimation [25], and the equation is equivalent to a categorical cross-entropy of correctly predicting positive labels.
Revised: 5 is built on top of noise-contrastive estimation [25], and the equation is equivalent to a categorical cross-entropy of correctly predicting positive labels.

Original: The choice of sim ( · ) can be a log-bilinear function [22], or a more complicated neural metric function [23].
Revised: The choice of sim ( · ) can be a log-bilinear function [22], or a more complicated neural metric function [23].

Original: In our experiments, we found it is simple and effective to take sim ( w 1 , w 2 ) =
Revised: In our experiments, we found it is simple and effective to take sim ( w 1 , w 2 ) =

Original: exp( w 1 · w 2 /τ ) where · means inner product and τ is a temperature parameter controlling the scale of the inner products.
Revised: exp( w 1 · w 2 /τ ) where · means inner product and τ is a temperature parameter controlling the scale of the inner products.

Original: In
Revised: In

Original: SupCon [24], if only 1 class is positive, all other classes are contrastive to it.
Revised: SupCon [24], if only 1 class is positive, all other classes are contrastive to it.

Original: However, in multi-label, if “beach" is positive in the label while “sea" isn’t for one particular sample, we cannot say these two classes are contrastive.
Revised: However, in multi-label, if “beach" is positive in the label while “sea" isn’t for one particular sample, we cannot say these two classes are contrastive.

Original: Their correlation will be captured implicitly by all the samples.
Revised: Their correlation will be captured implicitly by all the samples.

Original: Therefore, we do not assume contrastive relations between labels and preserve the label correlations.
Revised: Therefore, we do not assume contrastive relations between labels and preserve the label correlations.

Original: We instead choose the feature embedding to be the anchor and label embeddings to be positive/negative samples.
Revised: We instead choose the feature embedding to be the anchor and label embeddings to be positive/negative samples.

Original: If two label embeddings co-appear often as positive samples, they would implicitly become similar (see Fig. 3).
Revised: If two label embeddings co-appear often as positive samples, they would implicitly become similar (see Fig. 3).

Original: Eq.
Revised: Eq.

Original: 5 saves the effort of manually conﬁguring the positive and negative samples, and is totally data-driven.
Revised: 5 saves the effort of manually conﬁguring the positive and negative samples, and is totally data-driven.

Original: The number of positives or negatives could be greater than one.
Revised: The number of positives or negatives could be greater than one.

Original: Note that though L limits the max samples we can have, this formulation has already used many more samples compared to triplet loss, and we will show in experiments that this formulation is very effective.
Revised: Note that though L limits the max samples we can have, this formulation has already used many more samples compared to triplet loss, and we will show in experiments that this formulation is very effective.

Original: The triplet loss often used in multi-label learning [30] can be seen as a special case of Eq.
Revised: The triplet loss often used in multi-label learning [30] can be seen as a special case of Eq.

Original: 5 with only one positive and one negative.
Revised: 5 with only one positive and one negative.

Original: We illustrate the connection in the appendix.
Revised: We illustrate the connection in the appendix.

Original: Furthermore, one desired property of embedding learning is that when a good positive embedding is already close enough to our anchor embedding, it contributes less to the gradients, while poorly learnt embeddings contribute more to improve the model performance.
Revised: Furthermore, one desired property of embedding learning is that when a good positive embedding is already close enough to our anchor embedding, it contributes less to the gradients, while poorly learnt embeddings contribute more to improve the model performance.

Original: It has been shown that the contrastive loss can implicitly achieve this goal and a full derivation of the gradients [24].
Revised: It has been shown that the contrastive loss can implicitly achieve this goal and a full derivation of the gradients [24].

Original: Our objective function also includes a supervised cross-entropy loss term to further facilitate the training.
Revised: Our objective function also includes a supervised cross-entropy loss term to further facilitate the training.

Original: With the label embeddings w li and the feature embedding w fx , the cross entropy loss is given by
Revised: With the label embeddings w li and the feature embedding w fx , the cross entropy loss is given by

Original: [Equation]
Revised: [Equation]

Original: where function s ( · ) is the sigmoid function.
Revised: where function s ( · ) is the sigmoid function.

Original: In self-supervised learning, the contrastive loss typically helps the pretraining stage and the learnt representations are applied to downstream tasks.
Revised: In self-supervised learning, the contrastive loss typically helps the pretraining stage and the learnt representations are applied to downstream tasks.

Original: In the supervised setting, though some models [24] stick to the two-stage training process where the model is trained with contrastive loss in the ﬁrst stage and cross-entropy loss in the second stage, we didn’t observe its superiority to the one-stage scheme where we train the model with an objective function incorporating all losses.
Revised: In the supervised setting, though some models [24] stick to the two-stage training process where the model is trained with contrastive loss in the ﬁrst stage and cross-entropy loss in the second stage, we didn’t observe its superiority to the one-stage scheme where we train the model with an objective function incorporating all losses.

Original: This is partly because we also learn a latent space that is closely connected to label embeddings.
Revised: This is partly because we also learn a latent space that is closely connected to label embeddings.

Original: A joint training strategy reconciles different modules.
Revised: A joint training strategy reconciles different modules.

Original: We show in the experiments that the learnt embeddings are semantically meaningful and can reveal the label correlations.
Revised: We show in the experiments that the learnt embeddings are semantically meaningful and can reveal the label correlations.

Original: 2.2.3 Objective Function
Revised: 2.2.3 Objective Function

Original: The ﬁnal objective function is simply the summation of different losses,
Revised: The ﬁnal objective function is simply the summation of different losses,

Original: [Equation]
Revised: [Equation]

Original: where α, β are trade-off weights.
Revised: where α, β are trade-off weights.

Original: The model is trained with Adam [31].
Revised: The model is trained with Adam [31].

Original: Our model is optimized with
Revised: Our model is optimized with

Original: L but will be tested on ﬁve different metrics.
Revised: L but will be tested on ﬁve different metrics.

Original: This is different from the methods that optimize speciﬁc metrics [32, 33].
Revised: This is different from the methods that optimize speciﬁc metrics [32, 33].

Original: 2.2.4 Prediction
Revised: 2.2.4 Prediction

Original: During the testing phase, the input sample x will be passed to the feature encoder and decoder to obtain its embedding w fx .
Revised: During the testing phase, the input sample x will be passed to the feature encoder and decoder to obtain its embedding w fx .

Original: Label embeddings w li are ﬁxed during testing.
Revised: Label embeddings w li are ﬁxed during testing.

Original: The inner products between w fx and w li will be passed through a sigmoid function to obtain prediction probability for class i .
Revised: The inner products between w fx and w li will be passed through a sigmoid function to obtain prediction probability for class i .

Original: 2.3 Insights behind C-GMVAE
Revised: 2.3 Insights behind C-GMVAE

Original: C2AE and MPVAE have shown the importance of learning a shared latent space for both features and labels.
Revised: C2AE and MPVAE have shown the importance of learning a shared latent space for both features and labels.

Original: These methods share the same high-level insight similar to teacher-student regime: we map labels (teacher) to a latent space with some certain structure, which preserves the label information and is easier to be decoded back to labels.
Revised: These methods share the same high-level insight similar to teacher-student regime: we map labels (teacher) to a latent space with some certain structure, which preserves the label information and is easier to be decoded back to labels.

Original: Then the features (student) are expected to be mapped to
Revised: Then the features (student) are expected to be mapped to

Original: [Table] [Table]
Revised: [Table] [Table]

Original: this latent space to facilitate the label prediction.
Revised: this latent space to facilitate the label prediction.

Original: Two general concerns exist for these methods: 1) the uni-Gaussian space previously used is too restrictive to impose sophisticated structures for prior, 2) how to properly capture label correlations with embeddings.
Revised: Two general concerns exist for these methods: 1) the uni-Gaussian space previously used is too restrictive to impose sophisticated structures for prior, 2) how to properly capture label correlations with embeddings.

Original: For the ﬁrst, we learn a modality for each label class to form a mixture latent space.
Revised: For the ﬁrst, we learn a modality for each label class to form a mixture latent space.

Original: For the second, we replace the commonly used ranking and triplet losses with contrastive loss since contrastive loss involves more samples than triplet loss and has a larger capacity than ranking loss.
Revised: For the second, we replace the commonly used ranking and triplet losses with contrastive loss since contrastive loss involves more samples than triplet loss and has a larger capacity than ranking loss.

Original: [Table]
Revised: [Table]

Original:  Related Work
Revised:  Related Work

Original: Learning a shared latent space for features and labels is a common and useful idea.
Revised: Learning a shared latent space for features and labels is a common and useful idea.

Original: In singlelabel prediction tasks, CADA-VAE [34] learns and aligns latent label and feature spaces through distribution alignment losses.
Revised: In singlelabel prediction tasks, CADA-VAE [34] learns and aligns latent label and feature spaces through distribution alignment losses.

Original: Similar ideas can be seen in out-of-distribution detection as well [9].
Revised: Similar ideas can be seen in out-of-distribution detection as well [9].

Original: In multi-label scenarios, methods adopting this idea typically have a similar module that directly maps the multi-hot labels to embeddings [7, 35, 8].
Revised: In multi-label scenarios, methods adopting this idea typically have a similar module that directly maps the multi-hot labels to embeddings [7, 35, 8].

Original: This is a rather difﬁcult learning task.
Revised: This is a rather difﬁcult learning task.

Original: Suppose we have 30 label categories.
Revised: Suppose we have 30 label categories.

Original: There could be up to 2 30 label sets.
Revised: There could be up to 2 30 label sets.

Original: For probabilistic models like MPVAE, that means one latent label space has to represent up to 2 30 label combinations.
Revised: For probabilistic models like MPVAE, that means one latent label space has to represent up to 2 30 label combinations.

Original: In contrast, C-GMVAE learns per-category subspaces and forms a mixture prior distribution based on the observed samples’ label sets.
Revised: In contrast, C-GMVAE learns per-category subspaces and forms a mixture prior distribution based on the observed samples’ label sets.

Original: Learning a mixture latent space has been explored in prior works [10, 36].
Revised: Learning a mixture latent space has been explored in prior works [10, 36].

Original: But none of them applied their methods on multi-label prediction tasks.
Revised: But none of them applied their methods on multi-label prediction tasks.

Original: Contrastive learning has become one of the most popular self-supervised learning techniques.
Revised: Contrastive learning has become one of the most popular self-supervised learning techniques.

Original: It has also drawn attention in supervised tasks.
Revised: It has also drawn attention in supervised tasks.

Original: SupCon [24] ﬁrst demonstrated the effectiveness of supervised contrastive loss in image classiﬁcation.
Revised: SupCon [24] ﬁrst demonstrated the effectiveness of supervised contrastive loss in image classiﬁcation.

Original: It was soon generalized to other domains like visual reasoning [37].
Revised: It was soon generalized to other domains like visual reasoning [37].

Original: Nevertheless, these methods depend on vision-speciﬁc augmentation techniques.
Revised: Nevertheless, these methods depend on vision-speciﬁc augmentation techniques.

Original: Another related work is multi-label contrastive learning [38].
Revised: Another related work is multi-label contrastive learning [38].

Original: But the work does not deal with MLC.
Revised: But the work does not deal with MLC.

Original: Instead, it extends the contrastive learning to identifying more than 1 positive sample, which resembles a multi-label scenario.
Revised: Instead, it extends the contrastive learning to identifying more than 1 positive sample, which resembles a multi-label scenario.

Original: Some earlier works also attempted metric learning or triplet loss in MLC [39].
Revised: Some earlier works also attempted metric learning or triplet loss in MLC [39].

Original: Triplet loss typically only takes one pair of positive and negative samples for one anchor, while contrastive loss uses many more negative/positive samples.
Revised: Triplet loss typically only takes one pair of positive and negative samples for one anchor, while contrastive loss uses many more negative/positive samples.

Original: Recent papers found that more samples can greatly boost the performance [23, 26].
Revised: Recent papers found that more samples can greatly boost the performance [23, 26].

Original: Note that though our contrastive module is constrained by the maximum number of label classes, it has already used many more samples than triplet loss, and our observations support that more samples help with the performance.
Revised: Note that though our contrastive module is constrained by the maximum number of label classes, it has already used many more samples than triplet loss, and our observations support that more samples help with the performance.

Original:  Experiments
Revised:  Experiments

Original: We have various setups to validate the performance of C-GMVAE.
Revised: We have various setups to validate the performance of C-GMVAE.

Original: First, we compare the example-F1, micro-F1 and macro-F1 scores, Hamming accuracies and precision@1 of different methods.
Revised: First, we compare the example-F1, micro-F1 and macro-F1 scores, Hamming accuracies and precision@1 of different methods.

Original: Second, we compare their performances when fewer training data are available.
Revised: Second, we compare their performances when fewer training data are available.

Original: Third, an ablation study shows the importance of the proposed modules.
Revised: Third, an ablation study shows the importance of the proposed modules.

Original: Finally, we demonstrate the interpretability of label embeddings on an eBird dataset.
Revised: Finally, we demonstrate the interpretability of label embeddings on an eBird dataset.

Original: Our code is available anonymously 1 .

Original: 4.1 Setup
Revised: 4.1 Setup

Original: For the main evaluation experiments, we use nine datasets, including image datasets mirﬂickr , nuswide , scene [40, 41, 42], biology datasets sider , yeast [43, 44], ecology dataset eBird [45], text datasets reuters, bookmarks, delicious [46, 47, 48].
Revised: For the main evaluation experiments, we use nine datasets, including image datasets mirﬂickr , nuswide , scene [40, 41, 42], biology datasets sider , yeast [43, 44], ecology dataset eBird [45], text datasets reuters, bookmarks, delicious [46, 47, 48].

Original: All features are collected in the vector format [15, 8].
Revised: All features are collected in the vector format [15, 8].

Original: The feature pre-processing is standard following previous works [15, 8] and the datasets are publicly available 2 .
Revised: The feature pre-processing is standard following previous works [15, 8] and the datasets are publicly available 1 .

Original: Each of them is separated into training (80%), validation (10%) and testing [Table] [Table] [Table]
Revised: Each of them is separated into training (80%), validation (10%) and testing (10%) splits.

Original: %) splits.

Original: The datasets are also preprocessed to ﬁt the input formats of different methods.
Revised: The datasets are also preprocessed to ﬁt the input formats of different methods.

Original: We use mini-batch training with batch size 128.
Revised: We use mini-batch training with batch size 128.

Original: Each batch is randomly sampled from the dataset.
Revised: Each batch is randomly sampled from the dataset.

Revised: [Table]

Revised: [Table]

Original: The evaluation metrics are three F1 scores, hamming accuracy and precision@1.
Revised: The evaluation metrics are three F1 scores, hamming accuracy and precision@1.

Original: The evaluation process, model selection and preprocessing strictly follow previous works [49, 15, 8].
Revised: The evaluation process, model selection and preprocessing strictly follow previous works [49, 15, 8].

Original: Most numbers are also directly quoted from their papers for comparison.
Revised: Most numbers are also directly quoted from their papers for comparison.

Original: Our method is compared against MPVAE [8], LaMP [15], C2AE [7], SLEEC [6], HARAM [50], MLKNN [5], and BR [51].
Revised: Our method is compared against MPVAE [8], LaMP [15], C2AE [7], SLEEC [6], HARAM [50], MLKNN [5], and BR [51].

Original: MPVAE is a novel method which learns and aligns the probabilistic feature and label subspaces.
Revised: MPVAE is a novel method which learns and aligns the probabilistic feature and label subspaces.

Original: Label correlations are captured by a Multivariate Probit module.
Revised: Label correlations are captured by a Multivariate Probit module.

Original: LaMP adopts attention-based neural message passing to handle label correlations, which is a neural extension to previous CRF-based methods.
Revised: LaMP adopts attention-based neural message passing to handle label correlations, which is a neural extension to previous CRF-based methods.

Original: C2AE is one of the ﬁrst papers which use neural networks to learn and align latent spaces.
Revised: C2AE is one of the ﬁrst papers which use neural networks to learn and align latent spaces.

Original: C2AE imposes a CCA constraint on latent space.
Revised: C2AE imposes a CCA constraint on latent space.

Original: SLEEC explores the low-rank assumption in MLC.
Revised: SLEEC explores the low-rank assumption in MLC.

Original: Low-rank assumption also builds the foundation for other deep methods.
Revised: Low-rank assumption also builds the foundation for other deep methods.

Original: HARAM is one of the ﬁrst methods which introduced neural nets to MLC.
Revised: HARAM is one of the ﬁrst methods which introduced neural nets to MLC.

Original: Lastly, MLKNN is a classic MLC method using K-nearest neighbors (KNN).
Revised: Lastly, MLKNN is a classic MLC method using K-nearest neighbors (KNN).

Original: 4.2 Metrics
Revised: 4.2 Metrics

Original: We evaluate our method trained with objective Eq. 7 on several commonly used multi-label metrics.
Revised: We evaluate our method trained with objective Eq. 7 on several commonly used multi-label metrics.

Original: Suppose the ground-truth label is y and the predicted label is ˆ y .
Revised: Suppose the ground-truth label is y and the predicted label is ˆ y .

Original: We denote true positives, false positives, false negatives by tp j , fp j , fn j respectively for the j -th of L label categors.
Revised: We denote true positives, false positives, false negatives by tp j , fp j , fn j respectively for the j -th of L label categors.

Original: (i) HA:1 L
Revised: (i) HA:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] ] (ii) example-F1:
Revised: ii

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: Furthermore, precision@1 is the proportion of correctly predicted labels in the top-1 predictions.
Revised: Furthermore, precision@1 is the proportion of correctly predicted labels in the top-1 predictions.

Original: 4.3 Architecture and Hyperparameters
Revised: 4.3 Architecture and Hyperparameters

Original: As we state in the introduction, we do not require very sophisticated neural architectures in C-GMVAE.
Revised: As we state in the introduction, we do not require very sophisticated neural architectures in C-GMVAE.

Original: All the neural layers are fully connected.
Revised: All the neural layers are fully connected.

Original: The feature encoder is a fully connected neural network with 3 hidden layers and the activation function is ReLU.
Revised: The feature encoder is a fully connected neural network with 3 hidden layers and the activation function is ReLU.

Original: The label encoder is also fully connected comprising two hidden layers and the decoder has two hidden layers as well.
Revised: The label encoder is also fully connected comprising two hidden layers and the decoder has two hidden layers as well.

Original: More details of the model can be found in the appendix.
Revised: More details of the model can be found in the appendix.

Original: We set α = 1 , β = 0 .
Revised: We set α = 1 , β = 0 .

Original: 5 , E = 2048 from tuning for most runs.
Revised: 5 , E = 2048 from tuning for most runs.

Original: Grid search is applied to ﬁnd the best learning rate, dropout ratio, weight decay ratio for each dataset.
Revised: Grid search is applied to ﬁnd the best learning rate, dropout ratio, weight decay ratio for each dataset.

Original: We use one V100 GPU for all experiments.
Revised: We use one V100 GPU for all experiments.

Original: More architecture, hyper-parameter tuning and ﬁnal selection (Tab. 7 in Appx), and implementation details can be found in the appendix.[Figure] [Figure]
Revised: More architecture, hyper-parameter tuning and ﬁnal selection (Tab. 7 in Appx), and implementation details can be found in the appendix.

Original: 4.4 Evaluations
Revised: 4.4 Evaluations

Original: Full supervision
Revised: Full supervision

Original: In the full supervision scenario which is commonly adopted by the methods we compare against, we evaluate four metrics: example-F1 (ex-F1), micro-F1 (mi-F1), macro-F  (ma-F1), Hamming accuracy (HA) and precision@1.
Revised: In the full supervision scenario which is commonly adopted by the methods we compare against, we evaluate four metrics: example-F1 (ex-F1), micro-F1 (mi-F1), macro-F [Figure] [Figure] (ma-F1), Hamming accuracy (HA) and precision@1.

Original: ex-F1 score is the averaged F1-score over all the samples.
Revised: ex-F1 score is the averaged F1-score over all the samples.

Original: mi-F1 score measures the aggregated contributions of all classes.
Revised: mi-F1 score measures the aggregated contributions of all classes.

Original: ma-F1 treats each class equally and takes the class-wise average.
Revised: ma-F1 treats each class equally and takes the class-wise average.

Original: HA counts the correctly predicted labels regardless of samples or classes.
Revised: HA counts the correctly predicted labels regardless of samples or classes.

Original: Tab. 1, 2 and 3 present the performance of all the methods w.r.t.
Revised: Tab. 1, 2 and 3 present the performance of all the methods w.r.t.

Original: the metrics.
Revised: the metrics.

Original: We abbreviate nuswidevector to nus-vec , and bookmarks to bkms . C-GMVAE outperforms the existing state-of-the-art methods on all the datasets. The best numbers are marked in bold. All the numbers for C-GMVAE are averaged over 3 seeds for stability and the standard deviations are included in the table.
Revised: We abbreviate nuswidevector to nus-

Original: On ex-F1,

Original: C-GMVAE improves over MPVAE by 2.5%, and LaMP by 8.8% on average across all the datasets..

Original: Similarly, on mi-F1, C-GMVAE improves over MPVAE by 2.4% and LaMP 6.1% on average.

Original: On ma-F1, the improvements are as large as 4.1% and 11% respectively.

Original: C-GMVAE outperforms other methods consistently. Ablation study To demonstrate the strength of our C-GMVAE, we compare it with a uni-Gaussian latent model, a Gaussian mixture (GM) only latent model (without contrastive module), and a contrastive only model (without KL term) in Tab. 4.
Revised: Ablation study To demonstrate the strength of our C-GMVAE, we compare it with a uni-Gaussian latent model, a Gaussian mixture (GM) only latent model (without contrastive module), and a contrastive only model (without KL term) in Tab. 4.

Original: Our C-GMVAE (GM+contrastive) consistently outperforms other models by a margin.
Revised: Our C-GMVAE (GM+contrastive) consistently outperforms other models by a margin.

Original: For instance, on ma-F1, C-GMVAE improves over uniGaussian model by 7%.
Revised: For instance, on ma-F1, C-GMVAE improves over uniGaussian model by 7%.

Original: Training on fewer data Contrastive learning learns contrastive views and thus requires less information compared to generative learning which demands a more complete representation for reconstruction.
Revised: Training on fewer data Contrastive learning learns contrastive views and thus requires less information compared to generative learning which demands a more complete representation for reconstruction.

Original: Contrastive learning has the potential to discover the intrinsic structures present in the data, and therefore is widely used in self-supervised learning since it generalizes well.
Revised: Contrastive learning has the potential to discover the intrinsic structures present in the data, and therefore is widely used in self-supervised learning since it generalizes well.

Original: We observe this with C-GMVAE as well.
Revised: We observe this with C-GMVAE as well.

Original: To demonstrate this, we shrink the size of training data by 50% or 90% and train methods on them.
Revised: To demonstrate this, we shrink the size of training data by 50% or 90% and train methods on them.

Original: Surprisingly, we found C-GMVAE can often match the performance of other methods with only 50% of the training data.
Revised: Surprisingly, we found C-GMVAE can often match the performance of other methods with only 50% of the training data.

Original: Tab. 5 compares MPVAE trained on full data and
Revised: Tab. 5 compares MPVAE trained on full data and

Original: C-GMVAE trained on 50% of data.
Revised: C-GMVAE trained on 50% of data.

Original: Their performances approximately match.
Revised: Their performances approximately match.

Original: We further compare several major state-of-the-art methods including ours all trained on the same randomly selected 10%, 50% and 100% of the data and show their performance over C2AE.
Revised: We further compare several major state-of-the-art methods including ours all trained on the same randomly selected 10%, 50% and 100% of the data and show their performance over C2AE.

Original: Fig.
Revised: Fig.

Original: 2 shows the improvements over C2AE on ma-F1.
Revised: 2 shows the improvements over C2AE on ma-F1.

Original: Ours clearly outperforms others on fewer data.
Revised: Ours clearly outperforms others on fewer data.

Original: Interpretability Our work is also motivated by ecological applications [52], where it is important to understand species interactions.
Revised: Interpretability Our work is also motivated by ecological applications [52], where it is important to understand species interactions.

Original: Fig.
Revised: Fig.

Original: 3 shows a map of label-label embedding inner-product weights for the eBird dataset.
Revised: 3 shows a map of label-label embedding inner-product weights for the eBird dataset.

Original: The bird species on the x-axis and the y-axis are the same.
Revised: The bird species on the x-axis and the y-axis are the same.

Original: The ﬁrstbird species are water birds.
Revised: The ﬁrstbird species are water birds.

Original: The following 4 bird species are forest birds.
Revised: The following 4 bird species are forest birds.

Original: The last 3 bird species are residential birds.
Revised: The last 3 bird species are residential birds.

Original: The darker the grid is, the closer two birds will be.
Revised: The darker the grid is, the closer two birds will be.

Original: We subtract the diagonal to exclude the self correlation.
Revised: We subtract the diagonal to exclude the self correlation.

Original: The heatmap matrix clearly form three blocks.
Revised: The heatmap matrix clearly form three blocks.

Original: The ﬁrst block contains
Revised: The ﬁrst block contains

Original: Black-backed Gull, Rough-winged Swallow and Great Blue Heron.
Revised: Black-backed Gull, Rough-winged Swallow and Great Blue Heron.

Original: These three birds are water birds living near sea or lake.
Revised: These three birds are water birds living near sea or lake.

Original: The second block has Tufted Titmouse, Northern Flicker, Northern
Revised: The second block has Tufted Titmouse, Northern Flicker, Northern

Original: Mockingbird, and Cedar Waxwing.
Revised: Mockingbird, and Cedar Waxwing.

Original: These birds typically live in the forest with a lot of trees.
Revised: These birds typically live in the forest with a lot of trees.

Original: The remaining birds are commonly seen residential birds, Mourning Dove, House Sparrow and Common
Revised: The remaining birds are commonly seen residential birds, Mourning Dove, House Sparrow and Common

Original: Starling.
Revised: Starling.

Original: They live inside or near human residences.
Revised: They live inside or near human residences.

Original: Since human activities are wide-spread, the distribution of these birds are therefore quite broad.
Revised: Since human activities are wide-spread, the distribution of these birds are therefore quite broad.

Original: For example, Mourning Dove also has some correlations with forest birds in Fig.
Revised: For example, Mourning Dove also has some correlations with forest birds in Fig.

Original: But one can observe that for each group of birds, their intra-group correlations are always stronger than inter-group correlations.
Revised: But one can observe that for each group of birds, their intra-group correlations are always stronger than inter-group correlations.

Original: Therefore, the learnt embeddings do encompass semantic meanings.
Revised: Therefore, the learnt embeddings do encompass semantic meanings.

Original: The derived correlaions could also help the study of wildlife protection [53].
Revised: The derived correlaions could also help the study of wildlife protection [53].

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose a contrastive learning boosted Gaussian mixture variational autoencoder (C-GMVAE) multi-label predictor, a novel method for MLC.
Revised: In this work we propose a contrastive learning boosted Gaussian mixture variational autoencoder (C-GMVAE) multi-label predictor, a novel method for MLC.

Original: C-GMVAE combines the learning of a
Revised: C-GMVAE combines the learning of a

Original: Gaussian mixture latent space with the contrastive learning of feature and label embeddings.
Revised: Gaussian mixture latent space with the contrastive learning of feature and label embeddings.

Original: Not only does C-GMVAE achieve the state-of-the-art performance, it also provides insights into semisupervised learning and model interpretability.
Revised: Not only does C-GMVAE achieve the state-of-the-art performance, it also provides insights into semisupervised learning and model interpretability.

Original: Interesting future directions include the exploration of various contrastive learning mechanisms, model architecture improvements, and other latent space structures.
Revised: Interesting future directions include the exploration of various contrastive learning mechanisms, model architecture improvements, and other latent space structures.

Original: A.1 Connection with Triplet Loss
Revised: A.1 Connection with Triplet Loss

Original: Triplet loss [54] is one of the popular ranking losses used in multi-label learning [30].
Revised: Triplet loss [54] is one of the popular ranking losses used in multi-label learning [30].

Original: Given an anchor embedding v fx , a positive embedding v + and a negative embedding v − , they form a triplet ( v fx , v + , v − ) .
Revised: Given an anchor embedding v fx , a positive embedding v + and a negative embedding v − , they form a triplet ( v fx , v + , v − ) .

Original: A triplet loss is deﬁned as
Revised: A triplet loss is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where g is a gap parameter measuring the distance between ( v fx , v + ) and ( v fx , v − ) , and dist ( · , · ) is a distance function.
Revised: where g is a gap parameter measuring the distance between ( v fx , v + ) and ( v fx , v − ) , and dist ( · , · ) is a distance function.

Original: This hinge loss L trip encourages fewer violations to “positive > negative" ranking order.
Revised: This hinge loss L trip encourages fewer violations to “positive > negative" ranking order.

Original: Let τ = 1 / 2 .
Revised: Let τ = 1 / 2 .

Original: With the same triplet, we can write down a contrastive loss
Revised: With the same triplet, we can write down a contrastive loss

Original: [Equation]
Revised: [Equation]

Original: Note that in the second to the last equation, v + and v − have the same norm due to the normalization in our contrastive learning module.
Revised: Note that in the second to the last equation, v + and v − have the same norm due to the normalization in our contrastive learning module.

Original: By setting dist ( · , · ) to commonly used (cid:96) 2 distance and g = 1 , Eq. 9 is a fair approximation of Eq. 8.
Revised: By setting dist ( · , · ) to commonly used (cid:96) 2 distance and g = 1 , Eq. 9 is a fair approximation of Eq. 8.

Original: Therefore, triplet loss can be viewed as a special case of our contrastive loss.
Revised: Therefore, triplet loss can be viewed as a special case of our contrastive loss.

Original: But in contrastive loss, embeddings are normalized and more positives/negatives are available.
Revised: But in contrastive loss, embeddings are normalized and more positives/negatives are available.

Original: As shown in [23], contrastive loss generally outperforms triplet loss.
Revised: As shown in [23], contrastive loss generally outperforms triplet loss.

Original: A.2 Gradients of Contrastive Loss
Revised: A.2 Gradients of Contrastive Loss

Original: Recall our contrastive loss:
Revised: Recall our contrastive loss:

Original: [Equation]
Revised: [Equation]

Original: For the illustration purpose, we only consider one sample ( x, y ) instead of one batch:
Revised: For the illustration purpose, we only consider one sample ( x, y ) instead of one batch:

Original: [Equation]
Revised: [Equation]

Original: Deﬁne N ( y ) ≡ A \ P ( y ) .
Revised: Deﬁne N ( y ) ≡ A \ P ( y ) .

Original: We now derive the gradients w.r.t.
Revised: We now derive the gradients w.r.t.

Original: v fx .
Revised: v fx .

Original: [Equation]
Revised: [Equation]

Original: Further
Revised: Further

Original: [Equation]
Revised: [Equation]

Original: we have the unnormalized feature embedding w fx , v fx = w fx || w fx || .
Revised: we have the unnormalized feature embedding w fx , v fx = w fx || w fx || .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where I is an E × E identity matrix.
Revised: where I is an E × E identity matrix.

Original: The gradient of L CL w.r.t.
Revised: The gradient of L CL w.r.t.

Original: w fx can be derived using chain rule,
Revised: w fx can be derived using chain rule,

Original: [Equation]
Revised: [Equation]

Original: We can then observe that if v fx and v lt are orthogonal ( v fx v lt → 0 ), || v lt − ( v fx v lt ) v fx || will be close to 1 and the gradients would be large.
Revised: We can then observe that if v fx and v lt are orthogonal ( v fx v lt → 0 ), || v lt − ( v fx v lt ) v fx || will be close to 1 and the gradients would be large.

Original: Otherwise, for weak positives or negatives ( | v fx v lt | → 1 ), the gradients would be small.
Revised: Otherwise, for weak positives or negatives ( | v fx v lt | → 1 ), the gradients would be small.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B Supplementary Experimental Results B.1 Implementation Details
Revised: B Supplementary Experimental Results B.1 Implementation Details

Original: We use one Tesla V100 GPU on CentOS for every experiment.
Revised: We use one Tesla V100 GPU on CentOS for every experiment.

Original: The batch size is set to 128.
Revised: The batch size is set to 128.

Original: The latent dimensionality is 64.
Revised: The latent dimensionality is 64.

Original: The feature encoder is an MLP with 3 hidden layers of sizes [256, 512, 256].
Revised: The feature encoder is an MLP with 3 hidden layers of sizes [256, 512, 256].

Original: The label encoder has 2 hidden layers of sizes [512, 256].
Revised: The label encoder has 2 hidden layers of sizes [512, 256].

Original: The decoder contains 2 hidden layers of sizes [512, 512].
Revised: The decoder contains 2 hidden layers of sizes [512, 512].

Original: On reuters and bookmarks , we add one more hidden layer with 512 units to the decoder.
Revised: On reuters and bookmarks , we add one more hidden layer with 512 units to the decoder.

Original: The embedding size E is 2048 (tuned within the range [512, 1024, 2048, 3072]).
Revised: The embedding size E is 2048 (tuned within the range [512, 1024, 2048, 3072]).

Original: We set α = 1 (tuned within [0.1, 0.5, 1, 1.5, 2]), β = 0 .
Revised: We set α = 1 (tuned within [0.1, 0.5, 1, 1.5, 2]), β = 0 .

Original: 5 (tuned within [0.1, 0.5, 1, 1.5, 2.0]) for most runs.
Revised: 5 (tuned within [0.1, 0.5, 1, 1.5, 2.0]) for most runs.

Original: We tune learning rates from 0.0001 to 0.004 with interval 0.0002, dropout ratio from [0.3, 0.5, 0.7], and weight decay from [0, 0.01, 0.0001].
Revised: We tune learning rates from 0.0001 to 0.004 with interval 0.0002, dropout ratio from [0.3, 0.5, 0.7], and weight decay from [0, 0.01, 0.0001].

Original: Grid search is adopted for tuning.
Revised: Grid search is adopted for tuning.

Original: The ﬁnal hyper-parameter selections are shown in Tab. 7.
Revised: The ﬁnal hyper-parameter selections are shown in Tab. 7.

Original: Every batch in our experiments requires less than 16GB memory.
Revised: Every batch in our experiments requires less than 16GB memory.

Original: The number of epochs is 100 by default.
Revised: The number of epochs is 100 by default.

Original: B.2 Training on Fewer Data
Revised: B.2 Training on Fewer Data

Original: We provide relative performances of several major state-of-the-art methods including ours to C2AE, on HA, ex-F1, mi-F1, ma-F1 scores.
Revised: We provide relative performances of several major state-of-the-art methods including ours to C2AE, on HA, ex-F1, mi-F1, ma-F1 scores.

Original: All methods are trained on 10% or 50% of the data, including
Revised: All methods are trained on 10% or 50% of the data, including

Original: C2AE.
Revised: C2AE.

Original: The compared results have the same amount of data for training and thus the comparison is fair.
Revised: The compared results have the same amount of data for training and thus the comparison is fair.

Original: Fig.
Revised: Fig.

Original: 4, Fig.
Revised: 4, Fig.

Original: 5, Fig.
Revised: 5, Fig.

Original: 6 show the relative performance of various state-of-the-art methods over C2AE, on mirﬂickr , nus-vec , eBird respectively.
Revised: 6 show the relative performance of various state-of-the-art methods over C2AE, on mirﬂickr , nus-vec , eBird respectively.