C ONTRASTIVELY ENFORCING DISTINCTIVENESS FOR MULTI - LABEL CLASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recently, as an effective way of learning latent representations, contrastive learning has been increasingly popular and successful in various domains.
The success of contrastive learning in single-label classiﬁcations motivates us to leverage this learning framework to enhance distinctiveness for better performance in multilabel image classiﬁcation.
In this paper, we show that a direct application of contrastive learning can hardly improve in multi-label cases.
Accordingly, we propose a novel framework for multi-label classiﬁcation with contrastive learning in a fully supervised setting, which learns multiple representations of an image under the context of different labels.
This facilities a simple yet intuitive adaption of contrastive learning into our model to boost its performance in multi-label image classiﬁcation.
Extensive experiments on two benchmark datasets show that the proposed framework achieves state-of-the-art performance in the comparison with the advanced methods in multi-label classiﬁcation.
I NTRODUCTION
Multi-label image classiﬁcation is a fundamental and practical computer vision task, where the goal is to predict a set of labels (e.g., objects or attributes) associated with an input image.
It is an essential component in many applications such as recommendation systems (Jain et al., 2016;
Yang et al., 2015), medical image diagnosis (Ge et al., 2018b), and human attribute recognition (Li et al., 2016b).
Compared to single-label cases, multi-label classiﬁcation is usually more complex and challenging.
Recently, contrastive learning (CL) (Chen et al., 2020;
He et al., 2020; Li et al., 2020; Caron et al., 2020; Bachman et al., 2019) has been shown as an effective pretext approach to learn latent representations an unsupervised way, which can be further used for supervised tasks.
In general, CL aims to pull together an anchor and a similar (or positive) sample in embedding space and push apart the anchor from many dissimilar (or negative) samples.
Therefore, the choice of the positive and negative samples of an anchor is a key to achieving good performance with CL.
In self-supervised
CL (Chen et al., 2020), the positive sample is deﬁned as those augmented from the same image with the anchor, while the negative samples are all the other images in the minibatch.
More recently, supervised CL (Khosla et al., 2020; Sun et al., 2021; Yuan et al., 2021; Huynh, 2021) has been proposed, where all the images with the same label as the anchor are considered as the positive samples and vice versa for the negative ones.
Supervised CL has shown improvements in single-label image classiﬁcations than the self-supervised counterpart.
With the above successful examples, CL has drawn signiﬁcant research attention and has been applied in other tasks including image segmentation (Wang et al., 2021), adversarial training (Kim et al., 2020), and text to image learning (Radford et al., 2021).
Given the appealing properties and promising results of CL in single-label classiﬁcation, it is natural to adapt it into multi-label cases to boost performance.
However, this adaptation is non-trivial.
In single-label cases, an image usually contains one salient object, thus, the label of the object can also be viewed as the unique label of the image.
Therefore, it is reasonable to use one image-level representation of an image and to push the representation of the anchor close to its positive samples (e.g, augmentations of the anchor or images with the same label as the anchor), as done in selfsupervised and supervised CL.
However, with a single image-level representation for an image, it is hard to deﬁne the positive or negative samples for an anchor image by its multiple labels in the multi-label classiﬁcation.
For example, it would not be reasonable to assume that the image-level representations of images containing apples must always be close to each other, as apple is just one of many objects in those images and an apple may only take a small area of an image.
As a result, this setting hinders the application of existing CL methods to multi-label classiﬁcation.
To bridge this gap, we in this paper propose a novel end-to-end framework for multi-label image classiﬁcation that leverages the contrastive learning principle, termed MulCon or Multi-label Classiﬁcation with Contrastive Loss .
Instead of using image-level representations as in previous CL methods, we introduce a new module that learns multiple label-level representations of an image, which are generated with the attention from the globally class-speciﬁc embeddings to the image features learned by a convolutional neural network (CNN).
Each label-level embedding of an image corresponds to the image’s representation in the context of a speciﬁc label.
With these deﬁnitions, the supervised CL loss can be applied.
Speciﬁcally, if we look at one speciﬁc label and view a labellevel embedding of an image as the anchor in our proposed CL framework, it is straightforward to deﬁne the positive samples of an anchor, which are the label-level embeddings of the other images in the minibatch with the same label and vice versa for the negative samples.
For example, instead of sharing an image-level representation with other objects, the apple object of an image has its own embedding and the embeddings of the apples of all the images in a minibatch are pushed close to each other.
Therefore, our framework is intuitive in the multi-label setting.
In this way, the CL loss can enforce the coherence and consistency of the label-level representations of images, which further provides more discriminative power of the prediction procedure based on these representations.
The main contributions can be summarized as follows: 1) Contrastive learning has been shown successful in many single-label classiﬁcation problems, however, it is non-trivial to apply it in multilabel classiﬁcation.
We propose an intuitive and conceptually simple framework that encompasses contrastive learning.
Along with the proposed framework, we also introduce a practical training scheme of contrastive learning for multi-label classiﬁcation.
We conduct extensive experiments on large-scale benchmark datasets, showing that the proposed framework achieves the state-of-theart performance in multi-label image classiﬁcation.
P ROPOSED M ETHOD
In this section, we ﬁrst discuss about the background knowledge and relevant notations and then elaborate on the details of the proposed framework, MulCon.
2. B ACKGROUND AND N OTATIONS
Multi-label Classiﬁcation Following the standard setting of multi-label image classiﬁcation, we denote a minibatch of input images by X ∈ R N × W × H × 3 , where N is the batch size, H and W are the height and width of the images.
Each image x i ∈
X is associated with multiple labels selected from a set of L labels in total, which are denoted by a multi-hot binary vector y i ∈ { 0 , 1 } L .
For an active label j of x i , y ij = 1 and vice versa.
Our task is to build an end-to-end model that takes x i to predict its labels y i .
Attention The Attention mechanism (Luong et al., 2015; Xu et al., 2015) has been widely used in various areas of computer vision and natural language processing, which enhances the important parts of the data of interest and fades out the rest.
Assume that n q query vectors of size d q denoted as Q ∈ R n q × d q , and n v key-value pairs denoted as K ∈ R n v × d q , V ∈ R n v × d v .
The attention function maps the query vectors Q to outputs using the key-value pairs as follows:
[Equation]
where the dot product ( QK T ) ∈ R n q × n v and ω ( · ) is softmax function.
The dot product returns the similarity of each query and key value.
The output ω ( QK T ) V ∈ R n q × d v is the weighted sum over
V , where larger weight corresponds to larger similarity between query and key.
A powerful extension to the above (single-) attention mechanism is the multi-head attention introduced in (Vaswani et al., 2017), which allows the model to jointly attend to information from different representation subspaces at different positions.
Instead of computing a single attention function, this method ﬁrst projects Q, K, V onto h different vectors, respectively.
An attention function Att ( · ) is applied individually to these h projections.
The output is a linear transformation of the concatenation of all attention outputs:
[Equation]
where W o , W qi (cid:48) , W ki (cid:48) , W vi (cid:48) are learnable parameters of some linear layers.
QW qi (cid:
) ∈ R n q × d hq , KW ki (cid:48) ∈
R n v × d hq , V W vi (cid:
) ∈ R n v × d hv are vectors projected from Q, K, V respectively.
d hq = d q /h and d hv =
d v /h .
Following the architecture of the transformer (Vaswani et al., 2017; Lee et al., 2019), we deﬁne the following multi-head attention block:
[Equation]
where W q (cid:48) ∈ R
d q × d q is a learnable linear layer.
Base on the multi-head attention block, we further deﬁne a self-attention block as follows:
[Equation]
Contrastive Learning Contrastive learning (CL) has been an increasingly popular and effective representation learning approach (Chen et al., 2020;
He et al., 2020; Khosla et al., 2020).
As the ﬁrst proposed CL approach, self-supervised CL (Chen et al., 2020) is proposed to learn presentations in an unsupervised manner.
Speciﬁcally, a mini-batch is constructed from the original input images and their augmented versions.
Given a minibatch of 2 N instances I = { 1 ... 2 N } and an anchor instance i ∈ I , the augmented version of i , denoted as i a ∈
I is considered the positive sample, and the other 2( N − 1) instances within the mini-batch are considered negative examples.
The loss function of self-supervised CL is deﬁned as follows:
[Equation]
where z i is the image embedding, z i · z i a denotes the inner dot product between two embeddings, τ ∈ R + is a scalar temperature parameter, and A ( i ) =
I \
z i .
It can be seen that self-supervised CL pushes the embeddings of the samples augmented from the same image close to each other.
More recently, the supervised CL (Khosla et al., 2020) adapts CL into the supervised settings, which utilizes the label information to select positive and negative samples.
For supervised CL, the embeddings of the samples with the same labels are pushed close to each other, which achieves better performance in classiﬁcation tasks.
For more details of CL, we refer to Liu et al.
(2021b).
To our knowledge, CL has not been applied in solving multi-label classiﬁcation yet.
2. P ROPOSED M ETHOD
Unlike previous CL methods for single-label classiﬁcation that use image-level representations, we propose to learn multiple label-level representations for each image, which facilitates the application of CL in multi-label classiﬁcation.
We introduce MulCon, which consists of three main neural network modules: the label-level embedding network, the contrastive learning projection network, and the classiﬁcation networks as shown in Figure 1.
Label-Level Embedding Network
As the key module of our framework, the label-level embedding network takes an image
x
i as input and outputs its label-level representations, denoted by
g i ∈
R L × D , where each row of g i corresponds to the embedding of the image under the context of a speciﬁc label.
Speciﬁcally, the label-level embedding network consists of two components: 1)
The encoder block.
We ﬁrst adopt an encoder network that learns the image-level embedding as the backbone model as the backbone model: r i =
Enc ( x i ) ∈ R C × H × W , where C, H, W are the number of channels, height, and width of the output.
The backbone model can be implemented with an arbitrary model that learns good image features, e.g., a convolutional neural network (CNN) such as ResNet (He et al., 2016) or visual transformer (Dosovitskiy et al., 2020).
We then reshape r i ∈ R WH × C for the next step’s processing.
The label-wise attention block.
In this block, we introduce a set of vectors U ∈ R L × C , each row of which is to be learned as the global embedding for a speciﬁc label.
To generate an image’s label-level embeddings g i , we propose a cascade of several self-attention blocks and a multi-head attention block to capture the interactions between r i and U as follows:
[Equation]
To summarise, given an image x
i , we ﬁrst learn its representation r i ∈ R WH × C by the encoder, each row of which captures the feature of a location in the image.
We then pass r i to a self-attention block that captures the interactions between the features of the image.
Next, we feed r i to the multihead attention block, where the global label embeddings U is the query and r i is the key and value.
As the sizes of the input and output channels of the multi-head attention are C and D , respectively, we have n q = n v = C and d q = n q = D for Eq. 2.
With the multi-head attention block, our model can learn the “importance” (attention weight) of the image feature to a speciﬁc label.
For example, if an image contains an apple, the corresponding embedding is expected to be associated with a large attention weight from the apple label.
The multi-headed attention can also help a label pay its attention to multiple objects in an image, i.e., each of the attention heads can generate attention scores for a class-speciﬁc embedding over all the image-level embeddings.
In this case, if an image consists of multiple apples, each of the apples receives a speciﬁc attention score from the apple label.
Following up the multi-head attention is a self-attention block that implicitly helps improve label correlation.
With the attention from all the labels, we can derive the label-level embeddings g i of the image from its image-label embedding r i .
Contrastive Learning Projection Network
After obtaining g i ∈
R L × D , we use g ij ∈ R D to denote the label-level embedding of the input image i under the context of a speciﬁc label j ( j ∈ { 1 , · · · , L } ).
Following Chen et al.
Khosla et al.
(2020), our framework includes a projection network Proj ( · ) that maps g ij to a vector in another embedding space: z ij =
Proj ( g ij ) ∈ R d z , where the contrastive learning is performed.
Classiﬁcation Network Recall that the label-level embedding g ij captures the input image i ’s feature under the context of the label j .
Thus, it can be used to predict whether j is active in i .
Accordingly, we introduce a fully connected layer as a classiﬁer f jc to predict the probability of the label j being active.
Speciﬁcally, for each label j ∈ L the prediction score is s ij = σ ( f j c ( g ij )) ∈
We further denote s i ∈ (0 , 1) L .
2. L EARNING M UL C ON WITH C ONTRASTIVE L OSS
After introducing the framework, we describe the learning process of MulCon by showing the loss function ﬁrst.
Based on the label-level embeddings of the image i , i.e., g i , we introduce a loss function with two terms: the classiﬁcation loss and contrastive loss.
The overall training process is illustrated in Figure 2.
Classiﬁcation Loss For the classiﬁcation loss, given the predictive probabilities output from the classiﬁcation network s i and the ground-truth multi-hot label vector
y i , we apply the binary crossentropy (BCE) loss, which has been widely used for multi-label classiﬁcation:
[Equation]
It is noteworthy that other multi-label classiﬁcation losses than BCE can also be used in our framework, e.g., in Ben-Baruch et al.
Lin et al.
Label-level Contrastive Loss
In addition to the classiﬁcation loss, we introduce the label-level contrastive loss (LLCL) for multi-label classiﬁcation, which is one of the key contributions of this paper.
Because an image is associated with multiple labels, it is hard to directly apply CL as we analyzed before.
However, in MulCon, after learning the label-level embeddings for an image, we show that the multi-label problem can be transformed into a single-label one, where CL can be adapted in a straightforward way.
As LLCL works in the projected space, hereafter, we also call z i the label-level embeddings for image i , which is projected from g i .
Given a minibatch of N images, we ﬁrst forward-pass them through the label-level embedding network and contrastive learning projection network and then aggregate the label-level embeddings of all the images into set Z = { z ij ∈ R d z | i ∈ { 1 , · · · , N } ; j ∈ { 1 , · · · , L }} .
Similarly, we deﬁne the set of the ground-truth labels of the minibatch: Y = { y ij ∈ { 0 , 1 }| i ∈ { 1 , · · · , N } ; j ∈ { 1 , · · · , L }} .
If we view an image’s label-level embedding z ij as an instance instead of the image itself, z ij is associated with a single ground-truth label y ij .
We further deﬁne I = { z ij ∈ Z | y ij = 1 } as the set that contains the label-level embeddings with active ground-truth labels and A
( i, j ) =
I \ z ij as the set contains the embeddings in I with z ij excluded.
In the minibatch, we now consider z ij ∈ I as the anchor, which presents the feature of image i under active label j .
With LLCL, we aim to push z ij closer to the embeddings under the same active label j of other images in the minibatch, i.e., the positive set, deﬁned as
P ( i, j ) =
{ z kj ∈ A ( i, j ) | y kj =
y ij = 1 } .
With above notations and inspired by the supervised
CL, we deﬁne the contrastive loss for the anchor z ij as:
[Equation]
The loss for the whole minibatch is: L LLCL =
(cid:80) z ij ∈ I L ijLLCL .
Together with the classiﬁcation loss, we show the overall training loss of MulCon:
[Equation]
where the parameter γ controls the trade-off between the two losses.
Why and How does Contrastive Loss Help?
Now we would like to answer why contrastive loss helps in multi-label classiﬁcation, by showing that it serves as an important complementary to the classiﬁcation loss.
Speciﬁcally, with the BCE classiﬁcation loss applied in many multi-label classiﬁcation problems, each label can be viewed to be classiﬁed independently with a speciﬁc classiﬁer as discussed in Section 2.2.
That is to say, each classiﬁer focuses on the classiﬁcation of a speciﬁc label and cares less about the distinctiveness of the features of different labels.
Distinctiveness in classiﬁcation means that we expect features of the instances with the same label to be close to each other, which has been known as an important factor for achieving good classiﬁcation accuracy.
Similar to single-label problems, the label-level contrastive loss in MulCon is designed to enforce distinctiveness of the label-level embeddings.
To demonstrate this, we show the t-SNE (Van der
Maaten & Hinton, 2008) visualization of the (active) label-level embeddings of 1,000 randomly sampled images of the COCO dataset (Lin et al., 2014) in Figure 3.
These embeddings are from
MulCon trained by the BCE loss (i.e., Eq. 7) and the combined loss of BCE and LLCL (i.e., Eq. 9), respectively.
Each dot represents one label-level embedding under the context of a speciﬁc label and each color represents one class.
It can be seen that compared with the loss with BCE only, the additional contrastive loss makes the label-level embeddings of the same label fall into more compact clusters, which are better separated from the clusters of other labels.
This clearly shows the enforced distinctiveness of the embeddings with the contrastive loss.
However, unlike in single-label classiﬁcation, being over distinct in the embedding space is not always a good thing in multi-label classiﬁcation.
Speciﬁcally, label correlations are important in multi-label classiﬁcation (Chen et al., 2019c; You et al., 2020).
For example, when we see a cup in an image, the probability of a table existing below the cup in the image can be assumed to high.
In this case, we actually do not want the embeddings of cups and tables are too far or too separated from each other.
Therefore, the enforced distinctiveness with the contrastive loss needs to be judiciously controlled.
We propose a simple training strategy to tackle this task, which consists of two steps: the pre-training step and the contrastive learning step .
In the pre-training step (Step 1), we pretrain the backbone and label-level embedding network and the classiﬁcation network of MulCon with the
BCE loss only.
In the contrastive learning step (Step 2), we then plug in the contrastive projection network with LLCL but also keep the BCE loss.
In the ﬁrst step, the BCE loss learns the labellevel embeddings freely and implicitly obtains semantic structure by the effect of several attention blocks.
After the embeddings are learned, we ﬁnetune them with LLCL to enforce distinctiveness of the embeddings.
The network is ﬁntuned with small learning rate which helps improve feature distinctiveness without breaking the label semantic structure.
We empirically ﬁnd that the propose training policy works well in practice.
R ELATED W ORK ON M ULTI - LABEL I MAGE C LASSIFICATION
Our paper mainly focuses on two things: contrastive learning and multi-label classiﬁcation.
To our knowledge, most of the works in contrastive learning are for single-label classiﬁcation.
Thus, although there is an emerging literature of contrastive learning, the details are omitted as our paper solves a different problem.
Multi-label classiﬁcation has been a challenging and important problem in computer vision, where many methods have been proposed.
We mainly consider two lines of multi-label methods as our close works.
Methods capturing label/class correlations For example, (Wang et al., 2016; Yang et al., 2016;
Liu et al., 2017; Chen et al., 2018a) explicitly capture the class correlation by a CNN-based model followed by a Recurrent Neural Network (RNN) (Medsker & Jain, 2001).
Moreover, there are also approaches based on on probabilistic graphical model (Li et al., 2016a; 2014), which model label dependencies in the covariance of probabilistic distributions.
Finally, as label correlations can be formulated into graphs, some approaches apply Graph Convolutional Network (GCN)
(Zhou et al., 2020) or leverage graph structures to learn label presentations such as in Yang et al.
; Chen et al. (2019c); You et al.
(2020).(Li et al., 2019) proposes to align image feature with label feature.
Compared with these methods, ours is a different approach that controls label distinctiveness and correlation via contrastive loss.
Methods with attentions Another research line close to ours is using visual attention for multilabel classiﬁcation, such as in Guo et al.
; Chen et al.
(2018b;a);
Wang et al.
; Ba et al.
(2015); Huynh & Elhamifar (2020)
; Yazici et al.
In addition, attention mechanisms are also introduced in Chen et al. (2019a); Ye et al.
(2020) to capture label correlation.
Recently, vision transformers (Dosovitskiy et al., 2020) has been a new paradigm of computer vision tasks and there emerges new works that use vision transformers for multi-label classiﬁcation (Liu et al., 2021a;
Lanchantin et al., 2021; Cheng et al., 2021), where attention mechanisms are naturally used.
In our method, we use the attention mechanism for a different purpose, i.e., to assist contrastive learning.
Speciﬁcally, the attention is used to derive an image’s label-level embeddings from its image-level embeddings and the global-label embeddings.
Moreover, our framework is ﬂexible to also use vision transformers as the backbone model.
E XPERIMENTS
In this section, we compare our proposed model with the state-of-the-art multi-label classiﬁcation methods.
More results are shown in the appendix.
we use the two most popular benchmark datasets:
MS-COCO (Lin et al., 2014) and NUS-WIDE (Chua et al., 2009).
To evaluate the classiﬁcation performance, we use the standard metrics for multi-label classiﬁcation including the mean average precision (mAP) mean average precision (mAP), overall precision (OP), recall (OR), F1-measure (OF1) and per-category precision (CP), recall (CR), and F1-measure (CF1), computed over all prediction scores and top-3 highest prediction scores.
As F1 is a comprehensive metric computed by precision and recall, we consider mAP, CF1, and OF1 as more important metrics.
The formulas of the metrics are shown in the appendix.
For a fair comparison, we adopt ResNet-101 (He et al., 2016) as the backbone for our method to extract image-level features, which is the same for the other compared methods, unless otherwise speciﬁed.
The implementation and training details of our methods are shown in the appendix.
4. E XPERIMENTS ON MS-COCO
In the standard setting for multi-label classiﬁcation, MS-COCO contains 122,218 images withdifferent categories, where every image is associated with 2.9 labels on the average.
The dataset is divided into 82,081 images for training and 40,137 images for validation.
For a fair comparison, we choose state-of-the-art methods that use Resnet101 as backbone network, and further divide them into two groups.
Group 1 uses input image size 448 × 448 such as Multi Evidence (Ge et al., 2018a),
CADM (Chen et al., 2019b), ML-GCN (Chen et al., 2019c), KSSNet (Liu et al., 2018), MS-CMA (You et al., 2020) and MCAR (Gao & Zhou, 2021).
Group 2 uses larger input image size such as
SSGRL (Chen et al., 2019a), C-Trans (Lanchantin et al., 2021) and ADD-GCN
(Ye et al., 2020).
The results on MS-COCO are reported in Table 1.
The numbers of the compared methods are taken from the best reported results in their papers.
It can be seen that our approach achieves the stateof-the-art results especially on mAP, CF1 and OF1.
For example, in the 448 × 448 input resolution with all prediction scores, ours outperforms the second best (MS-CMA and MCAR) by 1.1% on mAP and 0.8% on CF1.
With the resolution increased to 576 × 576 , ours surpasses the second best (ADD-GCN) by 1.1% on mAP and 0.7% on CF1.
Although our method may not always achieve better results on precision (CR and OR), it consistently obtains better performance on F1, which is a more comprehensive metric covering both precision and recall. [Table] [Table] [Table]
4. R ESULTS ON NUS-WIDE
The NUS-WIDE dataset originally contained 269,648 images from Flicker and has been manually annotated with 81 visual concepts.
Since some URLs have been deleted, we follow (Ben-Baruch et al., 2020) to obtain 200,000 images with 2.4 labels per image on average.
We use the same hyperparameter setting as MS-COCO for training and testing on this dataset.
We also select the state-of-the-art methods that report the results on NUS-WIDE, including FitsNet (Romero et al., 2014), attention-transfer (Zagoruyko & Komodakis, 2016), s-CLs (Liu et al., 2018),
CMA
(You et al., 2020) and SRN (Zhu et al., 2017).
Following the standard and convention of many other works, we report mAP, CF1, OF1 computed with all prediction scores.
The results on NUS-WIDE are shown in Table 2.
It can be seen that MulCon achieves the best results in all the three metrics.
For example, it our performs SRN (the second best on mAP) by a signiﬁcant margin, i.e., nearly 2%.
4. A BLATION S TUDY
To fully understand the modules and training policies of MulCon, we provide a comprehensive ablation study in this section.
Speciﬁcally, we are interested in the following variants of our method.
1) R101 + BCE: The backbone model, Resnet101, trained with the BCE loss (i.e., Eq. 7)
This is a standard multi-label classiﬁcation baseline.
R101 + BCE + SCL
: Resnet101 trained with the
BCE
and supervised-CL (Khosla et al., 2020) (SCL) losses (i.e., Eq. 9).
This is a variant where the supervised CL loss is directly applied on the image-level features.
Note that for this variant, each image is an anchor, and thus we need to deﬁne the positive/negative sets for each anchor.
Following the spirit of (Khosla et al., 2020), we consider other images in the minibatch that have at least one common active label with the anchor as its positive samples and the others as the negative ones.
LLEN + BCE: The label-level embedding network (LLEN) with R as the backbone
trained with BCE, which corresponds the MulCon framework trained in Step 1 described in Section 2.3. 4)
LLEN + BCE + LLCL:
The variant where we train MulCon with the BCE and LLCL losses from the beginning, i.e., the second contrastive learning step described in Section 2.3. 5) MulCon: The complete model of trained with the policy described in Section 2.3.
Table 3 shows the results for the ablation study for our method in MS-COCO.
We have the following remarks: 1) By comparing between R101+BCE and R101+BCE+SCL, we can see that unlike in single-label classiﬁcation, directly applying SCL on the image-level features cannot improve the performance in multi-label cases (80.8 vs 80.8 mAP).
It can be observed that by using the proposed label-level embedding network instead of R101 (R101+BCE V.S. LLEN+BCE), we can can signiﬁcantly improve the mAP score (80.8 vs 83.8 mAP).
3) Note that the difference between LLEN + BCE + LLCL and MulCon is on the training policies: The former is trained with BCE and LLCL from the beginning while the later is ﬁrst pretrained with BCE (Step 1) and then futhur trained by
BCE + LLCL (Step 2).
We observe that if the LLCL loss is applied from the beginning of the training, it can barely improve the performance.
This is because LLCL may excessively enforce distinctiveness which makes the model care less about label correlations, as analyzed in Section 2.3.
It can be seen that our proposed two-step training policy improves signiﬁcantly.
4. Q UALITATIVE A NALYSIS
To qualitatively study our method, we ﬁrst exam the semantics captured by the label-level embeddings learned with MulCon by conducting an image retrieval experiment.
Speciﬁcally, given an input image x
i , we do a forward-pass to get its label-level embeddings
g i .
With the ground-truth labels of x i , we can specify a speciﬁc active label j , pick its embedding g ij from g i and use it as the query for image retrieval.
By comparing the Euclidean distance between the query embedding with the label-level embeddings of other images, we can retrieve the closest images to our query image and label.
Figure 4 shows the image retrieval results of MulCon and its variant with BCE only.
More qualitative analysis is provided in the appendix.
apple apple apple apple apple tie teddy bear book dining table book book book book book book book dining table book tie apple tie tie tie teddy bear teddy bear book book teddy bear teddy bear teddy bear teddy bear teddy bear tie teddy bear cat bed Query Model with contrastive loss Model without contrastive loss
C ONCLUSION
In this paper, we have introduced an end-to-end multi-label image classiﬁcation framework, MulCon, which leverages contrastive learning in multi-label classiﬁcation.
It has been shown that CL is not directly applicable in this domain, due to the fact that it is hard to deﬁne the positive/negative samples for an anchor with multiple labels.
To tackle this issue, we have introduced to learn labellevel embeddings for an image with the multi-head attention mechanism.
With label-level embeddings, we transform the multi-label classiﬁcation into a single case for each label-level embedding, which facilities a straightforward adaption of supervised CL.
We have provided analytical and empirical study of why CL helps in multi-label learning and proposed a simple training policy to control the distinctiveness enforced by CL.
Extensive experimental results and visualization show the effectiveness of our approach and its ability to achieve the state-of-the-art performance in multi-label image classiﬁcation.
A A PPENDIX
In this section, we provide more details on the following subjects:
• Implementation Details of MulCon •
Evaluation Metrics
• More Qualitative Analysis
A. I MPLEMENTATION D ETAILS OF M UL C ON
Projector Following other contrastive learning methods (Chen et al., 2020; Khosla et al., 2020), the projector Proj ( · ) has two linear layers with ReLU ( · ) activation.
Step 1:
Pretraining with BCE We use Adam optimizer (Kingma & Ba, 2015) and 1-cycle policy with the maximum learning rate is 2e-4.
The training batch size is set to 64, and the dimension of the label-level feature, i.e., D , is 1024.
Step 2:
Finetuning with LLCL We use Stochastic Gradient Descent (SGD) optimizer with momentum 0.9 and weight decay is 1e-4.
The learning rate is initialized as 0.01 and then reduced by a factor of 10 for every 20 epochs.
The batch size for this step is 32.
The temperature τ is 0.2, and γ is 0.1.
The number of head in multi-head attention is empirically set to 4, which gives us the best results.
Following Ben-Baruch et al.
, we also employ Cutout (DeVries & Taylor, 2017) with factor 0.5 as reguralization.
During training, exponential moving average (EMA) is applied to model’s parameters with decay factor 0.9997.
Augmentations in Step 2 It has been known that augmentations are important for contrastive learning.
Following previous methods (Chen et al., 2020; Khosla et al., 2020), we have an augmentation module to transform each training image before feeding them to the network.
Speciﬁcally, for the augmentation, we ﬁrst resize the input image to 448 × 448 , and then apply random horizontal ﬂip, and a Random Augmentation module (Cubuk et al., 2020).
Note that for the second step training with batch size 32, the actual batch size becomes 64 after data augmentation.
Note that we apply contrastive loss on label-level embeddings, not on images.
For a batch size ofimages, on average we have 94 label-level embeddings (with the active labels for each image).
After augmentation,the batch size of the label-level embeddings is 188 already, which is the maximum batch size that we can run with our computational resource.
We will consider larger batch size in future.
A. E VALUATION M ETRICS
Beyond mean average precision (mAP), the standard metrics reported in the experimental section are: overall precision (OP), recall (OR), F1-measure (OF1) and per-category precision (CP), recall (CR), F1-measure (CF1).
These metrics are computed as follows:
[Equation]
where TP i is true positive of class i , FP i is false positive of class
i , FN i is false negative of class i .
A. M ORE RESULTS ON VOC AND VG Experiments
with VOC 2007 . We compare our method with RDAL(Wang et al., 2017), RARL(Chen et al., 2018b), MCAR(You et al., 2020), SSGRL(Chen et al., 2019a) and ASL (Ben-Baruch et al., 2020) on
VOC2007(Everingham et al., 2015) dataset.
Following other works, we also use pretrained backbone on COCO and ﬁne-tune on VOC 2007.
Overall, our method has a comparable results with other methods.
We also try to use TResnet and ASL loss combine with our Contrastive framework, but we do not observe any improvement.
Figure
[Table]
Experiments with VG 500 .
We compare the performance on VG 500 (Krishna et al., 2017) with other methods such as R101(He et al., 2016), R101-SRN(Zhu et al., 2017), SSGRL(Chen et al., 2019a) and C-Trans(Lanchantin et al., 2021) in Table 5.
Our method also achieve a better results compare to other methods.
[Table]
A. C LASS EMBEDDING RETRIEVAL
We compute the precision@k (P@k) and recall@k (R@k) of class embedding retrieval between our method and Q2L (Liu et al., 2021a) in Table 6.
We use the ﬁrst 1000 images in MS-COCO for this experiment.
We extract all the class embedding to create the set X ∈ R N × d , N is the total number of embeddings and d is the embedding’s dimension.
Then for an embedding x
i , we construct a groundtruth label vector
y i such that y ij = 1 if x i and x j has the same label, y ij = 0 otherwise.
Then we have the ground truth set Y ∈ R N × N .
Given X , we can compute the pair-wise cosine similarity C ∈ R N × N , c ij = cossim
( x i , x j ) .
Given C and Y , the P@k and R@k is computed as follows:
[Equation]
where TP @ K, FP @
k, FN @
k are true positive, false positive and false negative at top k predictions.
Method
[Equation]
P@
P@
P@
P@
R@
R@
[Table]
R@
R@
R@
MulCon
[Table]
[Table]
[Table]
[Table]
Q2L
[Table]
[Table]
[Table] [Table] A. M ORE Q UALITATIVE A NALYSIS Multi-label Image Retrieval To further demonstrate the intuitive meanings of the learned labellevel embeddings in the multi-label setting, we provide an additional retrieval experiment where instead of using a single query label, we select multiple ones and concatenate their label-level embeddings as the query vector.
That is to say, given an image, we can retrieve the images with multi-label labels of interest.
Figure 5 shows the multi-label retrieval results.
Visualisation of Attention Maps We now show a visualization for the attention maps in terms of the active labels of an image produced by the multi-head attention used in the label-level embedding network.
As illustrated in the top row of Figure 6, the attention maps can precisely highlight the regions of the image in terms of each of its ground-truth labels.
In the bottom row of Figure 6, we show the attention maps of the heads of one label “person”.
It can be observed that the heads tend to capture the multiple instances of “person” in the image.
Figure 7 presents more attention maps visualization.
Here we visualize the mean attention map computed from 4 heads.
Analysis on the Global Label Embeddings A row of U is the embedding vector of a speciﬁc label, encoding the semantic information of the label.
Although randomly initialised, the embedding vectors are learned by minimising the loss function via gradient backpropogation.
In Table 7, we have computed the pairwise Euclidean distances between label embeddings and the closest 3 labels retrieved with minimum distances.
It can be seen that the label correlations are quite meaningful.