Refinement Matters: Textual Description Needs to be Refined for Zero-shot Learning (2022)

Abstract: Zero-Shot Learning (ZSL) has shown great promise at the intersection of vision and language, and generative methods for ZSL are predominant owing to their efficiency. Moreover, textual description or attribute plays a critical role in transferring knowledge from the seen to unseen classes in ZSL. Such generative approaches for ZSL are very costly to train and require the class description of the unseen classes during training. In this work, we propose a non-generative gating-based attribute refinement network for ZSL, which achieves similar accuracies to generative methods of ZSL, at a much lower computational cost. The refined attributes are mapped into the visual domain through an attribute embedder, and the whole network is guided by the circle loss and the well-known softmax cross-entropy loss to obtain a robust class embedding. We refer to our approach as Circle loss guided gating-based Attribute-Refinement Network (CARNet). We perform extensive experiments on the five benchmark datasets over the various challenging scenarios viz., Generalized ZSL (GZSL), Continual GZSL (CGZSL), and conventional ZSL. We observe that the CARNet significantly outperforms recent non-generative ZSL methods and most generative ZSL methods in all three settings by a significant margin. Our extensive ablation study disentangles the performance of various components and justifies their importance. The source code is available at https://github.com/Sethup123/CARNet.

## introduction humans can recognize samples from unseen classes by leveraging the visual information of seen categories and textual descriptions of seen and unseen classes @xcite @xcite . zero-shot learning, inspired by this recognition ability of humans, * these authors contributed equally to this work 1 the source code is available at https://github.com/ sethup123/carnet figure 1 : it can be seen that the attributes "small bird" and "long black beak" (words in bold) are common in all three species of the hummingbird. however, attributes like "ruby red throat" or "orange bird" (words in red) distinguish one species from another. hence the distinguishing attributes must be given more weight than the common attributes in the class attribute vector. we achieve this through the gating unit in the attribute refinement network. the generative model has recently been the most popular approach for zsl. it uses generators like vae @xcite or gan @xcite to generate synthetic samples for unseen classes using the class attribute vector. however, despite their promising results, such methods are not very efficient due to the following reasons: (i) the method requires the knowledge of the number of unseen classes and respective attribute vectors during training which is not always feasible, and (ii) retraining of the classifier with seen and unseen samples, with each new unseen classes. on the other hand, non-generative approaches for zsl alleviates the above problems but shows inferior accuracies. typically, the non-generative models learn mapping in three ways: (i) visual to attribute space @xcite or (ii) attribute to visual space @xcite , or (iii) joint embedding of attribute and visual space @xcite . it is to be noted that most of the existing non-generative or embedding-based zsl approaches are formulated to learn embedding from visual to attribute space. they assume that the seen and unseen classes share the same representational characteristics and are linked in the attribute space @xcite @xcite . however, this approach leads to the wellknown hubness problem @xcite , where the representations are skewed towards seen classes @xcite . another problem with this approach is that it implicitly loses the discriminative power of visual features that are generally extracted from a powerful pre-trained deep learning model (like resnet @xcite and googlenet @xcite ) but are then mapped to a different smaller attribute space @xcite . although the aforementioned issues in the nongenerative model are mitigated by mapping the attribute to visual features @xcite @xcite , they have lower classification accuracies. in this work, we propose a non-generative method with an attribute-refinement network (arn) that leverages the gating mechanism. the arn enables highly robust representation of the description/attribute vector for the seen and unseen classes. in recent years, the gating mechanism has shown good performance without any complex architecture in supervised learning tasks @xcite @xcite @xcite . in this paper, we propose a gating mechanism for refining the textual description in the zsl task. the arn learns to refine the attribute in a self-weighing manner from the seen class attribute (fig. 1 ). these refined attributes are mapped onto the visual space using an attribute embedder (ae) to obtain the class prototype vector of each class. the class prototype vector is then combined with the visual features in the feature-prototype combiner (fpc) to obtain classifications. the arn, ae, and fpc are trained jointly using the circle loss and standard softmax cross-entropy in such a way that it minimizes inter-class and maximizes intra-class similarity. the circle loss achieves better within-class compactness and between-class discrepancy compared to triplet loss and adaptive margin softmax loss, as it unifies both classification and pair-wise similarity representation objectives @xcite . we refer to our approach as circle loss guided gating-based attribute-refinement network (carnet). we evaluate the performance of carnet in three scenarios: (a) conventional zero-shot learning (zsl), where only the unseen classes are available during inference (b) generalized zero-shot learning (gzsl), where both the seen and unseen classes are available during inference and (c) continual generalized zero-shot learning (cgzsl), where data arrives as a sequence of tasks and only current task data is available during training, with the challenge of handling catastrophic forgetting of the past tasks. the performance of carnet for conventional zsl and gzsl is evaluated on five standard datasets. the cgzsl method is evaluated for the challenging cub and sun datasets. the extensive experiment shows that carnet outperforms the recent generative (unlike the generative model, we do not require the unseen class description during training) and the non-generative model by a significant margin. our ablation study emphasizes the significance of each component of the proposed learning algorithm. the main contributions of our work are summarised as follows: (i) we propose a gating-based attribute-refinement network (arn) to enhance the class description/attribute for zero-shot learning. (ii) the arn and ae are guided by circle loss to achieve better withinclass compactness and between-class discrepancy. (iii) we propose a highly competitive, simple, and fast non-generative method. our model achieves ∼ 70× speedup compared to generative zsl methods. ## related work the proposed carnet is evaluated for the three kinds of zsl settings: conventional zsl, gzsl, and cgzsl. we provide a brief survey on all these three settings. zsl aims to construct the recognition model for the samples from unseen classes using the textual description (i.e., attribute information) of unseen classes. these attribute information can be obtained through various ways, like humanannotated attributes @xcite , textual descriptions @xcite , and word vectors @xcite . in recent years, there has been a surge of interest in this area. the whole literature of zsl can be broadly categorized into two parts: generative and non-generative (i.e., embedding-based) approaches. the first popular category is the generative approach, which solves the zsl problem by synthesizing the unseen class samples. to synthesize the samples from the unseen classes, models leverage on a powerful generative models like conditional variational autoencoder (vae) @xcite or generative adversarial network (gan) @xcite @xcite @xcite or a combination of vae and gan @xcite . another popular category is the non-generative approach, and it does not need class attribute information of unseen classes during training. in the early zsl work @xcite @xcite , models directly predict the attribute confidence from images. methods based on this approach can be further divided into three groups. in the first group, we project visual feature into attribute (i.e., semantic) space @xcite @xcite @xcite . in the second group, both visual and attribute data are projected into intermediate space @xcite @xcite @xcite . in the third group, visual space is spanned by attribute to visual mapping @xcite @xcite . zsl methods developed based on the projection from attribute space to visual space approach are more suitable for mitigating the hubness problem, and recent works @xcite @xcite show promising results for the zsl and gzsl setting. surprisingly, despite the fast, accurate, and realistic setting, this approach has not been explored much in the past. in this work, we consider the non-generative model for further exploration and learn the mapping from the attribute space to the visual space similar to @xcite . the above-discussed zsl methods can handle data only in an offline setting, and cannot be used in a setting with a streaming sequence of tasks @xcite , known as continual gzsl (cgzsl). only a handful of research is available for cgzsl @xcite @xcite @xcite . for the extensive evaluation, apart from the conventional zsl and gzsl, carnet is also evaluated on the cgzsl setting as proposed in @xcite . ## problem definition in this section, we define the problem formally and introduce the notations. the objective of zsl is to learn a model that can generalize the novel classes (i.e., unseen classes) with the help of side information (attribute/descriptions) without training data for the novel classes. the attribute vector of each class is constructed by either using a word embedding vector generated from a language model or manually defining the key features like color, size, shape, pattern, etc. primarily, the zsl setting consists of two sets of classes known as seen and unseen classes. let d s tr and d ts be the training and testing data, respectively, for the c s seen and c u unseen classes. we also have set of seen ({c s }) and unseen ({c u }) classes where {c s } ∩ {c u } = ϕ i.e. seen and unseen classes set are disjoint. it is to be noted that {c s } and {c u } denote the set of seen and unseen classes, while c s and c u denote the number of seen classes and unseen classes, respectively. corresponding to each seen class i (@xmath0, there is a d-dimensional class attribute vector, i.e., a s @xmath1, respectively. in the zsl training, data is represented as: ## proposed method zero-shot learning (zsl) aims at classification in the absence of input images for unseen classes, using textual description, namely attribute vectors. in this section, we propose the carnet for zero-shot learning. carnet refines the class attribute vectors for better representation with the help of end-to-end joint loss. here, a gating-based attribute refinement network (arn) is used to refine the class attribute vectors. these refined class attribute vectors are mapped through an attribute embedder (ae) to extract an efficient class prototype vector corresponding to each class. the visual features are extracted using a pre-trained resnet-101 model. these visual features are then combined with the class prototype vectors in the feature-prototype combiner (fpc) for classification, as shown in fig. 2 . the arn, the ae, and the fpc are trained end-to-end based on the sum of two losses, namely, the circle loss and the softmax cross-entropy loss. these two losses guide the gating unit to yield refined attributes, which lead to better class prototype vectors through ae. in this section, we present a detailed description of our model carnet. ## experiments we conduct extensive experiments on five benchmark zsl datasets (description given in ## conclusion in this work, we developed the circle loss guided gating-based attribute-refinement network for handling zsl, gzsl, and continual-gzsl tasks. carnet refines the attribute through a gating unit where it improves the attribute representation by learning a self-weight on each attribute dimension in a projected space. these refined attributes improve the embedding, which helps to overcome the model bias towards the seen classes. the whole model is guided by the circle loss along with the standard softmax cross-entropy loss, which maximizes the inter-class separability and intra-class similarity. also, unlike the generative method, carnet does not require the attribute vector of the unseen classes during training. the proposed method is quite fast, as the attribute refinement network and the attribute embedder need to process only the class attribute vectors during training. this work shows that a simple mlp-based architecture can outperform various highly computationally expensive zsl methods. this approach needs to be explored with generative methods and other applications of zsl, like zero-shot for sketch-based image retrieval, action recognition, and natural language processing.