ZEST : Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarization (2020)

Abstract: We study the problem of recognizing visual entities from the textual descriptions of their classes. Specifically, given birds’ images with free-text descriptions of their species, we learn to classify images of previously-unseen species based on specie descriptions. This setup has been studied in the vision community under the name zero-shot learning from text, focusing on learning to transfer knowledge about visual aspects of birds from seen classes to previously-unseen ones. Here, we suggest focusing on the textual description and distilling from the description the most relevant information to effectively match visual features to the parts of the text that discuss them. Specifically, (1) we propose to leverage the similarity between species, reflected in the similarity between text descriptions of the species. (2) we derive visual summaries of the texts, i.e., extractive summaries that focus on the visual features that tend to be reflected in images. We propose a simple attention-based model augmented with the similarity and visual summaries components. Our empirical results consistently and significantly outperform the state-of-the-art on the largest benchmarks for text-based zero-shot learning, illustrating the critical importance of texts for zero-shot image-recognition.

## introduction in computer vision, zero shot-learning (zsl) for image classification is the problem of classifying images given auxiliary information. an image classification model is trained to classify images from a pre-defined set of classes. at test time, images from new classes are given, and the task is to transfer knowledge learned from seen classes during training to unseen test classes. a common setup for zsl assumes that the auxiliary information is a set of semantically meaningful properties (called attributes) describing the class (e.g., black-beak, long-tail) @xcite . a different zsl setup uses image captions as auxiliary information @xcite . typically, this auxiliary information is manually collected by human raters for each image (test and train alike) and averaged across images. a more realistic approach relies on available online text descriptions of classes (e.g., wikipedia) @xcite . it avoids ex-pensive annotation and exposure to test images. in this work, we classify bird species according to wikipedia descriptions. this task raises many challenges: (1) differences between the birds are very small, which makes it a fine-grained classification task; (2) this is an expert task, and the text contains terminology that is unlikely to be familiar to a layman; and, on top of that (3) the text descriptions of the classes are long, containing few visually relevant sentences. as opposed to previous work on text-based zsl employing textual descriptions @xcite that focused on the visual modality, here we focus on the text modality, and address a key question in zsl: how can we identify text components that are visual in nature? to get an intuition about the task setup and our proposed solution, consider the following situation. imagine you have never seen a zebra but have seen a horse. what if you were given a text describing a zebra: "zebras have hooves, mane, tail, pointed ears, and white and black stripes". this description would probably be very close to a description of a horse having "hooves, mane, tail, pointed ears" and you would probably be looking for an image that reminds you of a horse but has "white and black stripes". so, even without ever seeing a zebra, using text-descriptions of the zebra and knowledge already acquired about horses, one can correctly classify unknown classes like a zebra. our proposed solution has two-phases. first, based on the intuition that similar objects (or images thereof) tend to have similar texts, we encode a similarity feature that enhances text descriptions' separability. in addition, we leverage the intuition that the differences between text descriptions of species would be their most salient visual features, and extract visually relevant descriptions from the text. our experiments empirically demonstrate both the efficacy and generalization capacity of our proposed solution. on two large zsl datasets, in both the easy and hard scenarios, the similarity method obtains a ratio improvement of up to 18.3%. with the addition of extracting visually relevant descriptions, we obtain a ratio improvement of up to 48.16% over the state-of-the-art. we further show that our visual-summarization method generalizes from the cub dataset @xcite to the nab dataset @xcite , and we demonstrate its contribution to additional models by a ratio improvement of up to 59.62%. the contributions of this paper are threefold. first, to the best of our knowledge, we are the first to showcase the critical importance of the text representation in zero-shot image-recognition scenarios, and we present two concrete text-based processing methods that vastly improve the results. second, we demonstrate the efficacy and generalizability of our proposed methods by applying them to both the zero-shot and generalized zero-shot tasks, outperforming all previously reported results on the cub and nab benchmarks. finally, we show that visual aspects learned from one dataset can be transferred effectively to another dataset without the need to obtain dataset-specific captions. the efficacy of our proposed solution on these benchmarks illustrates that purposefully exposing the visual features in texts is indispensable for tasks that learn to align the vision-and-language modalities. ## background and related work zero-shot learning (zsl) aims at overcoming the need to label massive datasets for new categories, by learning the connections between images and prior auxiliary knowledge about their classes. at test-time, this auxiliary information compensates for the lack of previously-attained visual information about the new categories. text-based zsl is a specific multimodal instantiation of this learning task that uses natural language descriptions as the auxiliary information. models for text-based zsl are typically composed of three parts: (1) the text representation; (2) the image representation; (3) a compatibility function between the two. while most previous work focused mainly on the latter two components, here we focus on the text. most zsl studies for object recognition are aimed at processing the image modality. for example, @xcite ; lei ba et al. (2015); qiao et al. (2016); akata et al. (2016) rely on visual features extracted using convolutional neural network (cnn). more recent studies use object detection to detect the semantic parts of the object and extract visual features at the part-level @xcite @xcite . this approach makes the image more compatible with the text, as it enables text-terms such as "crest" to be linked to the visual representation of parts like "head". the auxiliary information provided to zsl tasks may be of various kinds, ranging from pre-defined semantic attributes @xcite @xcite , to captions @xcite to wikipedia article describing the species @xcite . here we assume the latter scenario. zsl studies that rely on wikipedia articles as auxiliary information improve the visual representation and the compatibility function, and use text representations such as bag-of-words and tf-idf, without further text processing. @xcite @xcite @xcite . @xcite used a simple bow and a l 1,2 -norm objective to suppress the noisy signal in the text. however, this basic treatment of the text is problematic, as it misses crucial information for detecting the correct class. recent studies @xcite have shown improved performance on multiple vision-and-language tasks using pre-trained bert-based models that jointly learn a representation for vision and language. however, they are tuned on relatively short texts and are not optimal for classifying long textual descriptions. in this work, we proceed in a different, yet complementary, direction to previous work, aiming to purposefully model the contribution of the textual modality to zsl. we aim to establish the importance of adequately processing the text into a sound representation of visually salient features, in order to increase the vision-and-language compatibility, which can then be effectively learned in an end-toend manner. ## strong baseline model the basic architecture, which term zest vanilla , is a simple multiplicative attention mechanism @xcite inspired by @xcite . we model the problem using an attention-based model, where the image is queried against a set of candidate documents. formally, let x s 1 , . . . , x s m be image feature vectors from a training-set, where x s @xmath0. the set of m training images corresponds to a set of l seen classes. each class has a single "class description" which is a document written by experts in free language (e.g. wikipedia). we denote d s 1 , . . . , d s l as a set of l document feature vectors, where d s @xmath1. likewise, let x u 1 , . . . , x u n be the image feature vectors from a test set, where x u @xmath2. the set of test images corresponds to a set of k unseen classes. likewise, each class has a single "class description". we denote d u 1 , . . . , d u k as a sets of document feature vectors, where x u @xmath3. finally, @xmath4. at inference, the label assignment of an image x u i is defined as: for an image representation x s i and a text representation d s j , an indicator function i( image encoding the image encoder's goal is to transform the image into a vector representation of the most salient visual features for the classification. we adopt the image encoder for text-based zsl of @xcite ; @xcite ; @xcite . it is based on a fast r-cnn with @xcite a vgg16 backbone for object detection to detect seven semantic parts in the cub dataset: "head","back","belly","breast","leg","wing","tail". each visual part's encoded features are then concatenated into a feature vector that functions as the image representation for the text-based zsl. text processing our basic encoder processes the text into a feature vector. similar to previous studies, we employ a tf-idf representation @xcite . we preprocess the text to tokenize words, remove stop words, and stem the remaining words. then, we extract a feature vector using tf-idf. this processing procedure is similar to the text processing presented by @xcite . the dimensionalities of tf-idf features for cub and nab are 7,551 and 13,217, respectively. ## the proposed approach our solution's key idea is to replace the general class's text representation with a text representation focusing on the most salient features for the visual recognition task. to do so, we employ two different (complementary) methods: (i) induce a similarity measure used for clustering; and (ii) extract visually relevant text descriptions. both methods are incorporated in our proposed end-to-end vision-and-language classification architecture, presented in figure 2. in what follows we describe the similarity component 4.1, and the extraction of visually relevant summaries 4.2. ## conclusion this work aims to establish a better way to represent the language modality in text-based zsl for image classification. our approach only relies on semantic information about visual features, and not on the visual features themselves. specifically, our two orthogonal text-processing methods, employ-ing textual similarity and visually-relevant summaries, lead to significant improvements across models, splits, and datasets, and illustrate that adequate text-processing is essential in text-based zsl tasks. we conjecture that text-processing methods will be essential in a range of vision and languagebased tasks, and hope this work will assist future research in better representing the language modality in various multi-modal tasks.