Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptions (2021)

Abstract: We study the impact of using rich and diverse textual descriptions of classes for zero-shot learning (ZSL) on ImageNet. We create a new dataset ImageNet-Wiki that matches each ImageNet class to its corresponding Wikipedia article. We show that merely employing these Wikipedia articles as class descriptions yields much higher ZSL performance than prior works. Even a simple model using this type of auxiliary data outperforms state-of-the-art models that rely on standard features of word embedding encodings of class names. These results highlight the usefulness and importance of textual descriptions for ZSL, as well as the relative importance of auxiliary data type compared to the algorithmic progress. Our experimental results also show that standard zero-shot learning approaches generalize poorly across categories of classes.

## introduction zero-shot learning (zsl) relates information from different modalities (e.g., text or attributes with images), and the hope is that a sparsity of information or training data in one modality can be compensated by the other. this is important when the cost of creation or collection of training data greatly differs between the different modalities. natural language descriptions and existing text repositories can provide this rich and practically accessible information about visual concepts and classes when no accompanying labeled images are available. recent works on zero-shot image classification @xcite @xcite show the field's substantial progress on many of the standard zsl benchmarks. however, most of those benchmarks mostly cover either a very small or narrow set of classes (e.g., only bird species), where human-made class attributes are often used as auxiliary data. unfortunately, on imagenet, where such attributes are not available, the performance is still very low. therefore, in this work, rather than focus on algorithmic development, we instead study how the auxiliary data type impacts performance. we evaluate the benefits of using textual descriptions of the imagenet classes. we collect text from wikipedia articles describing each class in imagenet, illustrated in figure 1 . throughout the paper, we refer to the dataset of imagenet classes and their corresponding wikipedia articles, as well as the extracted text, as imagenet-wiki. 1 the use of textual descriptions for zsl has been studied before on smaller datasets @xcite @xcite . however, this is the first work to use such textual descriptions on a large dataset, with a broad set of classes. the major differences between commonly used datasets are highlighted in our experimental results lead us to two significant observations. first, models using the textual descriptions perform much better than those using class name features. remarkably, a simple zsl model based on devise @xcite and trained on text from imagenet-wiki clearly outperforms more recent, state-of-the-art models that use word embedding features of class names as the auxiliary data. for the cada-vae model @xcite , imagenet-wiki leads to almost a 12 pp. improvement (38.63% vs. 50.50%) in top-5 accuracy on imagenet mp500 test split, which is far higher than achieved in any prior work. these training data for unseen classes or or or or or or auxiliary data auxiliary data auxiliary data auxiliary data auxiliary data auxiliary data figure 1 : imagenet classes and their corresponding auxiliary data: wikipedia articles or class names. above illustrates the difference between class names, the standard source of auxiliary data used on imagenet, and the textual descriptions we collected. the short, selected article fragments shown contain information that is visually discriminative for the classes. in zsl, for seen classes, we have access to both auxiliary data and images. however, for unseen classes only auxiliary data is available for training, with no images. results strongly suggest that algorithmic development on zsl models is not sufficient for further substantial progress. it is necessary to also consider the quality and type of auxiliary data used. fortunately, creating or collecting already available textual information about classes is practical, viable, and much less labor-intensive than collecting and annotating images with labels or attributes. our second main observation is that, regardless of the type of auxiliary data used, zsl models generalize poorly from one large class category to another. for example, excluding all animal classes from the training set leads to large performance drops on animal species in the test set. therefore, the field's algorithmic progress measured on small datasets with few categories ( ## imagenet-wikipedia correspondences imagenet classes correspond to a subset of synsets representing nouns in wordnet 3.0. each class is defined by its wnid (wordnet id), class phrases (which we also refer to as class names in this paper), and a gloss (brief description), for example (n02129165; "lion, king of beasts, panthera leo"; large gregarious predatory feline of africa and india having a tawny coat with a shaggy mane in the male). the synsets have a hierarchical structure, such that our example class is a descendant of classes like "big cat" and "animal". we create an automatic matching of imagenet classes to their corresponding wikipedia pages. 2 the matching is based on the similarity between the synset words and wikipedia titles, as well as their ancestor categories. unfortunately, such matching occasionally produces false-positives, as it is a difficult problem. one reason for this could be that classes with similar names might represent very different concepts -e.g. (n02389026; "sorrel"; a horse of a brownish orange to light brown color) with the herb called "sorrel". to ensure highquality correspondences in imagenet-wiki, we first compared all automatic correspondences from both @xcite and @xcite , then also manually verified all the matches, and modified them if necessary. the quality of the automatic matches often depends more, for example, on how irrelevant articles are filtered out than on the exact matching approach taken. we refer the interested reader to the source code we provide for the full details. ## encoding textual descriptions for zero-shot learning the text for each class is encoded into a single feature vector. having features of fixed dimensionality for each class allows us to use standard zsl approaches. as text encoders we use either albert @xcite , which is an nlp model based on transformer @xcite , or word embeddings models: glove @xcite or word2vec @xcite . for albert we consider two official pretrained models: albert-base and albertxxlarge, which have hidden layer sizes of 768 and 4096 respectively. albert @xcite , similarly to its predecessor bert @xcite , uses learnable positional encoding, which has fixed size. therefore in order to encode pages that can be longer than the maximal length, we split the text into partially overlapping sequences of 256 tokens with an overlap of 50 tokens. we encode each of the sequences with albert, and average the hidden states from the last layer over each token. such representations of all the sequences are then also averaged to get the final feature vector. we compared alternative ways to extract features in appendix e. for word embedding models, either glove (pennington et al., 2014) or word2vec @xcite , we use official pre-trained weights. to encode the text we simply embed each token and then average the embedding features over the whole sequence. finally, if a class has multiple corresponding articles, we average the representations obtained from each of them. ## experiments to compare the quality of different types of auxiliary data, we run experiments on two existing zsl models. we choose to run all experiments on the standard zsl task, instead of generalized zsl (gzsl). that means we predict only unseen classes and not the union between seen and unseen classes. although gzls is the more practical long-term scenario, the standard zsl setup better isolates the usefulness and discriminativeness of auxiliary data from the model's level of overfitting to the seen classes. as we introduce no new methods and focus on the importance of auxiliary data instead, the standards zsl setup is more informative in our case. ## limitations & future work the performance of our approach is likely to depend on some general quality of wikipedia articles. although we have not evaluated this dependence directly in our work, at least the article's length indeed correlates with the corresponding class's accuracy (appendix b). therefore, the adaptability of the general approach to some particular domains not well-covered in wikipedia can also be limited. additionally, we assume that the classes used are possible to match with wikipedia articles. with the zsl methods we use can utilize the information from our textual descriptions effectively, however, they only rely on features from pretrained, off-the-shelf models. given that the textual de-scriptions are complex and contain only a sparse amount of visually relevant information, future research is needed to find methods more effective at fully utilizing our textual descriptions. additionally, there is a potential of using our textual descriptions for different purposes, such as pretraining of image recognition models, similarly to the recent works by @xcite and @xcite . finally, the issue of generalization to groups of classes remains an open challenge. ## conclusions we demonstrate that simply employing wikipedia articles as auxiliary data for zsl on imagenet yields a state-of-the-art performance, even on a relatively weak model. additionally, we show that standard zsl models poorly generalize across categories of classes. finally, we share the imagenet-wiki dataset with wikipedia articles describing corresponding imagenet classes to facilitate further the research on zsl, as well as on the interaction between language and vision more generally.