Revisiting Document Representations for Large-Scale Zero-Shot Learning (2021)

Abstract: Zero-shot learning aims to recognize unseen objects using their semantic representations. Most existing works use visual attributes labeled by humans, not suitable for large-scale applications. In this paper, we revisit the use of documents as semantic representations. We argue that documents like Wikipedia pages contain rich visual information, which however can easily be buried by the vast amount of non-visual sentences. To address this issue, we propose a semi-automatic mechanism for visual sentence extraction that leverages the document section headers and the clustering structure of visual sentences. The extracted visual sentences, after a novel weighting scheme to distinguish similar classes, essentially form semantic representations like visual attributes but need much less human effort. On the ImageNet dataset with over 10,000 unseen classes, our representations lead to a 64% relative improvement against the commonly used ones.

## introduction algorithms for visual recognition usually require hundreds of labeled images to learn how to classify an object @xcite . in reality, however, the frequency of observing an object follows a longtailed distribution @xcite : many objects do not appear frequently enough for us to collect sufficient images. zero-shot learning (zsl) @xcite , which aims to build classifiers for unseen object classes using their semantic representations, has thus emerged as a promising paradigm for recognizing a large number of classes. being the only information of unseen objects, how well the semantic representations describe the visual appearances plays a crucial role in zsl. one popular choice is visual attributes @xcite @xcite carefully annotated by humans. for example, the bird "red bellied woodpecker" has the "capped head pattern" and "pointed wing shape". while strictly tied to visual appearances, visual attributes are laborious to collect, limiting their applicability to small-scale problems with hundreds of classes. for large-scale problems like imagenet @xcite that has more than 20, 000 classes, existing zsl algorithms @xcite mostly resort to word vectors of classes names @xcite that are automatically extracted from large corpora like common crawl. while almost labor free, word vectors are purely text-driven and barely aligned with visual information. as a result, the state-of-the-art zsl accuracy on imagenet falls far behind being practical @xcite . is it possible to develop semantic representations that are as powerful as visual attributes without significant human effort? a feasibility study by representing a class with its wikipedia page shows some positive signs -wikipedia pages do capture rich attribute information. for example, the page "red-bellied woodpecker" contains phrases "red cap going from the bill to the nape" and "black and white barred patterns on their back, wings and tail" that exactly match the visual attributes mentioned above. in other words, if we can identify visual sentences from a document to represent a class, we are likely to attain much higher zsl accuracy foot_0 . to this end, we present a simple yet effective semi-automatic approach for visual sentence extraction, which leverages two informative semantic cues. first, we leverage the section structures of wikipedia pages: the section header indicates what kind of sentences (visual or not) appear in the section. concretely, we search wikipedia pages of common objects following the sysnsets in im-agenet (e.g., fish, room), and manually identify sections that contain visual information (e.g., characteristics, appearance). we then apply these visual headers to the wikipedia pages of the remaining imagenet classes. second, we observe that visual sentences share some common contextual patterns: for example, they contain commonly used words or phrases of visual attributes (e.g., red color, furry surface). to leverage these patterns, we perform k-means sentence clustering using the bert features @xcite and manually select clusters that contain visual information. we keep sentences in these clusters and combine them with those selected by section headers to represent a document. see figure 1 for an illustration. to further increase the discriminative ability of the visual sentences between similar object classes (e.g., breeds of dogs), we introduce a novel scheme to assign weights to sentences, emphasizing those that are more representative for each class. we validate our approach on three datasets: ima-genet fall 2011 dataset @xcite , which contains 14, 840 unseen classes with wikipedia pages; animals with attributes 2 (awa2) @xcite , which has 50 animal classes; attribute pascal and yahoo (apy) @xcite , which has 32 classes. our results are promising: compared to word vectors on imagenet, we improve by 64% using visual sentences. on awa2 and apy, compared to visual attributes annotated by humans, we improve by 8% and 5%, respectively. moreover, our new semantic representations can be easily incorporated into any zsl algorithms. our code and data will be available at https: //github.com/heendung/vs-zsl . similar to ours, @xcite ; @xcite ; @xcite ; @xcite represent classes by documents, by counting word frequencies but not extracting visual sentences. @xcite extract single word attributes, which are not discriminative enough (e.g., "red cap" becomes "red", "cap"). none of them works on zsl with over 1,000 classes. @xcite ; le cacheux et al. (2020) collect images and tags of a class and derives its semantic representation from tags, which is not feasible for unseen classes on zsl. zero-shot learning algorithms. the most popular way is to learn an embedding space in which visual features and semantic representations are aligned and nearest neighbor classifiers can be applied @xcite @xcite @xcite @xcite . these algorithms consistently improve accuracy on datasets with attributes. their accuracy on imagenet, however, is saturated, mainly due to the poor quality of semantic representations @xcite . suppose that we have access to a semantic representation a c (e.g., word vectors) for each class @xmath0, one popular algorithm devise @xcite proposes the learning objective the same classes while minimizing that between classes. we can then classify a test image x by arg max @xmath1. here, we consider that every class @xmath2 ## conclusion zsl relies heavily on the quality of semantic representations. most recent work, however, focuses solely on algorithm design, trying to squeeze out the last bit of information from the pre-define, likely poor semantic representations. @xcite has shown that existing algorithms are trapped in the plateau of inferior semantic representations. improving the representations is thus more crucial for zsl. we investigate this direction and show promising results by extracting distinctive visual sentences from documents for representations, which can be easily used by any zsl algorithms.