T ext2 M odel: Text-based Model Induction for Zero-shot Image Classification (2024)

Abstract: We address the challenge of building task-agnostic classifiers using only text descriptions, demonstrating a unified approach to image classification, 3D point cloud classification, and action recognition from scenes. Unlike approaches that learn a fixed representation of the output classes, we generate at inference time a model tailored to a query classification task. To generate task-based zero-shot classifiers, we train a hypernetwork that receives class descriptions and outputs a multi-class model. The hypernetwork is designed to be equivariant with respect to the set of descriptions and the classification layer, thus obeying the symmetries of the problem and improving generalization. Our approach generates non-linear classifiers, handles rich textual descriptions, and may be adapted to produce lightweight models efficient enough for on-device applications. We evaluate this approach in a series of zero-shot classification tasks, for image, point-cloud, and action recognition, using a range of text descriptions: From single words to rich descriptions. Our results demonstrate strong improvements over previous approaches, showing that zero-shot learning can be applied with little training data. Furthermore, we conduct an analysis with foundational vision and language models, demonstrating that they struggle to generalize when describing what attributes the class lacks.

## introduction we explore the challenge of zero-shot image classification by leveraging text descriptions. this approach pushes the boundaries of conventional classification methods by demanding that models categorize images into specific classes based solely on written descriptions, without having previously encountered these classes during training. 1 in various domains, numerous attempts have been made to achieve zero-shot classification capacity ( §2). unfortunately, as we now explain, existing studies are limited in two major ways: (1) query-dependence; and (2) richness of language description. first, query-dependence. to illustrate the issue, consider a popular family of zero-shot learning (zsl) approaches, which maps text (like class labels) and images to a shared space @xcite @xcite @xcite . to classify a new image from an unseen class, one finds the closest class label in the shared space. the problem with this family of shared-space approaches is that the learned representation (and the knn classifier that it induces) remains "frozen" after training, and is not tuned to the classification task given at inference time. for instance, furry toys would be mapped to the same shared representation regardless of whether they are to be distinguished from other toys, or from other furry things (see figure 1 ). the same limitation also hinders another family of zsl approaches, which synthesize samples from unseen classes at inference time using conditional generative models, and use these samples with knn classification @xcite . some approaches address the query-dependence limitation by assuming that test descriptions are known during training @xcite , or by (costly) training a classifier or generator at inference time @xcite . instead, here we learn a model that produces taskdependent classifiers and representations without test-time training. the second limitation is language richness. natural language can be used to describe classes in complex ways. most notably, people use negative terms, like "dogs without fur", to distinguish class members from other items. previous work could only handle limited richness of language descriptions. for instance, it cannot represent adequately textual descriptions with negative terms @xcite @xcite . in this paper, we wish to handle the inherent linguistic richness of natural language. an alternative approach to address zero shot image recognition tasks involves leveraging large generative vision and language models (e.g., gpt4vision). these foundational models, trained on extensive datasets, exhibit high performance in zero and few-shot scenarios. however, these models are associated with certain limitations: (1) they entail significant computational expenses in both training and inference. (2) their training is specific to particular domains (e.g., vision and language) and may not extend seamlessly to other modalities (e.g., 3d data and language). (3) remarkably, even state-of-the-art foundational models encounter challenges when confronted with tasks involving uncommon descriptions, as demonstrated in §5.3. in addition to the limitations posed by large generative models, there is a growing demand for smaller, more efficient models that can run on edge devices with limited computational power, such as mobile phones, embedded systems, or drones. giant models that require cloud-based infrastructures are often computationally expensive and not suitable for real-time, on-device applications. furthermore, some companies are unable to rely on cloud computing due to privacy concerns or legal regulations that mandate keeping sensitive user data within their local networks (on-premises). our approach addresses these needs by enabling the automatic generation of task-specific models that are lightweight and capable of running on weaker de-vices without requiring cloud resources. here, we describe a novel deep network architecture and a learning workflow that addresses these two aspects: (1) generating a discriminative model tuned to requested classes at query time and (2) supporting rich language and negative terms. to achieve these properties, we propose an approach based on hypernetworks (hns) @xcite ). an hn is a deep network that emits the weights of another deep network (see figure 2 for an illustration). here, the hn receives a set of class descriptions and emits a multi-class model that can classify images according to these classes. interestingly, this text-image zsl setup has an important symmetric structure. in essence, if the order of input descriptions is permuted, one would expect the same classifiers to be produced, reflecting the same permutation applied to the outputs. this property is called equivariance, and it can be leveraged to design better architectures @xcite @xcite . taking invariance and equivariance into account has been shown to provide significant benefits for learning in spaces with symmetries like sets @xcite @xcite graphs @xcite and deep weight spaces @xcite . in general, however, hns are not always permutation equivariant. we design invariant and equivariant layers and describe an hn architecture that respects the symmetries of the problem, and term it t2m-hn: a text-to-model hypernetwork. we put the versatility of t2m-hn to the test across an array of zero-shot classification tasks, spanning diverse data types including images, 3d point clouds, and 3d skeletal data for action recognition. our framework exhibits a remarkable ability to incorporate various forms of class descriptions including long and short texts, as well as class names. notably, t2m-hn surpasses the performance of previous state-of-the-art methods in all of these setups. our paper offers four key contributions: (1) it identifies limitations of existing zsl methods that rely on fixed representations and distance-based classifiers for text and image data. it proposes taskdependent representations as an alternative; (2) it introduces the text-to-model (t2m) approach for generating deep classification models from textual descriptions; (3) it investigates equivariance and invariance properties of t2m models and designs t2m-hn, an architecture based on hns that ad-dataset sample description example name and type data type description awa (lampert et al., 2009) animal images class name (1) moose (2) elephant long ( 1 ) "an animal of the deer family with humped shoulders, long legs, and a large head with antlers.", (2) "a plant-eating mammal with a long trunk, large ears, and thick, grey skin." negative (1) "an animal without stripes and not gray", (2) "an animal without fur and without horns" attribute (1) "animals with fur" (2) "animals with long trunk" figure 2 : the text-to-model learning problem and our architecture. our model (yellow box) receives a set of class descriptions as input and outputs weights w for a downstream on-demand model (orange). the model has two main blocks: a pretrained text encoder and a hypernetwork that obeys certain invariance and equivariance symmetries. the hypernetwork receives a set of dense descriptors to produce weights for the on-demand model. ## related work in this section we cover previous approaches to leverage textual description to classify images of unseen classes. ## problem formulation our objective is to learn a mapping τ from a set of k natural language descriptions into the space of a k-class image classifier. here, we address the case where the architecture of the downstream classifier is fixed and given in advance, but this assumption can be relaxed as in @xcite . formally, let s @xmath0, . . . , s k } be a set of k class descriptions drawn from a distribution p k , where s j is a text description of the j th class. let τ be a model parameterized by a set of parameters ϕ. it takes the descriptors and produces a set of parameters w of a k-class classification model f (•; w ). therefore, we have τ ϕ : {s 1 , . . . s k } → r d , where d is the dimension of w , that is, the number of parameters of f (•; w ), and we denote @xmath1. let l : y × @xmath2, and let {x i , y i } n @xmath3. for k-class classification, @xmath4, . . . , k}. we can explicitly write the loss in terms of ϕ as follows. we propose to address the t2m problem, using an hns. an hn is a model that outputs the weights of another model @xcite . in our case, it receives a set of textual descriptions of classes to be recognized, and outputs the weights of a classifier that can discriminate them. figure 2 illustrates our architecture. it has two components. first, a text encoder g takes natural language descriptions and transforms them into dense descriptors; and second, an hn h takes these dense descriptors and emits weights for a downstream classifier. in this paper, we do not impose any special properties on the text encoder g. it can be any model trained using language data (no need for multi-modal data). ## experiments the t2m setup is about producing a model that can be applied to data from new classes. accordingly, the model trains on data from a set of training classes, alongside their text descriptions. then, it is tested on data from new classes, given the text descriptions of these classes. we evaluate t2m-hn in zero-shot classification, using three image datasets, one 3d point cloud dataset, and one action recognition dataset. we consider various forms of text description, including single-word class labels, few-word class names, and longer descriptions that could also include negative properties (i.e. properties that the images in the class do not have). finally, we study one-class classification based on text attributes. due to space constraints, we provide a concise description of our experimental settings here. further details can be found in appendix b. baselines: we compare our t2m-hn with five text-based zero-shot approaches for image recognition: (1) devise @xcite projects images to a pre-trained language model space by adding a projection head to a pre-trained visual classification model; (2) deep embedding model (dem) @xcite uses the visual space as the shared embedding space; (3) cizsl (elhoseiny and elfeki, 2019) trains conditional gans with a loss designed to generate samples from unseen classes without synthesizing unrealistic images. at inference time, the gan is conditioned on test descriptions, generates synthetic image representations, and test images are classified using knn w.r.t. to the synthetic images; (4) grawd @xcite trains a conditional gan with a loss that helps to reach regions in space that are hard to classify as seen classes; and (5) zsml (verma et al., 2020) combines meta-learning with a wgan, to generate samples from unseen classes, and use them to train a classifier at test time. when relevant, we also computed the performance obtained when using clip, blip2, and gpt4vision. note that those models were trained using massively large datasets, so it is reasonable to assume they have seen all classes studied here. this is hence not zero-shot classification, and the results can be viewed as a "skyline" value that zero-shot approaches should aim at. datasets: we experiment with three image datasets: (1) animals with attributes (awa) @xcite ; (2) sun @xcite ; and (3) cub @xcite ; a 3d point-clouds dataset: (4) modelnet40 @xcite ; and an action recognition dataset: (5) ba-bel 120 @xcite , containing sequences of body skeletons. ## conclusion we introduced the t2m learning algorithm, a novel approach that generates an image recognition classifier "on demand" using only class descriptions provided at test time. t2m allows for taskdependent class representations rather than fixed ones. we analyzed the group symmetries a t2m model must adhere to and introduced t2m-hn, a model based on hns that obeys these symmetries. through extensive experiments across various classification scenarios-including images, 3d point clouds, and action recognition-we explored the adaptability of the model to descriptions of differing complexities, from single and few-word class names, through long text descriptions, all the way to "negative" and attribute descriptions. our results clearly demonstrate the potential of the t2m modeling approach.