Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions (2020)

Abstract: Transfer learning using ImageNet pre-trained models has been the de facto approach in a wide range of computer vision tasks. However, fine-tuning still requires task-specific training data. In this paper, we propose N3 ( N eural N etworks from N atural Language) - a new paradigm of synthesizing task-specific neural networks from language descriptions and a generic pre-trained model. N3 leverages language descriptions to generate parameter adaptations as well as a new task-specific classification layer for a pre-trained neural network, effectively “fine-tuning” the network for a new task using only language descriptions as input. To the best of our knowledge, N3 is the first method to synthesize entire neural networks from natural language. Experimental results show that N3 can out-perform previous natural-language based zero-shot learning methods across 4 different zero-shot image classification benchmarks. We also demonstrate a simple method to help identify keywords in language descriptions leveraged by N3 when synthesizing model parameters.

## introduction a person with generic world knowledge can learn to perform a new task based on verbal instructions. on the other hand, despite recent successes in deep learning, it remains challenging to re-purpose pretrained visual classification models to recognize a new set of objects without labeling a new training dataset. a natural question emerges from this observation: can a computer also learn to recognize new objects, simply by reading the descriptions of them in natural language? concretely, can we create visual classifiers using language descriptions of the objects of interest? in this paper, we introduce a new paradigm for synthesizing task-specific neural networks for image classification simply from language descriptions of the relevant objects. we propose n 3 -neural networks from natural language, a metamodel that takes a list of object descriptions as input to produce a classification model for these objects, as illustrated in figure 1 . the capability of producing task-specific neural network from lan-guage descriptions makes n 3 ideal to a wide range of zero-shot tasks @xcite @xcite @xcite where we do not have visual training data but can still easily obtain language descriptions of the objects of interest. prior zero-shot learning methods aim to achieve a similar goal of applying pre-trained networks on unseen classes. in zero-shot image classification, a typical method of generalizing to unseen classes is to construct class embeddings to augment the classification layer of the pre-trained network or take retrieval-based approaches while utilizing generic visual features from pre-trained networks @xcite @xcite @xcite . extending on the idea of generating classification layers for the pre-trained network, n 3 modifies the parameters of all layers in the pre-trained network. while the underlying pre-trained network in previous approaches only extracts generic visual features, n 3 makes it possible to extract taskspecific ones. this approach effectively increases the capacity at which semantic information in the descriptions can affect the pre-trained network. in our experiments, we evaluated our proposed n 3 method with 4 popular zero-shot image classification benchmark datasets. we performed ablation studies to understand the importance of synthesizing task-specific feature extractors, the necessity of a pre-trained visual classification model and the effects of language representation choices on the efficacy of n 3 . in addition, we provide a simple approach to help interpret what aspects in the language descriptions are n 3 -generated models examining when making predictions. to summarize, our contributions are 3-fold: ## related work in this section, we are situating our work in the context of zero-shot learning and dynamic parameter generation for neural networks. zero-shot learning zero-shot learning studies how we can generalize our models to perform well for tasks without any labeled training data at all. achieving classification accuracy above chancelevel in such scenarios requires modeling the relationships between the seen classes during training and the unseen classes during testing. a typical and effective method is to manually engineer class attribute vectors to obtain representations of seen and unseen classes in a shared attribute space @xcite @xcite @xcite ). yet such a method requires laborious engineering of class attributes, which is not feasible for large-scale and/or fine-grained classification tasks @xcite @xcite . hence, there is also work in zero-shot learning that attempts to leverage textual data as object class representations @xcite . the majority of these models are committed to embedding-based retrieval approaches, where classification is re-formulated as retrieving the class embedding with maximal similarity @xcite . while they can handle well the case where there is an indefinite number of classes during test time, such approaches suffer from extra computation cost at inference time since they need to traverse all the seen data points. moreover, these models often rely on a pre-trained feature extractor for input, which is usually fixed and cannot be further adapted for the unseen classes @xcite @xcite . while there is a handful of work that tries to modify the parameters in-place during test time, they either rely on shallow language representation with limited expressiveness @xcite or require a significant amount of textual descriptions per class to train their model @xcite , both of which are not ideal. in our work, we aim to learn a model that can synthesize parameters for entire neural networks to adapt to the new tasks using short descriptions of the object classes. this leads to better metadata efficiency of our proposed method. ## n methodology in this section, we describe our approach for synthesizing task-specific neural networks to recognize new objects with their natural language descriptions and a generic pre-trained model. we denote a list of natural language descriptions for k objects, each containing l tokens as @xmath0,l } @xmath1, @xmath2, @xmath3. we denote a pre-trained model with parameters θ as f(•; θ), so that for a set of images x, f(x; θ) produces the classification prediction y . we can now precisely formulate our problem as follows: given a pre-trained classification model, f(•; θ), and the natural language description of k object classes, d, adapt the original parameters θ to the specialized parameters θ so that the finetuned f(•; θ ) model accurately classifies the k objects described in d. ## experimental setup we evaluate n 3 by comparing its efficacy in solving natural language guided zero-shot learning problems with prior state-of-the-art methods. in this section, we introduce our training method, datasets and evaluation protocols. ## results and discussion in this section, we compare n 3 method with prior work on 4 zero-shot image classification benchmark datasets. furthermore, we provide a simple approach to help interpret what part of the language input is being taken as evidence by the synthesized visual classifier to make predictions. we then show through ablation studies the importance of adaptation of all parameters in the pre-trained network, the necessity of pre-trained networks, and the effect of language representation choices on the success of n 3 . ## conclusion in this paper, we have demonstrated that small amount of unstructured natural language descriptions for object classes can provide enough information to fine-tune an entire pretrained neural network to perform classification task on class labels unseen during training. we have achieved state-of-the-art performance for natural language guided zero-shot image classification tasks on 4 public datasets with practical metadata requirements. in addition, we presented in-depth analysis and extensive ablation studies on various aspects of the model functioning mechanism and architecture design, showing the necessity of our design contribution in achieving good results. classes are randomly split into 160 "seen" classes for training and validation (within which 120 classes are randomly selected for training, and 40 for validation) and 40 "unseen" classes for testing, and a 40-class classification model is generated by n 3 . within the seen classes, 20% of the dataset is are excluded since @xcite used them for testing to report classification results on seen classes, which is irrelevant to our comparisons. meta models are trained using the 160 seen classes only. similarly, in oxford flowers, the 102 classes of flowers are split into 82 "seen" classes and 20 "unseen" classes randomly. we used a resnet-18 pre-trained on imagenet as our base model. note that while we didn't adopt the larger vgg-19 base model as previous work, later we will show that we are still able to outperform them significantly with the lesser resnet-18 architecture. during test time, n 3 generates neural networks based on test set class descriptions and these generated networks are used for zero-shot evaluation. as shown in we then move on to evaluate the performance of n 3 on more rigorous and difficult zero-shot metasplits on the cub-2011 and nab dataset. this split is called sce (super category exclusive) split, which ensures that the parent categories of unseen classes are exclusive to those of the seen classes. such split is used in many prior works @xcite @xcite to evaluate their proposed method. we evaluated n 3 using such meta-split and results are shown in