Shaping Visual Representations with Language for Few-Shot Classification (2020)

Abstract: By describing the features and abstractions of our world, language is a crucial tool for human learning and a promising source of supervision for machine learning models. We use language to improve few-shot visual classification in the underexplored scenario where natural language task descriptions are available during training, but unavailable for novel tasks at test time. Existing models for this setting sample new descriptions at test time and use those to classify images. Instead, we propose language-shaped learning (LSL), an end-to-end model that regularizes visual representations to predict language. LSL is conceptually simpler, more data efficient, and outperforms baselines in two challenging few-shot domains.

## introduction humans are powerful and efficient learners partially due to the ability to learn from language @xcite . for instance, we can learn about robins not by seeing thousands of examples, but by being told that a robin is a bird with a red belly and brown feathers. this language further shapes the way we view the world, constraining our hypotheses for new concepts: given a new bird (e.g. seagulls), even without language we know that features like belly and feather color are relevant @xcite . in this paper, we guide visual representation learning with language, studying the setting where no language is available at test time, since rich linguistic supervision is often unavailable for new concepts encountered in the wild. how can one best use language in this setting? one option is to just regularize, training representations to predict language descriptions. another is to exploit the compositional nature of language directly by using it as a bottleneck in a discrete latent variable model. figure 1 : we propose few-shot classification models whose learned representations are constrained to predict natural language task descriptions during training, in contrast to models which explicitly use language as a bottleneck for classification @xcite . for example, the recent learning with latent language (l3; @xcite ) model does both: during training, language is used to classify images; at test time, with no language, descriptions are sampled from a decoder conditioned on the language-shaped image embeddings. whether the bottleneck or regularization most benefits models like l3 is unclear. we disentangle these effects and propose language-shaped learning (lsl), an end-to-end model that uses visual representations shaped by language (figure 1 ), thus avoiding the bottleneck. we find that discrete bottlenecks can hurt performance, especially with limited language data; in contrast, lsl is architecturally simpler, faster, uses language more efficiently, and outperforms l3 and baselines across two few-shot transfer tasks. ## related work language has been shown to assist visual classification in various settings, including traditional visual classification with no transfer @xcite and with language available at test time in the form of class labels or descriptions for zero- @xcite or fewshot @xcite learning. unlike past work, we have no language at test time and test tasks differ from training tasks, so language from training cannot be used as additional class information (cf. @xcite or weak supervision for labeling additional in-domain data (cf. @xcite . our setting can be viewed as an instance of learning using privileged information (lupi; @xcite , where richer supervision augments a model only during training. in this framework, learning with attributes and other domain-specific rationales has been tackled extensively @xcite @xcite ; language less so. gordo and larlus (2017) use meteor scores between captions as a similarity measure for specializing embeddings for image retrieval, but do not directly ground language explanations. @xcite explore a supervision setting similar to ours, except in simple text and symbolic domains where descriptions can be easily converted to executable logical forms via semantic parsing. another line of work studies the generation of natural language explanations for interpretability across language (e.g. entailment; @xcite and vision @xcite tasks, but here we examine whether predicting language can actually improve task performance; similar ideas have been explored in text @xcite and reinforcement learning @xcite domains. ## language-shaped learning we are interested in settings where language explanations can help learn representations that generalize more efficiently across tasks, especially when training data for each task is scarce and there are many spurious hypotheses consistent with the input. thus, we study the few-shot (meta-)learning setting, where a model must learn from a set of train tasks, each with limited data, and then generalize to unseen tasks in the same domain. specifically, in n -way, k-shot learning, a task t consists of n support classes {s while the language approach we propose is applicable to nearly any meta-learning framework, we use prototype networks @xcite , which have a simple but powerful inductive bias for fewshot learning. prototype networks learn an embedding function f θ for examples; the embeddings of the support examples of a class n are averaged to form a class prototype (omitting task (t) for clarity): given a query example (x m , y m ), we predict class n with probability proportional to some similarity function s between c n and f θ (x m ): ## experiments here we describe our two tasks and models. for each task, we evaluate lsl, l3, and a prototype network baseline trained without language (meta; figure 1 ). for full details, see appendix a. shapeworld. first, we use the shapeworld @xcite dataset used by @xcite , which consists of 9000 training, 1000 validation, and 4000 test tasks (figure 2 ). 3 each task contains a single support set of @xmath0, generated with a minimal recursion semantics representation of the concept @xcite . each concept is a spatial relation between two objects, each object optionally qualified by color and/or shape, with 2-3 distractor shapes present. the task is to predict whether a query image x belongs to the concept. for ease of comparison, we report results with models identical to @xcite , where f θ is the final convolutional layer of a fixed imagenetpretrained vgg-16 @xcite fed through two fully-connected layers: however, because fixed imagenet representations may not be the most appropriate choice for artificial data, we also run experiments with convolutional networks trained from scratch: either the 4-layer convolutional backbone used in much of the few-shot literature @xcite , as used in the birds experiments we describe next, or a deeper resnet-18 @xcite . this is a special binary case of the few-shot learning framework, with a single positive support class s and prototype c. thus, we define the similarity function to be the sigmoid function s(a, b) = σ(a • b) and the positive prediction p (ŷ = 1 | x) = s (f θ (x), c). g φ is a 512dimensional gated recurrent unit (gru) rnn @xcite trained with teacher forcing. through a grid search on the validation set, we set λ @xmath1. birds. to see if lsl can scale to more realistic scenarios, we use the caltech-ucsd birds dataset @xcite , which contains 200 bird species, each with 40-60 images, split into 100 train, 50 validation, and 50 test classes. during training, tasks are sampled dynamically by selecting n classes from the 100 train classes. k support and 16 query examples are then sampled from each class (similarly for val and test). for language, we use the descriptions collected by @xcite , where this bird has distinctive-looking brown and white stripes all over its body, and its brown tail sticks up. the bird has a white underbelly, black feathers in the wings, a large wingspan, and a white beak. amt crowdworkers were asked to describe individual images of birds in detail, without reference to the species (figure 2 ). while 10 english descriptions per image are available, we assume a more realistic scenario where we have much less language available only at the class level: removing associations between images and their descriptions, we aggregate d descriptions for each class, and for each k-shot training task we sample k descriptions from each class n to use as descriptions w n . this makes learning especially challenging for lsl due to noise from captions that describe features only applicable to individual images. despite this, we found improvements with as few as @xmath2, which we report as our main results, but also vary d to see how efficiently the models use language. we evaluate on the @xmath3, @xmath4, and as f θ use the 4-layer convolutional backbone proposed by @xcite . here we use a learned bilinear similarity function, s(a, b) = a wb, where w is learned jointly with the model. g φ is a 200-dimensional gru, and with another grid search we set λ @xmath5. ## results results are in in the low-data regime, l3's lower performance is unsurprising, since it must generate language at test time, which is difficult with so little data. example output from the l3 decoder in figure 4 highlights this fact: the language looks reasonable in some cases, but in others has factual errors (dark gray bird; black pointed beak) and fluency issues. these results suggest that any benefit of l3 is likely due to the regularizing effect that language has on its embedding model f θ , which has been trained to predict language for test-time inference; in fact, the discrete bottleneck actually hurts in some settings. by using only the regularized visual representations and not relying exclusively on the generated language, lsl is the simpler, more efficient, and overall superior model. ## discussion we presented lsl, a few-shot visual recognition model that is regularized with language descriptions during training. lsl outperforms baselines across two tasks and uses language supervision more efficiently than l3. we find that if a model is trained to expose the features and abstractions in language, a linguistic bottleneck on top of these language-shaped representations is unnecessary, at least for the kinds of visual tasks explored here. the line between language and sufficiently rich attributes and rationales is blurry, and recent work @xcite suggests that similar performance gains can likely be observed by regularizing with attributes. however, unlike attributes, language is (1) a more natural medium for annotators, (2) does not require preconceived restrictions on the kinds of features relevant to the task, and (3) is abundant in unsupervised forms. this makes shaping representations with language a promising and easily accessible way to improve the generalization of vision models in low-data settings.