Describe Me an Auklet: Generating Grounded Perceptual Category Descriptions (2023)

Abstract: Human speakers can generate descriptions of perceptual concepts, abstracted from the instance-level. Moreover, such descriptions can be used by other speakers to learn provisional representations of those concepts. Learning and using abstract perceptual concepts is under-investigated in the language-and-vision field. The problem is also highly relevant to the field of representation learning in multi-modal NLP. In this paper, we introduce a framework for testing category-level perceptual grounding in multi-modal language models. In particular, we train separate neural networks to **generate** and **interpret** descriptions of visual categories. We measure the *communicative success* of the two models with the zero-shot classification performance of the interpretation model, which we argue is an indicator of perceptual grounding. Using this framework, we compare the performance of *prototype*- and *exemplar*-based representations. Finally, we show that communicative success exposes performance issues in the generation model, not captured by traditional intrinsic NLG evaluation metrics, and argue that these issues stem from a failure to properly ground language in vision at the category level.

## introduction grounded language use links linguistic forms (symbols) with meaning rooted in various perceptual modalities such as vision, sound, and the sensorymotor system @xcite . but grounding is not merely a solipsistic mapping, from form to meaning; rather, it results from a communicative context in which linguistic agents act on -and have goals in -the real world @xcite @xcite . large language models trained on vast amounts of text have been criticised for lacking grounded representations (bisk * equal contribution. ? baltimore_oriole crested_aucklet fish_crow least_aucklet common_tern baltimore_oriole crested_aucklet fish_crow least_aucklet common_tern baltimore_oriole crested_aucklet fish_crow least_aucklet common_tern that must be a crested aucklet! it's a goofy looking bird with a bright orange beak and musk gray body what's a crested aucklet? figure 1 : a simple learning scenario in which one speaker learns a visual concept from another speaker's description; the learner is then able to use their provisional representation to classify an entity as belonging to the concept. @xcite , and the fastgrowing field of multi-modal nlp has been working to address this problem @xcite . however, multi-modal models have several areas for improvement. recent work suggests that these models are affected by the distribution of items in training data, often over-representing specific scenarios and underrepresenting others @xcite . this, in turn, affects their ability to find a true balance between the levels of granularity in descriptions for novel concepts, as these models are expected to generalise @xcite . as a result, these models rely excessively on text and have to be supplied with various mechanisms, enforcing and controlling their attention on modalities such as vision @xcite @xcite . this raises questions about the nature of the relationship these models learn between linguistic and non-linguistic information. exploiting statistical regularities in multi-modal datasets can cause models to hallucinate. according to @xcite , neural image captioning systems can accurately describe objects in images but struggle to understand the overall situation, often relying on common contextual patterns associated with specific objects that co-occur. similar problems are common for other multi-modal models, datasets @xcite , and tasks such as visual question answering @xcite and embodied question answering @xcite . these examples, along with many others, illustrate that perceptual grounding cannot be achieved in the abstract but must be considered in a communicative context, which includes speakers' prior common ground, joint perception, and intentions @xcite . one important type of common ground is shared perceptual world knowledge, which need not necessarily rely on the immediate perceptual context. for instance, if someone mentions that red apples are sweeter than green ones, this communicates something, even to someone who is not concurrently looking at or tasting apples. we can acquire and use a (provisional) perceptual concept based on a natural language description produced by a conversation partner, a process referred to as fast mapping @xcite . can multi-modal language models generate a description of a perceptual category that similarly communicates the concept to an interlocutor? in this paper, we propose perceptual category description, which emphasises categorylevel grounding in a communicative context. this framework models a simple interactive scenario (figure 1 ) where (1) a describer, referred to as gen, generates a description of one or more a visual categories, (2) an interpreter, ipt, learns from the generated descriptions, and (3) classifies among both the seen classes, which it already has knowledge of, and the unseen classes described by gen. during training, the gen model has access to images and class labels from both the seen and "unseen" sets, but only receives supervision on ground-truth descriptions from the seen set. this ensures that during testing the generator is evaluated based on its ability to use category-level representations of the unseen classes, rather than memorising descriptions from the training data. the ipt model only has access to instances from seen at train time and performs zero-shot image clas-sification on unseen instances using descriptions produced by gen as auxiliary class information. zero-shot learning from text descriptions is not a novel task; our focus in this work is on the generation of perceptual category descriptions, using "communicative success"-the performance of the ipt model -as a semi-extrinsic evaluation metric. the proposed evaluation method differs from many standard automatic generation evaluation metrics, such as bleu @xcite , which are not designed to capture the level of communicative usefulness of the generated texts. in contrast to many language-and-vision tasks, we explore the ability of multi-modal models to perform grounding on class-level representations, distinct from instancelevel representations, e.g. images. foot_0 additionally, we highlight the issue of mismatch between intrinsic evaluation (generation metrics) and task-based evaluation, as indicated by the performance of the ipt. our results reveal challenges involved in developing better models with the ability to ground at the class level. we believe that our fine-grained analysis of the task, data and models sheds light on the problems associated with both generating and interpreting class-level image descriptions. we also contribute insights into the extent to which current evaluation methods for generated texts consider communicative context. the framework that we propose can be used for evaluating existing models of language grounding and can also aid in building new multi-modal models that perform grounding in communication. to support research in this direction, we have made our code and data available here: https://github.com/gu-clasp/ describe-me-an-auklet . ## background prototypes and exemplars cognitive theories of categorisation are psychologically-motivated accounts of how humans represent perceptual concepts and use them for classification. such theories have challenged the assumption that categories can be defined in terms of a set of necessary and sufficient features. in contrast, they try to account for phenomena like prototypically effects, in which certain members of a category are perceived as more representative of the class than others. in prototype theory, cognitive categories are defined by a prototype, an abstract idealisation of the category. membership in the class, then, is judged in reference to the prototype @xcite . in exemplar theory, (e.g., @xcite , concepts are still defined in relation to an ideal, but this time the ideal is an exemplar, which is a particularly representative member of the very category. put another way, an exemplar is of the same kind as the other members of the category, whereas prototypes, in general, are not. experimental evidence suggests that humans employ both exemplar and prototype-based strategies @xcite . perceptual categories play a role in natural language interpretation and generation. in fact, classifier-based meaning has been proposed as a way to ground language in perception @xcite . there are both formal and computational interpretations of this approach that support compositional semantics for lexical items with classifier-based perceptual meanings @xcite . in this paper, we explore how classifierbased meaning facilitates the generation of classlevel descriptions by testing three different gen model architectures: one motivated by prototype theory, one by exemplar theory, and one that uses a hybrid approach. ## models at a high level, gen and ipt each have two connected modules: an image classifier, and a grounded language module. both networks learn visual representations which are shared between the classification and language tasks. during training, ipt learns to interpret textual descriptions of seen classes by mapping them into its visual representation space. if it generalises well, textual descriptions of unseen classes should then be mapped to useful visual representations at test time, even though no images of unseen classes were available during training. contrariwise, gen is trained to generate descriptions of seen classes based on its visual representation of those classes. at test time, gen must extrapolate to generating descriptions of unseen classes, for which no ground-truth descriptions were provided during training. ## results our primary objective is to examine if we can learn models capable of grounding on the category level in the zero-shot image description generation setup. first, we address part of this question by exploring the performance of the ipt model when classifying new classes given gen-generated descriptions ( our first observation is that the model is moderately successful on the zero-shot classification task. when learning from the ground truth descriptions, the model performs well above the random baseline. while 0.19 is not very high for classification accuracy in general, it is not out of line for unseen results in zero-shot learning. it must be noted that classifying a bird as belonging to one of 200 species based only on a textual description would be a difficult task for some humans as well. that the model can use the ground truth text descriptions to learn class representations that are somewhat useful for image classification is encouraging for the prospect of using it to evaluate the gen models. however, we note that the performance of the model using descriptions generated from the best gen model is quite a lot worse than the ground truth. this suggests that while the descriptions generated by the best gen models are not totally useless, they are nevertheless not as next, we compare different gen models, as shown in ## discussion and conclusion one of the motivations behind adopting this task was its reliance on class-level representations, which distinguishes it from other image-specific language-and-vision tasks. we wanted to see how well the models can ground language in the absence of image pixels. our results revealed several interesting directions for further exploration in modelling, feature representation, and the evaluation of generation and interpretation. strikingly, the top-performing models were the gen-ex models, which effectively reintroduce specific images by picking out exemplars to generate from. of course, the models we used in this study were relatively simple, and more sophisticated neural models may yield different results. but this raises an interesting question for future work -what does it take to learn grounded representations that are useful in this particular communicative context? more generally, why do the gen model descriptions fall short in comparison to ground-truth for the zero-shot performance on the ipt model? there are two possible explanations. one is that the generated descriptions may lack the necessary visual information required for successful classification of unseen classes. secondly, the texts produced by the gen model may not be interpretable by the ipt model. recall that the ipt model was trained on ground truth descriptions from seen. these descriptions have a structure to them -certain regularities in conveying visual information. if the gen descriptions deviate from this structure, ipt may struggle to effectively utilise them, even if they do in some sense "contain" visual information. indeed, there is some evidence that this is what is happening. we see that nucleus sampling resulted in higher discriminativity scores, including for the gen-prot and gen-both models. although the generator produces sequences with adjective-noun phrases that identify the correct class, ipt cannot properly use them, perhaps because they appear in texts that are "ungrammatical" for the distribution ipt was trained on. as both the gen and ipt models are simple approximation functions of the data on which they are trained, they may rely too heavily on patterns and regularities, which can hinder their ability to learn to recognise and generate better category-level descriptions. this points to an important research direction, as it might highlight the limitations of many current existing multi-modal models which are built on top of the transformer architecture. such models might still be useful in various domains but often face challenges in learning higher-level concepts about the world. a different question is whether generation metrics reflect the communicative power of texts as measured by the interpreter's performance. in the case of gen-ex, ipt performs best with texts generated using beam search ( in future work, we will focus on improving interpretation performance by emphasising fine-grained differences between class features. inspecting how the generation of more descriptive and more discriminative class descriptions can be achieved is also important. additionally, we will examine the impact of a more interactive context on the task, which could be studied in a reinforcement learning setup @xcite .