Evaluating pragmatic abilities of image captioners on A 3 DS (2023)

Abstract: Evaluating grounded neural language model performance with respect to pragmatic qualities like the trade off between truthfulness, contrastivity and overinformativity of generated utterances remains a challenge in absence of data collected from humans. To enable such evaluation, we present a novel open source image-text dataset “Annotated 3D Shapes” (A3DS) comprising over nine million exhaustive natural language annotations and over 12 million variable-granularity captions for the 480,000 images provided by Burgess & Kim (2018).We showcase the evaluation of pragmatic abilities developed by a task-neutral image captioner fine-tuned in a multi-agent communication setting to produce contrastive captions. The evaluation is enabled by the dataset because the exhaustive annotations allow to quantify the presence of contrastive features in the model’s generations. We show that the model develops human-like patterns (informativity, brevity, over-informativity for specific features (e.g., shape, color biases)).

## introduction and related work in human communication, language is rarely used as a unimodal channel; rather, language is mostly used in reference to the surroundings, i.e., it is grounded in the physical world. thus, in order to build artificial agents that could be potentially employed in scenarios requiring natural communication with humans, it is crucial to develop approaches for training such agents to communicate about the world in a human-like way @xcite . however, automatically evaluating the human-likeness of a trained system without costly human feedback is a recurring problem in nlp. in this paper, we set out to provide tools for evaluating human-like pragmatic abilities of grounded models and evaluate a model trained interactively via reinforcement learning, which is commonly suggested to give rise to task-oriented behavior @xcite . grounding of neural language models has been advanced greatly in recent years through image captioning models. starting with the work by @xcite and @xcite , neural encoder-decoder architectures have been dominating the field, recently extending to unified architectures @xcite . however, these approaches are task neutral, i.e., the models are trained to produce generally true image captions. in contrast, humans are highly flexible and pragmatic in their use of language and, e.g., adapt the granularity of their utterances to the requirements of the communicative task @xcite . it is generally guided by conversational maxims, suggesting that cooperative speakers should only provide as much information as required in a given context, be truthful, relevant, and brief @xcite . therefore, faced with a simple referential task of picking out a target item among an array of distractors, humans tend to mention contrastive features of the target (e.g., @xcite , i.e., the ones setting it apart from distractors. on the other hand, biases towards producing shape and color descriptions even when these aren't contrastive have been identified (e.g., @xcite . for grounded language models, the underlying pragmatic reasoning formalized as nested bayesian inference about the behavior of speakers and listeners (goodman and frank, 2016) inspired decoding schemes applied on top of standardly trained models (e.g., @xcite @xcite @xcite . however, evaluating the pragmatic qualities of models' predictions when they are applied to specific tasks (e.g., referential tasks) remains a challenge. currently standard metrics like bleu-n, rouge, cider and meteor @xcite @xcite , 2015; lin, 2004 ) for evaluating models' generations make reference to the surface form of ground truth image annotations. they cannot provide insight into models' mechanics and possible biases based on context-dependent functional aspects like mentioning contrastive features or being overinformative. given that model predictions might not always be syntactically well-formed and yet still count as functionally expedient for a human (e.g., see fig. 1 ), evaluating pragmatic aspects of natural language image captions is important. we propose a new dataset and metrics facilitating such evaluation in the next sections. ## conclusion we provide the a3ds dataset alongside evaluation metrics for investigating referential pragmatic abilities acquired by grounded language models on this dataset. with this dataset, we identify that an im-age captioner fine-tuned interactively via reinforcement learning developed a strikingly human-like shape bias, while being less overinformative than a task-neutral model. future research could expand such evaluations by including metrics which investigate additional aspects that might matter to human referential expression generation (e.g., the current metrics are agnostic to the surface order of discriminative features, while humans have preferences towards certain adjective ordering; @xcite ). although these results are specific to the given architecture, with this work we hope to inspire research opening up black box language models-an important task in the age of llms.