HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales (2023)

Abstract: Current captioning datasets focus on object-centric captions, describing the visible objects in the image, often ending up stating the obvious (for humans), e.g. “people eating food in a park”. Although these datasets are useful to evaluate the ability of Vision & Language models to recognize and describe visual content, they do not support controlled experiments involving model testing or fine-tuning, with more high-level captions, which humans find easy and natural to produce. For example, people often describe images based on the type of scene they depict (“people at a holiday resort”) and the actions they perform (“people having a picnic”). Such concepts are based on personal experience and contribute to forming common sense assumptions. We present the High-Level Dataset, a dataset extending 14997 images from the COCO dataset, aligned with a new set of 134,973 human-annotated (high-level) captions collected along three axes: scenes, actions and rationales. We further extend this dataset with confidence scores collected from an independent set of readers, as well as a set of narrative captions generated synthetically, by combining each of the three axes. We describe this dataset and analyse it extensively. We also present baseline results for the High-Level Captioning task.

## introduction conceptual grounding broadly refers to the idea that symbols (e.g. language) are grounded in perception @xcite . perceptually grounded communication is made possible by the fact that perceptual experiences are largely shared. however, individual experience can also license subjective inferences which inform not just what we express through language, but also what we choose to assume and leave unexpressed @xcite . among the many modalities available in the perceptual spectrum, visual grounding has always been of primary interest as it provides a relatively straightforward way to link linguistic expressions to physical objects. consistent with this claim, a glance at many widely used datasets and models in image captioning reveals a bias towards 'objectcentric' descriptions, whereby models are trained on image-text pairs where the text consists of explicit mentions of objects visible in the scene. however, experience and perception also motivate other, non-object-centric ways of talking about the world, for example, when we talk about scenes, or when we describe actions or their underlying rationales. while such 'high-level' descriptions are also perceptually grounded, they incorporate world knowledge and subjective experience. for example, the object-centric description in in this work, we tackle the issue of grounding high-level linguistic descriptions in the visual modality, proposing the high-level (hl) dataset: a resource for vision and language (v&l) modeling which aligns existing object-centric captions with human-collected high-level descriptions of images along three different axes: scenes, actions and rationales. the high-level captions capture the human interpretation of the scene which are complementary to object-centric captions used in current v&l datasets, e.g. in coco @xcite . we take a step further, and we collect confidence scores from independent annotators, which serve to shed image axis caption scene the picture is shot in a ski resort action they are just relaxing after a round of skiing rationale they want to have a good time together object-centric (coco) a woman and a boy sitting in the snow outside of a cabin. some efforts have been made to understand how low-level concepts improve generalization capabilities and connect to high-level concepts. objectcentric captions help to improve the generalization over unseen objects @xcite and play a role in the model understanding of abstract concepts @xcite . in our work, we are interested in the relations between what @xcite refer to as 'conceptual' and 'non-visual' descriptions, which we re-frame as a distinction between low-level (object-centric) and high-level descriptions in multimodal learning. we release a novel dataset to foster research in this direction. motivation for the present work is also provided by recent research exploring the visual correlates of inferences, temporal and causal relationships (e.g., @xcite , which also have implications for generation. in visual storytelling, for instance, a model has to understand actions and interactions among the visually depicted entities @xcite @xcite . identifying actions is a prerequisite for predicting their motivations or rationales as well as explaining automatically generated descriptions of images @xcite . actions and intention are paramount to performing commonsense and temporal reasoning on visual inputs. along these lines, @xcite creates dynamic stories on top of static images, where the task is to predict priors and subsequent actions and rationales. our work is similar in spirit, as we align high-level descriptions of actions and rationales with low-level descriptions of static images. some work has also been done to test multimodal model grounding capabilities from a more linguistic perspective. @xcite build a benchmark to test models on a variety of linguistic phenomena, like spatial relations, counting, existence, etc. @xcite assess the integration of complementary information of v&l models across modalities, while @xcite test multimodal models on compositional reasoning. in this context, the hl dataset proposed here can offer another benchmark for v&l models' understanding of high-level descriptions of images. such descriptions are licensed by the entities depicted in the visual modality and the relationships between them but they do not mention them explicitly. ## data in this section, we describe the protocol used to collect annotations for scenes, actions and rationales and the subsequent collection of confidence scores through crowdsourcing. differently from previous works, such as coco, where human annotators are instructed to be objective and to mention only the objects clearly visible in the picture, we elicit high-level concepts in the form of captions by encouraging the annotators to rely on their subjective interpretation of the image. ## dataset analysis in this section, we analyse the captions collected in the high-level dataset. to provide insights on the kind of captions collected, we analyse the distribution of the captions across different axes, also comparing them with the object-centric coco captions foot_1 . furthermore, we perform a grammatical error analysis, which we report in appendix a.1. ## baselines and results in this section, we show how the dataset can be used to finetune models to generate high-level, aspect-specific descriptions, e.g. image-to-scene or image-to-action. below, in section 6, we also describe a data augmentation and generation experiment, to merge the three axes into more 'narrativelike' descriptions of images. we provide baselines for this task by fine-tuning three models, namely git @xcite , blip @xcite , and clipcap @xcite on each separate axis. all the baselines were trained for a maximum of 10 epochs using a learning rate of 5e-5, adam optimizer, and halfprecision (fp16). a second observation, consistent with the analysis presented in earlier sections, is that on all metrics, models fine-tuned to generate rationale-based descriptions receive lower scores. we hypothesise that this is due in part to the greater variability in this axis, and to its inherent difficulty, as reflected in lower confidence scores. future work could leverage these scores as additional signal in fine-tuning models on captions that require more inference, compared to more descriptive ones. ## data augmentation and narrative generation we now describe how we extend the dataset to combine the three axes to compose a short 'narrative', which describes the scene, action and rationale in tandem. we call this new dataset hl narratives. to do this, we leverage the individual axes and synthesise this part of the data using a pre-trained language model. since scenes, actions, and rationales were elicited individually in a visually grounded and controlled setting, a synthesised version of the three individual captions should also be true of the image to the same extent (modulo the variations in confidence that we observe). ## further uses of the hl dataset we envision a wide set of use cases and tasks enabled by the hl dataset. git (pre): a group of people on the beach git (ft): people enjoying sunbathing, the picture was taken on the beach and are going to have fun and entertainment git (pre): two girls looking at their cell phones git (ft): they are reading a text message outside on the street, waiting for their friend. v&l generative tasks our captions support image captioning generation tasks which encompass a broader range of visually grounded linguistic descriptions than the highly object-centric, 'conceptual' descriptions which dominate the captioning literature @xcite . moreover, the decomposition along three axes can be exploited to compose narratives of the image, as in image paragraph generation @xcite and visual storytelling @xcite . they can be used in combination with the question each axis corresponds to, in order to generate micro-dialog scenarios. we would also argue that the high-level captions are also more natural and human-like, since they were collected without enforcing any restriction on the content to be described. given that the images are also aligned with object-centric captions, it is possible to envisage a scenario in which a model is trained to generate high-level captions, which are 'explained' or justified with reference to lowlevel, object-centric properties (see @xcite , for some work in this direction). in this way, the dataset can be leveraged to provide captions and explanations. furthermore, the confidence scores serve for the identification of hard samples in the data, both for evaluation purposes and to provide additional training signals, as recently shown by @xcite . multimodal grounding hl dataset is also a useful resource to benchmark the grounding capabilities of large pre-trained v&l models. along these lines, @xcite study the capability of v&l models to understand scene descriptions in zero-shot settings, finding that only largescale pre-trained v&l models have enough generalization capabilities to handle unseen high-level scene descriptions. @xcite analyse the impact of exposure to high-level scene descriptions on multimodal representations in models pretrained on object-centric captions. they show that exposure to high-level concepts mainly affects the model's attentional resource allocation over the visual input, even though the low-level concepts learned during pre-training provide enough signal to support and easily adapt to scene descriptions during fine-tuning. this is also supported by @xcite who find that low-level concepts are needed to learn higher-level concepts, though this does not hold in the other direction. ## conclusions in this paper, we introduced the high-level (hl) dataset. we extended 14,997 images from the popular coco dataset with 134,973 human-annotated high-level descriptions systematically collected over three axes: scene, action, and rationale. we aligned high-level captions with object-centric captions and we provided human-collected confidence scores to measure the degree of commonsense expressed in the high-level captions. we also provided baseline results on generating captions for individual axes, as well as synthesised narrative captions by combining these three high-level axes of description. differently from current v&l captioning datasets, the high-level captions capture the human interpretation of the scene allowing for inference and expectations. we discussed how they can be used also in combination with low-level captions to improve research in visual commonsense reasoning and multimodal grounding of visual concepts into linguistic expressions and for generative tasks, hoping to foster future research in this direction. we ask two postgraduate students experts in linguistics to correct grammatical errors in a sample of 9900 captions, 900 of which are shared between the two experts. they are shown the image-caption pairs and they are asked to edit the caption whenever they identify a grammatical error. the most common errors reported by the annotators are: in order to quantify the extent to which the corrected captions differ from the original ones, we compute the levenshtein distance @xcite between them. we observe that 22.5% of the sample have been edited and only 5% with a levenshtein distance greater than 10. this suggests a reasonable level of grammatical quality overall, with no substantial grammatical issues. this can also be observed from the levenshtein distance distribution reported in figure 11 . moreover, the human evaluation is quite reliable as we observe a moderate inter-annotator agreement (α = 0.507, @xcite ) computed over the shared sample.