Less Descriptive yet Discriminative: Quantifying the Properties of Multimodal Referring Utterances via CLIP (2022)

Abstract: In this work, we use a transformer-based pre-trained multimodal model, CLIP, to shed light on the mechanisms employed by human speakers when referring to visual entities. In particular, we use CLIP to quantify the degree of descriptiveness (how well an utterance describes an image in isolation) and discriminativeness (to what extent an utterance is effective in picking out a single image among similar images) of human referring utterances within multimodal dialogues. Overall, our results show that utterances become less descriptive over time while their discriminativeness remains unchanged. Through analysis, we propose that this trend could be due to participants relying on the previous mentions in the dialogue history, as well as being able to distill the most discriminative information from the visual context. In general, our study opens up the possibility of using this and similar models to quantify patterns in human data and shed light on the underlying cognitive mechanisms.

## introduction during a conversation, speakers can refer to an entity (e.g., the girl in fig. 1 ) multiple times within different contexts. this has been shown to lead to subsequent referring expressions that are usually shorter and that show lexical entrainment with previous mentions @xcite . this trend has been confirmed in recent vision-and-language (v&l) datasets @xcite @xcite : referring utterances become more compact (i.e., less descriptive), and yet participants are able to identify the intended referent (i.e., they remain pragmatically informative). several approaches @xcite @xcite have tackled the generation of image captions from the perspective of pragmatic informativity; @xcite have compared the figure 1 : referring utterance chain from photobook @xcite . the chain has 4 ranks (4 references to the target image, in red outline). for simplicity, only the 5 distractor images from rank 1 are shown. in this work, we use a transformer-based pretrained multimodal model to study the interplay between descriptiveness and discriminativeness in human referring utterances produced in dialogue. due to their unprecedented success in numerous tasks, pretrained v&l models-such as lxmert @xcite , @xcite , uniter @xcite and align @xcite -have recently attracted a lot of interest aimed at understanding the properties and potential of their learned representations as well as the effect their architectures and training setups have @xcite . these include probing such models in a zero-shot manner, i.e., without any specific fine-tuning (hendricks and nematzadeh, 2021; @xcite ; quantifying the roles of each modality @xcite ; inspecting attention patterns @xcite ; and evaluating their learned multimodal representations against human judgments @xcite . we focus on one model: contrastive language-image pre-training (clip, @xcite , which learns via contrasting images and texts that can be aligned or unaligned with each other. this contrastive objective makes clip particularly suitable for modelling referential tasks that inherently include such comparisons. here, we use clip to gain insight into the strategies used by humans in sequential reference settings, finding that although the descriptiveness of referring utterances decreases significantly, the utterances remain discriminative over the course of multimodal dialogue. the code to reproduce our results is available at https://github.com/ecekt/ clip-desc-disc . ## data we focus on photobook (pb; @xcite , a dataset of multimodal task-oriented dialogues where players aim to pick the images they have in common without seeing each other's visual contexts (which consist of 6 images coming from the same domain). the game is played over several rounds in which the previously seen images reappear in different visual contexts, giving the players an opportunity to refer to such images again. as a result, chains of utterances referring to a single image are formed over the rounds as the players build common ground. see fig. 1 for a simplified representation of a chain. 1 in total, pb consists of 2,500 games, 165k utterances, and 360 unique images from coco @xcite . all our experiments are conducted on a subset of 50 pb games with manually annotated referring utterances, which contains 364 referential chains about 205 unique target images. we refer to this subset as pb-gold. 2 although a dataset of automatically-extracted chains using all pb data is also available @xcite , as reported by the authors these chains may contain errors. we therefore opt for using the smaller but higherquality pb-gold subset since we are interested in analysing human strategies. given that we use a pretrained model without fine-tuning, experimenting with large amounts of data is not a requisite. pb-gold's chains contain 1,078 utterances, i.e., 2.96 utterances per chain on average (min 1, max 4). we henceforth use the term 'rank' to refer to the position of an utterance in a chain. the average token length of utterances is 13.34, 11.03, 9.23, and 7.82, respectively, for ranks 1, 2, 3, and 4. 3 this decreasing trend, which is statistically significant at @xmath0.01 with respect to independent samples t-tests between the ranks, is in line with the trend observed in the whole dataset @xcite . pb-gold's vocabulary consists of 926 tokens. ## model we use clip @xcite , a model pretrained on a dataset of 400 million image-text pairs collected from the internet using a contrastive objective to learn strong transferable vision representations with natural language supervision. 4 in particular, we employ the vit-b/32 version of clip, which utilizes separate transformers to encode vision and language @xcite @xcite . as the model learns to align images and texts, this enables zero-shot transfer to various v&l tasks such as image-text retrieval and image classification and even certain non-traditional tasks in a simple and efficient manner @xcite @xcite @xcite . this makes it an intriguing tool to investigate the properties of visually grounded referring utterances. in this work, we freeze clip's weights and do not fine-tune the model or perform prompt engineering, since we aim to exploit the model's pretrained knowledge for the analysis of human referring strategies. ## descriptiveness in our first experiment, we investigate the degree of descriptiveness exhibited by referring utterances in the photobook game, i.e., the amount of information they provide about the image out of context. we consider each target image and corresponding referential utterance at a given rank in isolation, i.e., without taking into account the other competing images nor the dialogue history. we quantify descriptiveness as the alignment between an utterance and its image referent using clipscore @xcite , assuming that a more descriptive utterance will attain a higher score. for all the target image-utterance pairs in the chains of pb-gold, we use clip to obtain a vector t representing the utterance and a vector v representing the image. clipscore is then computed as the scaled cosine similarity between these two vectors, with range [0, 2.5]: foot_4 clipscore(t, v) = 2.5 * max(cos(t, v), 0). we compute the average clipscore per rank over the whole pb-gold dataset. results. we find that earlier utterances are better aligned with the target image features and that there is a monotonically decreasing trend over the 4 ranks (fig. 4 , blue bars). the differences between all pairs of ranks are statistically significant (according to independent samples t-tests, @xmath1.01), except for the comparison between the last 2 ranks (@xmath2.05). since earlier referring utterances tend to be longer (see sec. 2), we check to what extent length may be a confounding factor. we find that there is only a weak correlation between token length and clipscore (spearman's ρ = 0.29, @xmath3.001). we compare these results on photobook with text-to-image alignment computed with the same method on two other datasets: (1) coco @xcite ), foot_5 which includes 5 captions per image provided independently by different annotators as shown in fig. 2 ; here we do not expect to find significant differences in the level of descriptiveness across the captions, and (2) image description sequences (ids, ilinykh et al., 2019) 7 where one participant describes an image incrementally as shown in fig. 3 , by progressively adding sentences with further details; here we do expect a similar pattern to photobook, albeit for different reasons (because participants add less salient information; @xcite . fig. 4 shows that these expectations are confirmed. according to clip, coco captions (green bars) are more descriptive than ids descriptions and pb referring utterances, and are equally aligned with the image across 'ranks' (the order is arbitrary in this case). in contrast, ids incremental descriptions (yellow bars) are intrinsically ordered and show a significant decreasing trend similar to pb. ## discriminativeness in order for a listener to select the target image among distractor images, a referring utterance should be discriminative in its visual context. our results in the previous section show that descriptiveness decreases over time-what is the trend regarding discriminativeness? to address this question, in our second experiment we use clip from the perspective of reference resolution. we focus on local text-to-image alignment, initially ignoring the previous dialogue history. to this end, we feed clip a single referring utterance together with the visual context of the speaker who produced that utterance. clip yields softmax probabilities for each image contrasted with the single text. as a metric, we use accuracy: 1 if the target image gets the highest probability; 0 otherwise. results. the overall accuracy is 80.15%, which is well above the random baseline of 16.67%. in fig. 5 , we break down the results per rank (blue bars). a 4 × 2 chi-square test (4 ranks vs. correct/incorrect) did not yield significant differences in accuracy between the ranks, @xmath4.05. thus, although descriptiveness decreases over time, discriminativeness is not significantly affected. an analysis of the entropy of the softmax distributions reveals that entropy increases monotonically over the ranks (this difference is statistically significant according to an independent samples t-test between ranks 1 and 4; h 1 = 0.62, h 4 = 0.79, @xmath5.01). that is, the model is more uncertain when trying to resolve less descriptive utterances. there is indeed a negative correlation between entropy and clipscore computed between the target image and the corresponding utterance (spearman's ρ = -0.5, @xmath6.001). ## analysis how do participants manage to maintain discriminativeness while decreasing descriptiveness? do they rely on the previous mentions present in the dialogue history? do they refine their referring strategy by distilling the most discriminative information in a given context? ## conclusion we used a pre-trained multimodal model claimed to be a reference-free caption evaluator, clip @xcite , to quantify descriptiveness and discriminativeness of human referring utterances within multimodal dialogues. we showed that (i) later utterances in a dialogue become less descriptive in isolation while (ii) remaining similarly discriminative against a visual context. we found that the addition of dialogue history helps decrease and control the entropy of resolution accuracy even when the speakers produce less descriptive referring utterances. in addition, we found that the proportion of discriminative words increases over the ranks. these suggest that participants playing the photobook game @xcite show a tendency towards distilling discriminative words and utilize the dialogue history to keep task performance stable over the dialogue. this outcome resonates with the findings by @xcite who observe that photobook dialogue participants tend to limit fluctuations in the amount of information transmitted within reference chains, in line with uniform information density principles (e.g., @xcite . interestingly, future work could explore novel ways of incorporating the clip model or its representations into a reference resolution or generation model embedding dialogue history and visual context to obtain human-like outcomes.