Listener Model for the P hoto B ook Referential Game with CLIPS cores as Implicit Reference Chain (2023)

Abstract: PhotoBook is a collaborative dialogue game where two players receive private, partially-overlapping sets of images and resolve which images they have in common. It presents machines with a great challenge to learn how people build common ground around multimodal context to communicate effectively. Methods developed in the literature, however, cannot be deployed to real gameplaysince they only tackle some subtasks of the game,and they require additional reference chains inputs, whose extraction process is imperfect. Therefore, we propose a reference chain-free listener modelthat directly addresses the game’s predictive task, i.e., deciding whether an image is shared with partner. Our DeBERTa-based listener model reads the full dialogue, and utilizesCLIPScore features to assess utterance-image relevance. We achieve >77% accuracy on unseen sets of images/game themes, outperforming baseline by >17 points.

## introduction photobook (haber et al., 2019) is a collaborative dialogue game of two players. in a game round, each player receives 6 images of an identical themethe two largest objects in all images share the same categories, e.g., dog, car, etc. the players have some of their images in common. their goal is to communicate through text dialogue, and individually mark 3 privately highlighted images as either common (i.e., shared with partner) or different. a full game lasts 5 rounds. after each round, some of each player's images are replaced with different ones under the same theme. images may reappear in later rounds after being swapped out. this game setup encourages building and leveraging common ground with multimodal contexts, which humans are known to do to facilitate conversation (clark and wilkes-gibbs, 1986; brennan and clark, 1996). models proposed in past works on the dataset (haber et al., 2019; takmaz et al., 2020) are unable to realistically play the game due to several reasons: (i) they only address subtasks in the game whose time span is one utterance, rendering it unnecessary for the models to keep track of the entire game's, or round's, progress; (ii) the models operate on additional input of reference chains, i.e., past utterances referring to each image, whose (rule-based) extraction process is imperfect and hence complicates learning and evaluation; and, (iii) utterances outside of reference chains, e.g., 'i don't have that one', may also be important pieces of information. to address the drawbacks above, we propose a full (i.e., able to play real games), reference chainfree listener model, which accepts all dialogue utterances of a round foot_0 and the 6 context images, and predicts whether the 3 target (highlighted) images are common/different. our listener model is based on a pretrained deberta transformer (he et al., 2021). to incorporate visual context, clipscores (hessel et al., 2021) between each utterance and the 6 given images are infused with deberta hidden states. we employ clipscore as it offers strong prior knowledge about the relevance of an utterance to each of the 6 images, which may serve as a soft, implicit version of reference chain used in previous studies. also, we chose deberta since it is one of the top performers in the superglue benchmark (sarlin et al., 2020) which provides a reasonablysized (∼100m parameters) version to suit our purpose and computation resources. we further devise a label construction scheme to create dense learning signals. our model scores @xmath0. our code is available at github.com/ slseanwu/photobook-full-listener. ## related work in typical collaborative dialogue tasks, two agents (i.e., players) hold incomplete or partially overlapping information and communicate through text to reach a predefined goal. the task-oriented setup enables simple evaluation for dialogue systems via task success rate, instead of resorting to costly human evaluation. tasks and datasets proposed in the literature focus either on set logic (he et al., 2017), image understanding (de vries et al., 2017; haber et al., 2019), or spatial reasoning (udagawa and aizawa, 2019). they challenge dialogue systems to process multiple modalities, discard irrelevant information, and build common ground. researchers have utilized graph neural networks (he et al., 2017), vision-and-language transformers (lu et al., 2019; tu et al., 2021), and pragmatic utterance generation (frank and goodman, 2012; fried et al., 2021) to tackle the tasks. @xcite to our knowledge, there has not been a system that fully addresses the photobook task. it may be particularly challenging due to the setup with multiple highly similar images and an unbounded set of information (e.g., scene, actions) the images may contain. previous photobook works targeted two subtasks: reference resolution (haber et al., 2019; takmaz et al., 2020) and referring utterance generation (takmaz et al., 2020). the former resolves which of the 6 context images an utterance is referring to, while the latter generates an informative utterance for a pre-selected image. pro- as for the reference resolution task, haber et al. (2019) employed lstm encoders. one (query) encoder takes a current dialogue segment, while the other (i.e., context encoder) receives the 6 images' resnet features, and the associated reference chain segments. @xcite dot products between query encoder output and 6 context encoder outputs are taken to predict the image the current segment refers to. takmaz et al. (2020) largely kept the setup, but they used bert (devlin et al., 2019) embeddings and contextualized utterances via weighted averaging instead of lstms. takmaz et al. (2020) claimed an 85% reference resolution accuracy, but they also reported an 86% precision foot_3 on reference chain extraction, making it difficult to conclude whether prediction errors are due to model incompetence, or incorrect input data/labels. (we find that some parts of extracted reference chains either point to the wrong image or provide no information at all. foot_4 ) yet, we do agree that keeping track of which images have been referred to is vital for the game. therefore, we aim to build a full listener model that does not depend on explicit reference chains, but gathers such information from implicit hints given by an image-text matching model, i.e., clip (radford et al., 2021). ## experiments and results our listener model is trained with the maximum likelihood estimation (mle) loss function: ## conclusions and future work in this paper, we first discussed why it is difficult to deploy existing reference chain-dependent pho-tobook models to real gameplay, and demonstrated that clipscore's image-text matching capability may provide implicit reference chains to the task. we then developed a novel listener model that is reference chain-free, and able to realistically play the game given text dialogue and the set of context images, just as what human players see. the model is built on a deberta transformer backbone, and brings in visual context by infusing utterance-level clipscores with its hidden states. on the newly proposed full listener task, i.e., predicting whether an image is shared with partner, our model achieves 77∼84% accuracy on unseen sets of images, surpassing baseline (takmaz et al., 2020) by over 17 points. ablation studies also showed that feeding clipscores and imposing dense learning signals are both indispensable to our model's success. future studies may leverage parameter-efficient transfer learning (he et al., 2022; houlsby et al., 2019; hu et al., 2022; perez et al., 2018) to cope with image data scarcity of photobook (and potentially other datasets and tasks). it is also interesting to develop a speaker model that uses temporary beliefs from our listener model and takes pragmatics (frank and goodman, 2012; fried et al., 2021) into account to generate informative responses. pairing such a model with our listener model may complete the collaborative dialogue task end-to-end.