Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge (2022)

Abstract: Cognitively plausible visual dialogue models should keep a mental scoreboard of shared established facts in the dialogue context. We propose a theory-based evaluation method for investigating to what degree models pretrained on the VisDial dataset incrementally build representations that appropriately do scorekeeping. Our conclusion is that the ability to make the distinction between shared and privately known statements along the dialogue is moderately present in the analysed models, but not always incrementally consistent, which may partially be due to the limited need for grounding interactions in the original task.

## related literature representing dialogue context implicitly as the continuous hidden states of neural networks trained in an end-to-end fashion has been a prevailing practice since the works of @xcite , @xcite and @xcite . this paradigm also enables multimodal input like images to be easily integrated @xcite . however, there is evidence that the human ability of collaborative grounding still lacks in such models, in part due to the limitations of training regimes and datasets @xcite . we witness extensive efforts to look into how these models encode and make use of dialogue history, capture salient information and produce visually grounded representations @xcite @xcite . the analysis and evaluation of current dialogue models (as @xcite , @xcite , @xcite , @xcite , @xcite , inter alia) often rely on diagnostic classifiers @xcite and probing tasks @xcite , common tools to examine whether representations built by neural networks encode linguistic information. another purposeful area of research on dialogue revolves around inference. @xcite discuss conversation entailment, i.e. determining whether a conversation discourse entails a hypothesis. annotating or generating entailments, contradictions and neutral statements in dialogue datasets is usual in recent works @xcite @xcite . with insights from these three pillars, we propose a probing task for scorekeeping @xcite on visual dialogues, formalised in the next section. ## problem statement based on the premise that humans keep a mental scoreboard of presupposed propositions and permissible courses of action as a function of what has been stated in a conversation @xcite and on the public/private dichotomy discussed in ginzburg ( 2012 ), we propose a formalisation for the "kinematics of scorekeeping" @xcite on visdial. each dialogue in the visdial dataset is a tuple @xmath0, q, a, t, p ) representing an interaction between a questioner q and an answerer a. they exchange turns t , which establish propositions p , about a scene depicted in an image i. a sees i, but q does not. both are provided with a caption k, which for simplicity we take to be the first turn of a, t 0 = k; other turns comprise a question and an answer, t @xmath1, a i ), so that @xmath2 probing task and model. we design a classification task to examine whether the continuous representations of pretrained visual dialogue models incrementally encode information about the scoreboard represented by s. the probing classifier is a function f : p d × r d,@xmath3, where p d is the set of propositions in a dialogue d, r is the space of hidden representations of a visual dialogue encoder and c are the scoreboard classes. based on the probing classifier architecture in hewitt and liang (2019), we approximate f as a neural network which maps a dialogue representation r concatenated to a continuous representation z of a proposition to a vector v with a probability distribution over classes, @xmath4; z])) (bias term omitted), as illustrated in figure 1 . the class is then predicted with the argmax function. ## data visual dialogues and encoders. we use the vis-dial dataset v.1.0 @xcite and the three q and a encoders (rl_div, sl and iccv_rl) from @xcite and @xcite . the first work implemented an end-to-end model to train a and q using reinforcement learning. the latter is a follow-up study that adds an auxiliary objective function to encourage q to ask more diverse questions. 4 the visdial training set contains images from the ms coco dataset @xcite . proposition embeddings z are built with @xcite . ## results ## scope and limitations the results on this paper comprise three visual dialogue models trained using a similar setting on the same dataset. the preprocessing steps used by these models replace some tokens by a unk token and truncate long captions, which prevents some information to become shared as assumed. further investigation with other models and data is necessary in future research in order to support more general conclusions. the results also rely on the capabilities of the classifier. although we performed hyperparameter search, the probing classifier does not completely overfit the full training dataset, thus other architectures and hyperparatemeters can be further investigated. the rule-based generation of propositions has limitations. it cannot generate propositions for all qa pairs and some rules end up not always yielding grammatically valid sentences, for instance because of countable/uncountable nouns, detection of singular/plural forms and mistakes and typos deriving from the dialogues themselves. besides, spuri-ous patterns deriving from the implemented rules or other confounds and inherent biases (e.g. figure 2 ) may exist and be predictive of the classes, which could be captured by the probing classifier and influence (likely overestimating) the results. enforcing a balance on the training set in terms of true/false to a solves one source of bias but causes its distribution to differ from the validation and test set. the test set also has a different distribution because of its varying number of turns. finally, while the assumptions proposed in section 3 are necessary idealizations for using visdial for this task, they simplify essential aspects of dialogues, e.g. the uncertainty about a fact actually being shared, memory limitations and the many kinds of inference that are used in the accommodation of shared knowledge, such as presuppositions, implicatures, entailments and implicit information. our method cannot capture background knowledge not explicitly stated in dialogue turns. 8 ## conclusion we have proposed a novel way to do theory-based evaluation of visual dialogue models. using diagnostic propositions, we investigated to what degree neural network visual dialogue models incrementally build up representations that are appropriate to do scorekeeping of shared commitments throughout a dialogue. the evaluated models trained on visdial capture part of this process, but not always consistently, possibly because this ability is not an elementary component of the training regime. the relatively impoverished nature of the original task in terms of coordination phenomena can also limit the capability of models to build good dialogue representations (schlangen, 2019). future work should extend the evaluation to other models and reflect on how better and ecologically valid diagnostic datasets for visual dialogues can be constructed.