C omp G uess W hat?!: A Multi-task Evaluation Framework for Grounded Language Learning (2020)

Abstract: Approaches to Grounded Language Learning are commonly focused on a single task-based final performance measure which may not depend on desirable properties of the learned hidden representations, such as their ability to predict object attributes or generalize to unseen situations. To remedy this, we present GroLLA, an evaluation framework for Grounded Language Learning with Attributes based on three sub-tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zero-shot evaluation. We also propose a new dataset CompGuessWhat?! as an instance of this framework for evaluating the quality of learned neural representations, in particular with respect to attribute grounding. To this end, we extend the original GuessWhat?! dataset by including a semantic layer on top of the perceptual one. Specifically, we enrich the VisualGenome scene graphs associated with the GuessWhat?! images with several attributes from resources such as VISA and ImSitu. We then compare several hidden state representations from current state-of-the-art approaches to Grounded Language Learning. By using diagnostic classifiers, we show that current models’ learned representations are not expressive enough to encode object attributes (average F1 of 44.27). In addition, they do not learn strategies nor representations that are robust enough to perform well when novel scenes or objects are involved in gameplay (zero-shot best accuracy 50.06%).

## introduction several grounded language learning tasks have been proposed to capture perceptual aspects of language @xcite @xcite . however, the advances in this field have been primarily driven by the final performance measures and less on the grounding capability of the models. in fact, in some cases, high-performance models exploit dataset biases to achieve high scores on the final task @xcite . in turn question answer 1 is it an appliance? yes 2 does it blend? no 3 is it the oven? no 4 is it the microwave? yes left-of left-of microwave oven food mixer is appliance has buttons is white is appliance has buttons is white is appliance has buttons usedto_blend is white legend abstract situated the literature, several methods have been proposed to analyse what kind of information is captured by neural network representations @xcite . most of these works examine the hidden state representations learned by models trained on only textual data. however, many aspects of human semantic representations are grounded in perceptual experience @xcite . this paper explores the idea that visually grounded representations ought to be a result of systematic composition of grounded representations @xcite . for instance, the understanding of the word "microwave" is grounded in perception of objects with specific attributes such as shape, colour, and size -see figure 1 for an example. therefore, investigating whether the representations learned by a model exhibit forms of attribute composition is beneficial for assessing model interpretability and generalisation. in this work, we propose grolla -a multitask evaluation framework for grounded language learning with attributes that expands a goaloriented evaluation -based on the standard final task measure, with two auxiliary tasks: 1) object attribute prediction (ap), and 2) zero-shot evaluation (zs). the attribute prediction task is designed to evaluate the extent to which the model's latent representations associated with objects are useful for predicting their attributes. the prediction performance on this task can be related to a degree of compositionality of the learned representations. we adopt a behavioural, i.e., task-driven, approach to assessing aspects of compositionality for visually grounded representations, whereby the extent to which a representation is compositional depends on: (a) its ability to predict object attributes, and (b) its ability to generalise to novel contributions of object attributes. to support (b), we design a zeroshot evaluation that measures the extent to which the learned representations can be reused in a task involving objects unseen during training. by optimising for both the final end-goal measure as well as the auxiliary tasks, we aim to drive the design of models that can solve the task more reliably and whose representations are easier to interpret as a result of being a composition of visual attributes. this paper presents three main contributions: (1) we define grolla -a multi-task evaluation framework for grounded language learning that augments the final end-goal measure(s) with auxiliary tasks aimed at assessing the degree of attribute grounding of the model's representations; compguesswhat?! is a benchmark of 65, 700 dialogues (see section 3). it is based on guess-what?! (de vries et al., 2017) dialogues and enhanced by including object attributes coming from resources such as visa attributes @xcite , visualgenome @xcite and imsitu @xcite . ## evaluation framework our evaluation framework for grounded language learning tasks is based on three different sub-tasks: 1) goal-oriented evaluation; 2) object attribute prediction evaluation; 3) zero-shot evaluation. ## related work text-only multi-task evaluation in recent years progress in nlp has been driven by multi-figure 3 : example gameplay in the out-of-domain scenario of the two best performing systems gdse-rl and devries-rl. the models have to play the game considering the crocodile as target object. this is a zero-shot scenario because the object category crocodile is not among the mscoco categories. ## conclusions & future work we proposed compguesswhat?! as an implementation of grolla, a multi-task evaluation framework for grounded language learning with attributes. we found that the best performing model achieves a grolla score of 50.06%; notably this model's out-of-domain accuracy is under 30%, as compared to the human performance on the original guesswhat?! dataset of 90. @xcite . clearly, even models with high in-domain gameplay success rates still have difficulty generalising to new scenarios. in the following, we discuss insights gained from the evaluation and new research directions for this task. the attribute prediction task shows that model representations are not able to accurately recover attribute representations. we argue that this result calls for new approaches to exploiting and representing textual and visual data. we believe that models should be equipped with a co-grounding operator that fuses the textual and visual modalities. for instance, in the context of compguesswhat?!, it would be used to learn a representation for the current turn that is influenced by both the language and visual modality. compguesswhat?! requires models to learn to combine the co-grounded information provided for every turn. therefore, we propose that compguesswhat?! represents a benchmark dataset for evaluating the design of such an attribute compositionality operator that would be a possible implementation of compositionality à la @xcite . in this work, we have shown how our multi-task evaluation framework can be be applied to guess-what?!. however, the same framework could be applied to other multi-modal tasks. for example, in image captioning, the goal-oriented evaluation would be the textual similarity metrics (e.g. bleu); the attribute-prediction task would use the decoder representation to predict the attributes of the objects in the image @xcite ; and the zero-shot setting could leverage the nocaps dataset @xcite . likewise, in the vision-and-dialog navigation task @xcite , the goal-oriented evaluation is the navigation task; attribute prediction is based on predicting the attributes of the hidden object when the agent decides it is in the correct room, and the zeroshot setting could evaluate model performance on novel combinations of rooms and object types. finally, from the evaluation presented here, it emerges that these models learn task-specific representations that do not generalise to unseen object categories. we hope that grolla and the compguesswhat?! data will encourage the implementation of learning mechanisms that fuse taskspecific representations with more abstract representations to encode attributes in a more compositional manner. in addition, we will use the compguesswhat?! image annotations to design a visual grounding evaluation to assess the ability of the model to attend to the correct objects during the turns of the dialogue.