A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions (2020)

Abstract: Recent models achieve promising results in visually grounded dialogues. However, existing datasets often contain undesirable biases and lack sophisticated linguistic analyses, which make it difficult to understand how well current models recognize their precise linguistic structures. To address this problem, we make two design choices: first, we focus on OneCommon Corpus (CITATION), a simple yet challenging common grounding dataset which contains minimal bias by design. Second, we analyze their linguistic structures based on spatial expressions and provide comprehensive and reliable annotation for 600 dialogues. We show that our annotation captures important linguistic structures including predicate-argument structure, modification and ellipsis. In our experiments, we assess the model’s understanding of these structures through reference resolution. We demonstrate that our annotation can reveal both the strengths and weaknesses of baseline models in essential levels of detail. Overall, we propose a novel framework and resource for investigating fine-grained language understanding in visually grounded dialogues.

## introduction visual dialogue is the task of holding natural, often goal-oriented conversation in a visual context @xcite . this typically involves two types of advanced grounding: symbol grounding @xcite , which bridges symbolic natural language and continuous visual perception, and common grounding @xcite , which refers to the process of developing mutual understandings through successive dialogues. as noted in @xcite ; @xcite , the continuous nature of visual context introduces challenging symbol grounding of nuanced and pragmatic expressions. some further incorporate par-tial observability where the agents do not share the same context, which introduces complex misunderstandings that need to be resolved through advanced common grounding @xcite . despite the recent progress on these tasks, it remains unclear what types of linguistic structures can (or cannot) be properly recognized by existing models for two reasons. first, existing datasets often contain undesirable biases which make it possible to make correct predictions without recognizing the precise linguistic structures @xcite @xcite . second, existing datasets severely lack in terms of sophisticated linguistic analysis, which makes it difficult to understand what types of linguistic structures exist or how they affect model performance. to address this problem, we make the following design choices in this work: as shown in figure 1 , we consider spatial expressions as predicates with existing markables as their arguments. we distinguish the argument roles based on subjects and objects foot_1 and annotate modifications based on nuanced expressions (such as slightly). by allowing the arguments to be in previous utterances, our annotation also captures argument ellipsis in a natural way. in our experiments, we focus on reference resolution to study the model's comprehension of these linguistic structures. since we found the existing baseline to perform relatively poorly, we propose a simple method of incorporating numerical constraints in model predictions, which significantly improved its prediction quality. based on our annotation, we conduct a series of analyses to investigate whether the model predictions are consistent with the spatial expressions. our main finding is that the model is adept at recognizing entity-level attributes (such as color and size), but mostly fails in capturing inter-entity relations (especially placements): using the terminologies from @xcite , the model can recognize the what but not the where in spatial language. we also conduct further analyses to investigate the effect of other linguistic factors. overall, we propose a novel framework and re-source for conducting fine-grained linguistic analyses in visually grounded dialogues. all materials in this work will be publicly available at https: //github.com/alab-nii/onecommon to facilitate future model development and analyses. ## related work linguistic structure plays a critical role in dialogue research. from theoretical aspects, various dialogue structures have been studied, including discourse structure @xcite , speech act @xcite and common grounding @xcite . in dialogue system engineering, various linguistic structures have been considered and applied, including syntactic dependency @xcite , predicate-argument structure (pas) @xcite , ellipsis @xcite , intent recognition @xcite , semantic representation/parsing @xcite and frame-based dialogue state tracking @xcite . however, most prior work focus on dialogues where information is not grounded in external, perceptual modality such as vision. in this work, we propose an effective method of analyzing linguistic structures in visually grounded dialogues. recent years have witnessed an increasing attention in visually grounded dialogues @xcite @xcite . despite the impressive progress on benchmark scores and model architec-tures @xcite @xcite @xcite @xcite @xcite , there have also been critical problems pointed out in terms of dataset biases @xcite @xcite @xcite @xcite ) which obscure such contributions. for instance, @xcite points out that existing dataset of reference resolution may be largely solvable without recognizing the full referring expressions (e.g. based on object categories only). to circumvent these issues, we focus on onecommon corpus where the visual contents are simple (exploitable categories are removed) and well-balanced (by sampling from uniform distributions) to minimize dataset biases. although various probing methods have been proposed for models and datasets in nlp @xcite @xcite @xcite , fine-grained analyses of visually grounded dialogues have been relatively limited. instead, @xcite proposed a diagnostic dataset to investigate model's language understanding: however, their dialogues are generated artificially and may not reflect the true nature of visual dialogues. @xcite also acknowledges the importance of linguistic analysis but only dealt with coarse-level features that can be computed automatically (such as dialogue topic and diversity). most similar and related to our research are @xcite ; udagawa and aizawa (2020), where they conducted additional annotation of reference resolution in visual dialogues: however, they still do not capture more sophisticated linguistic structures such as pas, modification and ellipsis. finally, spatial language and cognition have a long history of research @xcite . in computational linguistics, @xcite developed the task of spatial role labeling to capture spatial information in text: however, they do not fully address the problem of annotation reliability nor grounding in external visual modality. in computer vision, the visualgenome dataset @xcite provides rich annotation of spatial scene graphs constructed from raw images, but not from raw dialogues. @xcite ; @xcite also worked on modelling spa-tial prepositions in single sentences. to the best of our knowledge, our work is the first to apply, model and analyze spatial expressions in visually grounded dialogues at full scale. ## discussion and conclusion in this work, we focused on the recently proposed onecommon corpus as a suitable testbed for fine-grained language understanding in visually grounded dialogues. to analyze its linguistic structures, we proposed a novel framework of annotating spatial expressions in visual dialogues. we showed that our annotation can be conducted reliably and efficiently by leveraging referring expressions prevalent in visual dialogues, while capturing important linguistic structures such as pas, modification and ellipsis. although our current analysis is limited to this domain, we expect that upon appropriate definition of spatial expressions, argument roles and canonicalization, the general approach can be applied to a wider variety of domains: adapting and validating our approach in different domains (especially with more realistic visual contexts) are left as future work. 6 left/right for x-value, above/below for y-value, lighter/darker for color and smaller/larger for size. secondly, we proposed a simple idea of incorporating numerical constraints to improve exophoric reference resolution. we expect that a similar approach of identifying and incorporating semantic constraints (e.g. coreferences and spatial constraints) is a promising direction to improve the model's performance even further. finally, we demonstrated the advantages of our annotation for investigating the model's understanding of visually grounded dialogues. our tests are completely agnostic to the models and only require referent predictions made by each model. by designing simple tests like ours (subsubsection 4.2.1/4.2.2), we can diagnose the model's performance at the granularity of canonical attributes/relations under consideration: such analyses are easy to extend (by adding more tests) and critical for verifying what capabilities current models have (or do not have). based on further analyses (subsubsection 4.2.3), we also revealed various linguistic structures that affect model performance: we expect that capturing and studying such effects will be essential for advanced model probing in visual dialogue research. overall, we expect our framework and resource to be fundamental for conducting sophisticated linguistic analyses of visually grounded dialogues. koichiro yoshino, shinsuke mori, and tatsuya kawahara. 2011. spoken dialogue system based on information extraction using similarity of predicate argument structures. in proceedings of the sig-dial 2011 conference, pages 59-66. association for computational linguistics. xintong yu, hongming zhang, yangqiu song, yan song, and changshui zhang. 2019. what you see is what you get: visual pronoun coreference resolution in dialogues. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 5123-5132, hong kong, china. association for computational linguistics. sina zarrieß, julian hough, casey kennington, ramesh manuvinakurike, david devault, raquel fernández, and david schlangen. 2016. pentoref: a corpus of spoken references in task-oriented dialogues. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 125-131, portorož, slovenia. european language resources association (elra). zilong zheng, wenguan wang, siyuan qi, and song-chun zhu. 2019. reasoning visual dialogs with structural and partial observations. in computer vision and pattern recognition (cvpr), 2019 ieee conference on. here, we show additional examples of our spatial expression annotation. in figure 4 , we show an example dialogue annotated with spatial attributes (colored in red). since our goal is not to achieve strict inter-annotator agreement but to conduct efficient and useful analysis, we allow certain flexibility in determining the spans of spatial expressions: for instance, the coordinated spatial expression ("small and light") can be annotated as a single expression or as different expressions ("small and light"). copulas (is, being), articles (a, the), particles (to, with) and modifiers were allowed to be either omitted or included in spatial expressions. spans were allowed to be non-contiguous, but must be annotated at the token level and restricted to be within a single utterance. note that spatial attributes (tiny, light) in the first markable ("a lonely tiny light dot") are not annotated, since they are inside the markable and their spans and subjects are relatively obvious.