The VDG Challenge: Response Generation and Evaluation in Collaborative Visual Dialogue (2023)

Abstract: We propose the VDG Challenge: a shared task that addresses and benchmarks the task of utterance generation in collaborative visual dialogue. The task features two challenging datasets, an evaluation protocol and a tentative schedule. Our shared task will allow researchers to unravel problems of modelling multi-modal interaction and fit of the existing approaches in the NLP and NLG communities.

## introduction in the early 70s, the shrdlu system @xcite was a revolutionary development. many believed it had the ability to understand human language, as it was able to describe what it saw in an environment based on human queries. however, the illusion of intelligence of shrdlu was dispelled as it became clear that the system did not know, for example, what a "box" is. fast forward to today's world, we have made significant progress with more advanced language models. for instance, models like @xcite and various transformer-based architectures for language understanding and generation @xcite have demonstrated the ability to understand language to some extent. moreover, integrating language with other modalities has become essential in improving these models and making them more applicable to real-world scenarios @xcite . in fact, language-andvision research has been making steps in this direction @xcite , as its aim is to build language systems that can map language with the world @xcite . despite the progress made, research efforts in multi-modal nlp have primarily concentrated on more specific tasks, such as referring expression generation @xcite , image description generation @xcite , and visual question answering @xcite . this focus is understandable because modelling human-human or human-world interaction is a challenging task, and there are several reasons for this. firstly, human language use is highly dynamic, adaptive, and difficult to represent computationally. successful communication relies on understanding the context, which can encompass textual and multi-modal information, as well as grasping the speaker's intentions. deciding what to say, how to say it, and when to say it are fundamental aspects of effective communication that require a nuanced understanding of language and context. secondly, building a robust model requires high-quality dialogue data, which is challenging to gather and ensure that it possesses the properties observable in human language use. simply put, there are multiple components involved in modelling human dialogue, and they must all be integrated harmoniously to create a truly effective conversational model. our proposal, the visual dialogue generation (vdg) challenge, aims to create a platform that addresses the challenges in modelling multi-modal human-like situated dialogue @xcite . specifically, our setup revolves around a collaborative visual dialogue, where two participants are placed in an environment with individual visual scenes and are asked to solve a specific task through language interaction. within this setup, we focus on a particular task of next utterance generation, which is part of a broader communicative context. the primary goal of the challenge is to build and evaluate (neural) modelling proposals that can generate better responses given specific contexts. these contexts are defined as sets of previously generated utterances and visual scenes that collectively form a single language game. each language game may serve a different purpose, such as describing, asking, or clarifying. importantly, our aim is not to build a conversational agent capable of holding a full-scale dialogue with a human. instead, we narrow our focus to a single step: generating a response given a particular situation. by doing so, we can concentrate on examining the quality and value of the generated texts, which is important for building a model of conversation. the challenge will use two datasets: the cups @xcite and meetup @xcite , both of which are multi-modal and rich in various dialogue phenomena, such as clarification requests and turn-taking, crucial for a complete collaborative process @xcite ). in addition, cups corpus has data in two languages, english and swedish. while there have been a few proposed visual dialogue models and datasets @xcite , they suffer from rigidity and a lack of many phenomena frequently observed in natural human dialogues. our proposal aims to learn from better high-quality dialogue data, even though the datasets are relatively small in size. as a result, this challenge specifically focuses on transfer learning, learning from small data, and benchmarking the ability of existing generative models to generate responses in human-like multi-modal dialogues. an important feature of our data is that the dialogues were produced with specific (and different) tasks in mind, resulting in highquality interactions. this raises questions about how much interactive knowledge is shared between different contexts and domains, and how much of it is specific to certain situations. additionally, we aim to learn from the natural language generation (nlg) community about the challenges and issues that arise when building generative multi-modal models, including biases, ethical concerns, and the naturalness of generated responses. ## datasets both cups and meetup were collected in a taskoriented setting. in cups (figure 1 ), two participants were asked to locate missing cups on a table in a virtually generated scene. it is worth noting that the cups missing for each participant were not necessarily the same ones. these cups varied in colour, type, and location, and each participant could only see a subset of them from a different view. to communicate and identify each other's missing cups, participants used the chat interface. importantly, there were no restrictions on how the task should be approached, allowing participants the freedom to choose their strategies. for additional scenes from the cups corpus, refer to appendix a. in meetup (figure 2 ), participants are randomly placed in a room, which is shown as a real image. this room is part of a larger house area, consisting of connected real images. participants are informed that they can move around the house by executing specific actions, for example, using "/s" to move south from the current room shown in the image. through navigation and interaction in the chat interface, participants are required to ensure that they end up in the same room, where they both see an identical image. it is worth noting that the house layouts often contained multiple rooms of the same type, and participants were frequently asked to meet in a room of this specific type. similar to cups, there were no restrictions placed on the participants regarding how they should solve the task. ## task description the main objective of this challenge is to benchmark and evaluate generative models in the context of multi-modal dialogues. specifically, the task is to generate the utterance u i given the context c, where i represents the utterance number. the context c is formed by the dialogue history and visual scene(s). an important question arises: how effectively can the models utilise the context for generating the next utterance? to address this, we first split each dialogue in the dataset into conversational games @xcite . each game corresponds to a specific sub-task such as asking for more information or describing a scene. in the cups corpus, game annotations for dialogues in swedish are already available, and we plan to prepare annotations for dialogues in english and meetup dialogues as well. these conversational games can be seen as mini-contexts that help achieve a particular goal @xcite . the sentences we select belong to the context of a specific conversational game c l , which, in turn, is part of a broader dialogue context c d . we provide two types of contexts: (i) conversational game context c l : this includes the utterances preceding the target utterance u i within a single conversational game, and (ii) dialogue context c d : this provides information on how the target utterance fits into the overall context of all other utterances and games. it is important to include the full dialogue context, as the flow of dialogues is not always linear; sometimes, a single conversational game can be embedded within another game. we emphasise that a single utterance can often be just a fragment of a broader set of utterances that together convey a specific idea. ## evaluation campaign we will conduct a human evaluation and develop an evaluation protocol for the system submissions. to collect human judgments, we will leverage amazon mechanical turk, and we might also explore using prolific as an alternative platform, which could potentially provide more qualified human crowdworkers foot_3 . the human evaluation procedure will be modelled after the one used in the webnlg+ 2020 challenge @xcite , given that one of the authors of this proposal has experience in running human evaluation. for the human evaluation, crowd-workers will assess the quality of the generated responses based on the dialogue history and visual history (in the case of meetup) or scene views (in the case of cups). these evaluations may be compared against the ground-truth human responses. to rank the responses, a likert scale from 1 to 5 will be employed, where a higher score indicates better quality of the generated response. in human evaluation we will focus on several aspects including: additionally, we will ask human evaluators to perform fuzzy matching of the generated utterances with the ground-truth responses. this approach takes into consideration that the system's output might not be entirely relevant locally but could still be relevant globally. by doing so, the models will not be penalised by automatic metrics for generating responses that differ from the ground-truth significantly, as long as they remain relevant to the conversation itself. to ensure the quality of human evaluation, we will prepare a set of utterances in contexts that clearly represent both low and high points on the likert scale for each of the aspects mentioned earlier. these examples will be shown to crowdworkers before they begin evaluating the actual outputs of the submitted systems. conducting a few test rounds for human evaluation will help us understand the workers' performance and the level of guidance they require to perform well in our task. this process will help us build a pool of highly skilled workers who are trained to evaluate challenge submissions foot_4 . the challenge winners will be selected based on multiple criteria. instead of focusing solely on models that perform well overall, we will also consider models that excel in specific tasks. for instance, we will look for better transfer learning approaches, multi-lingual models, or uni-modal approaches that perform well across various metrics. by examining individual properties of the submitted systems, we aim to document and benchmark the task of utterance generation in visual dialogue from multiple perspectives. ## conclusion we present the vdg challenge as a platform to advance research in grounded situated dialogue. we believe that the task of generating the next utterance in collaborative visual dialogue holds significant value for the nlg (natural language generation) community, especially considering the remarkable performance and attention achieved by large language models in the nlp field. our primary objective is to establish a comprehensive task bench-