On the Underspecification of Situations in Open-domain Conversational Datasets (2023)

Abstract: Advances of open-domain conversational systems have been achieved through the creation of numerous conversation datasets. However, many of the commonly used datasets contain little or no information about the conversational situation, such as relevant objects/people, their properties, and relationships. This absence leads to underspecification of the problem space and typically results in undesired dialogue system behavior. This position paper discusses the current state of the field associated with processing situational information. An analysis of response generation using three datasets shows that explicitly provided situational information can improve the coherence and specificity of generated responses, but further experiments reveal that generation systems can be misled by irrelevant information. Our conclusions from this evaluation provide insights into the problem and directions for future research.

## introduction human communication heavily relies on information about a situation-a set of observable information about relevant objects, their properties, and relationships in the world at a specific time and place. crucial information can often be implicitly assumed during a conversation, particularly when it can be inferred from co-observed facts (e.g., "they are talking at the front desk in a hotel.") or shared a priori (e.g., "the person was staying in the hotel"). incorporating such information is essential for both the development and evaluation of conversational technologies. to illustrate this point, we show a conversation example from dailydialog @xcite , along with responses generated by three systems: blenderbot2 @xcite , @xcite , and blenderbot2: sure. do you want me to load your luggage in the car? it's raining. gpt-3: thank you for staying with us. do you need help with your luggage? chatgpt: okay, thank you. drive safe and stay dry. chatgpt (gpt-3.5) in in this position paper, we discuss the current state of open-domain conversational datasets concerning how situations are represented ( §2). specifically, we consider situational statements 3 that provide partial information about immediately observable (e.g., today's weather), commonly known (e.g., umbrellas are often used on rainy days), or directly derivable facts related to the task, speaker, and goals (e.g., the hotel's check-out and a guest's required action). some of these elements have already been effectively integrated into modern conversational systems, particularly for closeddomain, task-oriented dialogues. we argue that open-domain conversational tasks and datasets should be equipped with some form of situational information. additionally, we conducted case studies on several datasets to explore the potential benefits and challenges associated with situational information ( §3). our analysis indicates that distinguishing between relevant and irrelevant situational information can be challenging for data-driven response engines, offering opportunities for future research. ## status quo in open-domain response generation tasks, systems generate responses in natural language based on input dialog history (a list of utterances from previous turns). dialog history often serves as the primary, and sometimes sole, source of context information in many datasets. in this section, we discuss how conventional task design can be improved through the explicit inclusion of situational information. ## situated response generation in order to analyze the impact of incorporating situational information into response generation, we conducted an empirical analysis using two neural generation models and three english datasets. 5 ## related work conversation history: there is a rich line of work on how to induce useful contextual information from conversation history, for example, by designing dedicated components for capturing contextual information @xcite and using external knowledge @xcite @xcite . while conversation history contains rich information, we need to also incorporate situational information, which is often unspoken, and to this end, we should think about how to design tasks and datasets. prompt design: our analysis is closely related to work on in-context learning, or prompting, with plms. in particular, much attention has been paid to the effective provision of demonstrative examples @xcite @xcite . this paper discussed the problem from a different perspective, namely what clues should be included in prompts (situations) and how plms perform (misleading by distractors). our observation regarding the latter is consistent with prior work that revealed the vulnerability to perturbations in input @xcite . future work should explore ways to robustly identify relevant situational information to generate optimal responses. ## conclusion our main claim is that situational information, which may or may not be stated explicitly by humans, should be represented and incorporated as input in open-domain conversational tasks and datasets in order to advance the capabilities of conversation systems. we posited that the absence of situational information results in an underspecified problem space, causing a severe problem for both the development and evaluation of conversation systems. our experiments on three textual datasets highlight the benefits and difficulties of providing explicit and implicit situational information to response generation systems, which motivates future research on situated conversation systems.