C onvo S ense: Overcoming Monotonous Commonsense Inferences for Conversational AI (2024)

Abstract: Mastering commonsense understanding and reasoning is a pivotal skill essential for conducting engaging conversations. While there have been several attempts to create datasets that facilitate commonsense inferences in dialogue contexts, existing datasets tend to lack in-depth details, restate information already present in the conversation, and often fail to capture the multifaceted nature of commonsense reasoning. In response to these limitations, we compile a new synthetic dataset for commonsense reasoning in dialogue contexts using GPT, ℂonvoSense, that boasts greater contextual novelty, offers a higher volume of inferences per example, and substantially enriches the detail conveyed by the inferences. Our dataset contains over 500,000 inferences across 12,000 dialogues with 10 popular inference types, which empowers the training of generative commonsense models for dialogue that are superior in producing plausible inferences with high novelty when compared to models trained on the previous datasets. To the best of our knowledge, ℂonvoSense is the first of its kind to provide such a multitude of novel inferences at such a large scale.

## introduction effective dialogue is accomplished by a profound grasp of language and a thorough comprehension of the world. such comprehension is crucial to the construction of responses that are pertinent, coherent, and captivating within an ongoing dialogue. a pivotal element of this worldview is commonsense: self-evident information that is universally acknowledged among humans @xcite . over time, there has been a concerted endeavor to create datasets that facilitate commonsense reasoning. early work, such as the widely recognized conceptnet @xcite , focused predominantly on physical commonsense related to entities. lately, efforts have shifted toward building datasets encompassing social-and event-based commonsense, such as atomic @xcite . this new wave of datasets targets complex human concepts, including emotions, desires, and motivations. as human conversations largely revolve around sharing personal experiences and life events @xcite , it is critical for virtual agents to possess a robust understanding of human experiences to conduct effective dialogue. datasets such as atomic hold promise as they provide insights directly relevant to human experience; however, a drawback lies in their lack of contextual awareness as they hinge on isolated, concise phrases for commonsense inferences. this limitation poses challenges for dialogue-oriented tasks because utterances should not be viewed in isolation but must be interpreted within their context @xcite . several initiatives have recently aimed to curate commonsense inferences tailored for dialogue contexts @xcite @xcite . however, a trade-off currently exists between the breadth of inference types covered and the scope of dialogue contexts encompassed within these existing datasets. while some datasets cover a wide range of relations, they are limited to a small number of dialogues @xcite , whereas others capture a large number of dialogues but on a limited set of relations @xcite . in addition, a few challenges can be encountered in these datasets. for example, the inferences in these datasets are often too succinct and derive only straightforward conclusions with minimal elaboration @xcite , which do not convey implicit commonsense. some studies instruct annotators to recycle information from the ongoing conversation, undermining the speculative nature of inferences and detracting from the potential of offering fresh insights to enhance dialogue understanding @xcite . moreover, although multiple plausible inferences can be drawn from a single dialogue context, only a few datasets support this multifaceted nature @xcite , impeding the development of models capable of generating diverse inferences, and thus limiting their utility in real applications. we present convosense, a commonsense dataset generated by gpt encompassing 10 popular inference types with over 500,000 inferences across 12,000 dialogues ( §4). our dataset shows greater contextual novelty and enhanced inference diversity and detail while maintaining exceptional reasonability compared to existing datasets ( §3). we also explore several strategies to build generative models producing inferences for dialogue contexts ( §5). our experiments show that models trained on convosense excel in generating plausible inferences with greater detail and novelty, compared to ones trained on existing datasets ( §6). to the best of our knowledge, this is the first dialogue-based commonsense dataset that not only covers an extensive array of inference types at large-scale but also provides a plethora of diverse, novel inferences tailored to each dialogue context. our convosense dataset and inference models can be accessed through our open-source project: https://github.com/emorynlp /convosense . ## related work recent work has focused on integrating commonsense into various tasks, including story generation and explanation @xcite , dialogue summarization and explanation @xcite @xcite , and response generation @xcite @xcite . many of this work relies on existing datasets, such as conceptnet @xcite and atomic @xcite , which only contain single-word or short-phrase premises and conclusions. although there are commonsense datasets curated for long dialogue contexts, they tend to be of small size @xcite , express simple inferences @xcite , or copy context from the provided utterances @xcite . on the other hand, gpt has recently been used to create a variety of datasets. @xcite and @xcite constructed dyadic dialogue datasets at large-scale, while @xcite generated commonsense triples in the atomic style @xcite . however, the atomicstyle inferences are not necessarily suitable for dialogue, as they struggle to handle long contexts and often lack depth. cicero human participants were tasked with composing responses to five commonsense questions (e.g., what is the event that directly causes or could cause target?) based on dialogue contexts and explicitly instructed to incorporate information from the preceding or forthcoming utterances. the first version produced a single inference for each example @xcite , whereas the second version produced multiple examples of both good and bad inferences @xcite . reflect @xcite supplied both human-generated commonsense inferences and following utterance responses that could be derived from a specified commonsense inference. the inferences were collected by instructing human participants to answer a commonsense question, while the next-utterance responses were composed by new human participants who were provided with the dialogue context and one of the human-generated inferences. ## evaluating gpt-generated inferences in order to support the development of a largescale and high coverage commonsense dataset for dialogue that improves upon existing works, we hypothesize that we can leverage large language models (llms) to accomplish this task in an efficient and low-cost manner. from initial pilot tests type label(s) definition(s) com cic ref subsequent isbefore what could happen after this? [2] * 22k 600 subsequent-what subsequent event happens or could happen following the target? [3] events what might happen after? [4] antecedent isafter what could have happened before this? [2] * 600 what might have happened before? [4] cause xreason what could be the cause of this event? [2] 80 21k cause what is the event that directly causes or could cause target? [3] prerequisite xneed what does x need to do before the event can happen? [1] 1k 10k prerequisites what is or could be the prerequisite of target? [3] motivation xintent why does x cause the event? [1] 800 12k motivation what is an emotion or basic human drive that motivates or could motivate target? [3] attribute xattr how would x be described? [1] 400 600 how would you describe speaker? [4] reaction xreact how does x feel after the event? [1] 300 600 what is speaker feeling now? [4] reaction o oreact how do others feel after the event? [1] 70 6k 600 what is the possible emotional reaction of the listener in response to target? [3] what is responder feeling now? [4] desire xwant what would x likely want to do after the event? [1] 1k desire o owant what would others likely want to do after the event? [1] 100 constituents hassubevent what is a substep that happens within this event? [2] 800 obstacle hinderedby what could obstruct the occurrence of this event? [2] 200 effect causes what does this event cause to happen? [2] 30 effect s xeffect what effect does the event have on x? [1] 400 effect o oeffect what effects does the event have on others? [1] 90 ## convosense dataset given our assessment of high-quality, novel, and detailed gpt-generated commonsense inferences across various dialogue contexts and inference types (section 3), we construct a substantial conversational commonsense dataset using gpt, termed convosense. ## generative model evaluation we evaluate the six generative models (section 5.2) on the ten commonsense inference types ( ## limitations and ethical considerations this work does not intend to present an exhaustive set of commonsense inferences for dialogue. while we adhere to established inference types relevant to dialogue from existing literature, there could be overlooked types or unique challenges within specific dialogue domains that remain to be explored. furthermore, it is important to recognize that some social commonsense inference types may be associated with stereotypes and biases. when employing a model that produces commonsense inferences in a setting that impacts human users, caution must be exercised to prevent unjust or prejudiced decisions. although exploration of the prevalence of harmful biases is out of the scope of the current work, we welcome future investigations into quantifying these aspects of our resources. finally, we adhered to openai's terms of service and related policies when utilizing gpt, and we acknowledge that any subsequent utilization of our models and data should refer to these policies. ## future work although convosense is composed of diverse multi-inference dialogue data ( in addition, the integration of commonsense understanding into dialogue applications has shown promising results in improving performance on tasks such as response generation, summarization, and reading comprehension in previous works. in light of this, our work on improving commonsense resources and models presents an opportunity for further advancements in these dialogue applications. in particular, future work exploring how to capitalize on our commonsense model for dialogue response generation is highly compelling, since commonsense errors are one of the most common issues for modern dialogue agents @xcite . however, previous works have revealed that naive integration of commonsense inferences into neural models do not necessarily produce improvements @xcite . as a result, we leave the integration of our commonsense model to future work to allow for thorough investigation of its impact on response generation, covering aspects such as the impact of different commonsense inference types, the filtering of relevant inference types per dialogue context, and the effect of synthesizing multiple inferences into dialogue responses. ## conclusion in this work, we present convosense, an automatically constructed dataset of multi-output commonsense inferences for dialogue. convosense surpasses existing datasets in size, advances inference detail and novelty, and attains comparable (if not superior) reasonability when compared to existing datasets. our investigation into various techniques for generating multiple inferences reveals that diverse beam search on single-output generative models yields the best outcomes. by publicly releasing our trained models, we enable other works to benefit from the remarkable improvements in commonsense reasonability and novelty achieved by this work.