Commonsense-Focused Dialogues for Response Generation: An Empirical Study (2021)

Abstract: Smooth and effective communication requires the ability to perform latent or explicit commonsense inference. Prior commonsense reasoning benchmarks (such as SocialIQA and CommonsenseQA) mainly focus on the discriminative task of choosing the right answer from a set of candidates, and do not involve interactive language generation as in dialogue. Moreover, existing dialogue datasets do not explicitly focus on exhibiting commonsense as a facet. In this paper, we present an empirical study of commonsense in dialogue response generation. We first auto-extract commonsensical dialogues from existing dialogue datasets by leveraging ConceptNet, a commonsense knowledge graph. Furthermore, building on social contexts/situations in SocialIQA, we collect a new dialogue dataset with 25K dialogues aimed at exhibiting social commonsense in an interactive setting. We evaluate response generation models trained using these datasets and find that models trained on both extracted and our collected data produce responses that consistently exhibit more commonsense than baselines. Finally we propose an approach for automatic evaluation of commonsense that relies on features derived from ConceptNet and pre-trained language and dialog models, and show reasonable correlation with human evaluation of responses’ commonsense quality.

## introduction open-domain dialogue response generation (rg) models aim to provide human-like natural language responses given dialogue histories @xcite . to improve generated response quality, many studies have been conducted to develop knowledge-grounded rg @xcite , personalized dialogue agents @xcite , empathetic response @xcite , etc. for all the above-mentioned directions for rg, large-scale dialogue data geared towards the specific goals is crucial, since most current state-of-the-art neural rg models require training on appropriate and large data. therefore several datasets have been collected to support such research efforts such as knowledge-grounded dialogues @xcite , per-sonachat @xcite , and empathetic-dialogues @xcite . producing natural and logically-coherent responses given dialogue contexts involves making commonsense inferences during the communication. for example, if someone says "i'm going to perform in front of a thousand people tomorrow..." the listener is likely to conclude that the speaker is probably feeling nervous and respond by comforting them: "relax, you'll do great!" in contrast to other efforts to make rg models more empathetic or knowledgeable, there is a lack of commonsense focused dialogue data for both training neural models and evaluation. an ideal dataset for studying commonsense in rg needs to simulate how humans have multiturn conversations as much as possible. existing commonsense-focused work in rg uses extracted post-response pairs from reddit @xcite , which are single-turn and rough approximations for real-life conversations. aiming to bridge the gap in commonsense for dialogue response generation, we collect a large-scale multi-turn open-domain dialogue dataset that is focused on commonsense knowledge. we first consider extracting commonsense-focused dialogues from three existing dialogue datasets by identifying responses that contain commonsense inferences using conceptnet @xcite . this filtering results in 21k dialogues. then we collect 25k new dialogues focusing on social commonsense inferences, where prompts are context sentences describing an event in the socialiqa data @xcite . to study commonsense in rg, we train large generative language models on our datasets and compare with models trained on existing datasets. we find through sampled human evaluation that our dataset helps to generate more commonsensical responses (average score of 6.9 out of 10 compared to 4.8 using other data), and automatically generated responses still have a large gap in comparison to human performances (9.2 out of 10). to help lower the evaluation cost and increase the efficiency of evaluating commonsense in rg, we further propose an automatic metric using combined neural and symbolic features derived from conceptnet, and show that this metric has reasonable correlation with human annotations and symbolic features contribute positively to system performance. our contributions are as follows: (1) we create the first large-scale open-domain dialogue dataset focusing on social commonsense inferences. this includes a new collection of 25k dialogues based on socialiqa event prompts, and conceptnet filtered data from some existing data sets. (2) we benchmark our dataset and show that models trained on our dataset helps make models produce more commonsensical responses. (3) we propose the first automatic metric for evaluating the commonsense plausibility in response generation that reaches statistically significant correlation with human annotations. ## commonsense focused dialogue collection to collect more commonsense focused dialogues for response generation model training and evaluation, our effort is along two directions: filtering existing data to collect dialogues with responses that consist of commonsense (section 3.1), and curating new data using prompts from a commonsense reasoning multiple-choice benchmark socialiqa (section 3.2). ## experiment setup and evaluation methods the focus of this study is to examine how commonsense plays a role in dialogue response generation. in previous sections, we propose a simple filtering method to obtain commonsense-focused dialogues from existing three datasets and crowdsource more dialogues based on the socialiqa commonsense reasoning benchmark. here we aim to evaluate response generation models' ability to produce responses that follow commonsense and if training on commonsense-focused dialogue data helps boost model performance. in addition to using automatic referenced metrics and human evaluation, we also propose a new automatic unreferenced metric aiming to evaluate responses for commonsense quality. ## conclusion we present our empirical study on commonsense in dialogue response generation. to obtain data for commonsense-focused analysis in open domain response generation, we use two strategies: filtering existing dialogue data using a commonsense knowledge graph concepetnet, and collecting new dialogues using prompts from multiple-choice commonsense benchmark. our data has a few limitations such as our filtering process focuses on wordlevel matching to conceptnet, which might omit more complex commonsense relations embedded in dialogues. we leave deeper analysis of how implicit commonsense is represented in dialogues and how to elicit multi-hop granular reasoning steps during communications to future work. our experimental results show that our newly collected data helps boost response generation model performance based on human evaluation of commonsense. to close the gap in automatic evaluation metric for response generation, we propose a simple mlp regressor using both neural and symbolic features, and show promising correlation with human judgements. we hope our work helps with future attempts to make models produce responses with more commonsense, which is a challenging but crucial task to tackle in dialog systems. history a: my pet dog is always there for me. no matter what, i can count on her. a: james , why are you watching tv ? your train leaves at 4:30. there are just 2 hours left . have you finished packing ? a: i am up for a promotion at my job. i positive i will get it. b: that is great! you should celebrate and enjoy this moment human and model responses with annotation scores (1-10) human response b: i feel the same way about my dog. they're the greatest! [10] b: i 've almost finished packing . now i 'm waiting for the jacket i lent to david last week . [9] a: i already have plans to go out to a nice restaurant and celebrate with my family [10] dd b: i hope so. [3] b: yes, i've already got the files ready. [1] a: you are so wonderful! [3] ed b: that is so sweet. how long has she been with you? [10] b: i hope you get through it [1] a: yeah i know, it is such a blessing to have a mom. [1] tc b: i have a dog, and it is a good thing that i don't have a cat, because i am afraid of them. my dog is a border collie. [2] b: i am watching tv, just to catch up on some shows, how about you? [2] a: i love when my kids get something too! my kids have been very successful and i love how they have friends. [2] filtered existing (fe) b: that's great. you must miss her. [10] b: i'm not sure. i'm going to the movies. i'm not sure if i 'll have time to finish packing. [3] a: i am so happy for my promotion. [4] fe+crowdsourced b: that is nice. i have two dogs. [7] b: oh, yes. i have. i 'll be home at 7:30. [6] a: i hope so. i am so excited about it. [7] fe+filtered crowdsourced b: that's nice. what breed is it? [10] b: oh, i finished packing. i 'll go to the station and get my things. [9] a: thanks! i am going to get a new car! [8]