ACCENT : An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems (2023)

Abstract: Commonsense reasoning is omnipresent in human communications and thus is an important feature for open-domain dialogue systems. However, evaluating commonsense in dialogue systems is still an open challenge. We take the first step by focusing on event commonsense that considers events and their relations, and is crucial in both dialogues and general commonsense reasoning. We propose ACCENT , an event commonsense evaluation metric empowered by commonsense knowledge bases (CSKBs). ACCENT first extracts event-relation tuples from a dialogue, and then evaluates the response by scoring the tuples in terms of their compatibility with the CSKB. To evaluate ACCENT, we construct the first public event commonsense evaluation dataset for open-domain dialogues.Our experiments show that ACCENT is an efficient metric for event commonsense evaluation, which achieves higher correlations with human judgments than existing baselines.

## introduction open-domain dialogue systems aim to have natural and engaging conversations with users @xcite . the abundance of dialogue corpus @xcite and the development of neural models @xcite enable open-domain dialogue systems to generate grammatically correct and meaningful responses @xcite @xcite . despite the success, systems still struggle to consistently produce commonsensecompliant responses as humans do. as shown in figure 1 example a, the generated response is not compliant with commonsense since "need an oxygen mask" is not a reasonable prerequisite for "like to paint". commonsense issues for dialogue systems can also be manifested when we consider the dialogue history. for instance, in figure 1 example b, the system's response "that is interesting!" after the user talks about their car accident violates commonly accepted social norms @xcite . in this work, we study automatic dialogue commonsense evaluation by focusing on event commonsense @xcite , which concerns commonsense knowledge about events and their relations. our focus on event commonsense is motivated by the following three observations: first, advanced open-domain dialogue systems have been pre-trained on large corpus and thus suffer less from factoid commonsense issues @xcite . second, events and their relations are key components of commonsense reasoning @xcite , and our study shows overall commonsense and event commonsense are highly correlated (see §4). third, event commonsense aligns well with the interactive nature of open-domain dialogue systems @xcite to complete certain social goals. to automatically evaluate event commonsense in open-domain dialogues, we propose accent, a reference-free automatic event commonsene evaluation metric which leverages commonsense knowledge bases (cskbs) and measures the quality of generated responses without having groundtruth reference responses. for example, comparing the examples in figure 1 against the cskb easily reveals commonsense errors in the responses because when "personx likes to paint", what he/she needs may be "to get a paint brush" instead of "to get an oxygen mask", and when "personx loses arms from a car accident", the other person is expected to feel "sad". while these judgments are intuitive to human, two challenges exist in automating the evaluation process. first, there is a considerable gap between free-form conversational data and the compact commonsense knowledge in the cskb. second, locating relevant knowledge in the cskb is non-trivial. accent addresses these challenges through a pipeline method that uses an intermediate symbolic representation for commonsense reasoning. accent first extracts event-relation tuples from the target response and its preceding dialogue history via a prompt-based generative model trained in a low-resource setting. those extracted tuples bridge the gap between the free-form dialogue and the compact form of cskb. then, a compatibility score is computed to decide how well each extracted tuple aligns with the cskb. to train and evaluate accent, we construct the first publicly available event commonsense evaluation dataset for open-domain dialogues (see §4). besides collecting human commonsense judgments, we request annotators to manually extract event-relation tuples for further analysis. our main contributions are three-fold: ## background: event commonsense endowing machines with human-like commonsense reasoning capabilities has been an ultimate goal of artificial intelligence research for decades @xcite . while many early works focused on factoid commonsense or the knowledge about concepts @xcite , event commonsense emerges as an important aspect for machine commonsense measurement @xcite . compared with concepts or entities, events are more informative, involving actions, participants, time etc. besides, event commonsense also requires understanding various relations between events @xcite which would facilitate complex reasoning, especially in interactive scenarios such as dialogues. among the current commonsense resources (related works in appendix a), atomic 20 20 @xcite is a comprehensive cskb including physical-entity, event-centered, and socialinteraction knowledge. its event-centered and social-interaction components take up 84.4% tuples of the entire knowledge base, providing knowledge regarding how events/human actions are associated with other events/actions. for example, given the event "x runs out of stream", according to atomic 20 20 , this event may happen after "x exercises in the gym", and the person x is likely to "feel tired". ## method we present accent, as a framework for event commonsense evaluation. figure 2 gives an overview of accent with two major components. ## data collection we construct the first event commonsense evaluation dataset for open-domain dialogues through crowdsourcing on amazon mechanical turk (mturk). in this section, we describe the collection procedure and the details of the dataset. ## related work automatic open-domain dialogue evaluation the evaluation of open-domain dialogue systems has long been a challenge due to the system's open-ended goal @xcite , and simply scoring the overall quality is far from enough @xcite . thus, researchers have decomposed the evaluation of open-domain dialogues into multiple facets and developed corresponding automatic evaluation metrics @xcite . while aspects like context coherence @xcite , diversity @xcite , engagement @xcite , have been systematically studied in the literature, the aspect of commonsense has long been neglected. the closest related work is @xcite which is mainly about commonsense-focused dialogues collection but also proposes an automatic metric for commonsense evaluation by training an mlp regressor on both symbolic and neural features. the symbolic features include the numbers of one-hop and two-hop triplets in concept-net @xcite that can be found between the target response and its dialogue history. although this metric utilizes the cskb explicitly, it is limited to the direct search with surface form and only considers the number of triplets, and the cskb used in the work is more about concepts but not event commonsense. ## conclusion we present accent, an automatic evaluation metric for event commonsense evaluation of opendomain dialogue systems. we show that by using event-relation tuples as the symbolic intermediate representations, accent can effectively utilize the cskb and achieve a decent correlation with human judgments for dialogue commonsense evaluation.