COMET - M : Reasoning about Multiple Events in Complex Sentences (2023)

Abstract: Understanding the speaker’s intended meaning often involves drawing commonsense inferences to reason about what is not stated explicitly. In multi-event sentences, it requires understanding the relationships between events based on contextual knowledge. We propose COMET-M (Multi-Event), an event-centric commonsense model capable of generating commonsense inferences for a target event within a complex sentence. COMET-M builds upon COMET (Bosselut et al., 2019), which excels at generating event-centric inferences for simple sentences, but struggles with the complexity of multi-event sentences prevalent in natural text. To overcome this limitation, we curate a Multi-Event Inference (MEI) dataset of 35K human-written inferences. We train COMET-M on the human-written inferences and also create baselines using automatically labeled examples. Experimental results demonstrate the significant performance improvement of COMET-M over COMET in generating multi-event inferences. Moreover, COMET-M successfully produces distinct inferences for each target event, taking the complete context into consideration. COMET-M holds promise for downstream tasks involving natural text such as coreference resolution, dialogue, and story understanding.

## introduction human understanding of narratives involves building mental representations of the described events @xcite . we make commonsense inferences about the unstated but plausible causes, effects, and mental states of the participants. this fundamental ability is crucial for nlp systems for human-level performance on tasks such as question answering, summarization, and story understanding. additionally, reasoning about events can help improve the coherence and consistency of generated text, making it more natural and human-like. in this work, we focus on deriving event-centric commonsense inferences from sentences with multiple events, prevalent in written and spoken language. consider an ambiguous news headline such as "stevie wonder announces he'll be having kidney surgery during london concert" foot_0 . deriving inferences from such sentences is challenging due to the intricate causal and temporal relationships between events. for example, the announcement occurs during the concert, but the surgery happens later. moreover, it is crucial to stay true to the context when making inferences about specific events. existing commonsense knowledge models such as comet @xcite can generate commonsense inferences along dimensions such as causes, effects and the mental states of the event participants. for example, given the sentence "stevie wonder will be having surgery", comet predicts "he is sick" as a reason. however, since comet was trained on simple sentences with a single predicate, it falls short on deriving inferences from multi-event sentences, as we show in fig. 1 . comet's inferences conflate the different events, for example, predicting "he goes to the hospital" as both an event that happened before (the surgery) and after (the announcement). mapping the inferences back to the events that they refer to is not trivial. in addition, the assumption that the two events happened at the same time leads to incorrect inferences such as "he has to cancel the concert". this hurts comet's applicability in tasks involving complex sentences. to address this challenge, we introduce the task of multi-event inference (mei) which involves generating distinct inferences for each target event within a complex sentence. we collected the multi-event-inference (mei) dataset consisting of 35k (sentence, target event, dimension, inference) tuples. we use mei to continue training the latest version of comet @xcite , creating comet-m, a new multi-event commonsense model. comet-m can generate distinct inferences for each target event in the sentence, while taking into account the entire context. as shown at the bottom of fig. 1 , when the announces event is the target, our model predicts that he would later thank his fans for their support. conversely, for the having event, it predicts that he will then recover from the surgery. comet-m improved upon comet in the mei task in terms of both automatic and human evaluation. additionally, we found that baselines trained on synthetic multi-event inferences generated from lms improve over the original comet in some aspects, but may not be sufficient to reach the same level of performance. comet-m has the potential to benefit discourse tasks such as summarization, dialogues and story understanding. further, mei can be used to create tasks of varying difficulty for future research. 2 ## related work reasoning about events is essential for a wide range of nlp tasks such as summarization @xcite , natural language generation @xcite , dialogue systems @xcite , 2 code, models and data are available here. and reading comprehension @xcite . earlier approaches represented event types in terms of their participants and subevents, in a structured representation referred to as a "script" @xcite or "narrative chain" @xcite . for example, a person gets hired at a company, works for it, and then quits, gets fired, or retires. script knowledge for common events is typically mined from text @xcite @xcite . knowledge about common events could then be leveraged in order to represent and make predictions about new events. later efforts leveraged crowdsourcing to collect large-scale knowledge graphs of event-centric commonsense @xcite . the atomic knowledge graph @xcite defines events along nine dimensions pertaining to causes, effects, and the mental states of the event participants. it consists of 880k (event, relation, inference) triplets collected through crowdsourcing. the latest version of atomic was extended to 1.1m triplets and additional relations @xcite . although atomic is largescale, it cannot cover the complete range of eventcentric commonsense knowledge. additionally, aligning a given context with the knowledge graph can be challenging, due to lexical variability and because inferences are context-sensitive. to that end, @xcite developed comet (commonsense transformers). comet is based on pre-trained language models (lms) such as gpt @xcite or bart @xcite , which are further fine-tuned on structured knowledge from atomic. as a result, it can dynamically generate event-centric commonsense inferences along atomic's dimensions for new contexts. this hybrid approach is successful because it combines the structure and accuracy of human-written commonsense facts with the vast commonsense knowledge that lms learn from their massive pre-training corpora @xcite . comet is widely used and has been applied to many downstream tasks such as dialogue and question answering @xcite @xcite @xcite . several subsequent variants of comet have been released in recent years. comet-distill @xcite is a version of comet that is trained on atomic-style events obtained by knowledge distillation from gpt-3 @xcite , while visualcomet @xcite generates inferences about the causes and effects of events depicted in an image. finally, comet-atomic 2020 @xcite is the most recent version of comet, which was trained on more data and a larger underlying lm. the variant of comet most relevant to our work is paracomet @xcite , a paragraph-level comet model. the key differences between comet-m and paracomet are as follows. first, comet-m is trained on various genres for broader applicability, whereas para-comet was trained on the rocstories corpus @xcite in the fiction genre. second, comet-m is trained using human-written inferences, unlike paracomet's silver-standard supervision. our evaluation in sec 6 demonstrates that silver-standard baselines lag behind the goldstandard baseline in the mei task on important aspects. further, comet-m focuses on generating inferences for multiple events within a single sentence, while paracomet focuses on simple sentences in multi-sentence paragraphs. in another recent work, @xcite improved an existing event coreference system by incorporating gpt-3 generated implicit events that likely occurred before or after each event mention. differently from @xcite , comet-m is relation question hasprerequisite what are typically the prerequisites for the event? isbefore what typically happens immediately before the event? isafter what typically happens immediately after the event? xreason what can cause the event? causes what could be the effect of the event? hinderedby what can hinder the event? an openly accessible, generic model that can generate inferences along multiple dimensions and is applicable across multiple domains. ## multi-event-inference dataset in order to create the multi-event-inference (mei) dataset, we obtained annotations of target events within complex sentences, through crowdsourcing (sec 3.2). as the source of the events, we considered datasets from different domains (sec 3.1). ## model the goal of multi-event inference (mei) is to generate commonsense inferences for a target event within a complex sentence. formally, given a context sentence c that includes k event mentions e 1 , e 2 , ..., e k , we want to generate a set of commonsense inferences for each event e j across the 6 inferential relations shown in c i r i [gen] t i where the input is composed of c, the context containing event e j enclosed @xmath0, and r, a special token denoting the relation. @xcite is a special delimiter token that indicates the start of the tail of a given relation for a given head entity. our format is the same as that used in comet except for @xmath1. for consistency with comet training, we maintain the same relation names. the output t is the target inference in free text. during inference, we prompt the model in the same format as the training heads, c i r i @xcite 5 experimental setup ## conclusions we focused on the task of generating commonsense inferences for multiple events in complex sentences. to address this task, we proposed a new model, comet-m, trained comet-m on humanwritten inferences that we collected for multiple events within complex sentences. our experiments, through automatic and human evaluations, demonstrated the effectiveness of comet-m in generating event-specific and relevant inferences. in the future, we intend to investigate the effectiveness of incorporating comet-m into discourse tasks such as dialogue, summarization, and story generation.