D isco S ense: Commonsense Reasoning with Discourse Connectives (2022)

Abstract: We present DiscoSense, a benchmark for commonsense reasoning via understanding a wide variety of discourse connectives. We generate compelling distractors in DiscoSense using Conditional Adversarial Filtering, an extension of Adversarial Filtering that employs conditional generation. We show that state-of-the-art pre-trained language models struggle to perform well on DiscoSense, which makes this dataset ideal for evaluating next-generation commonsense reasoning systems.

## introduction much of the recent work in commonsense reasoning has focused on evaluating a pre-trained language model's (lm) ability to predict the most plausible ending/option given a context. even after devising bias reduction techniques @xcite to mitigate the effects of annotation artifacts and make the task difficult, state-of-the-art lms have managed to achieve or even surpass human performance on numerous commonsense downstream tasks @xcite @xcite . nevertheless, these lms are still very far from being able to perform commonsense reasoning as well as humans. hence, the fact that they have begun to ace existing benchmarks implies that time is ripe to design a new challenging benchmark that can reliably target their limitations. motivated by this observation, we present dis-cosense, a benchmark for performing commonsense reasoning through understanding a wide variety of discourse connectives. figure 1 shows an example taken from discosense. as can be seen, an example is composed of a context (e.g., "our waitress was very nice, but she kept on forgetting my stuff.") and a discourse connective (e.g., "for example"), and the goal is to choose the most plausible ending out of four options. if we ignore the discourse connective, then all four options may our waitress was very nice, but she kept on forgetting my stuff. seem plausible because we do not know what the writer's intent is. once we consider both the context and the discourse connective, then it is clear that only option b) is plausible. the reason is that "for example" signals an exemplification relation between its arguments, and what follows the discourse connective is expected to be an example of the waitress keeping on forgetting the writer's stuff. using commonsense knowledge, we know that (1) "my beer and fries" is an example of "my stuff", and (2) her taking forever to bring the writer stuff implies she kept on forgetting his/her stuff. what if we replace "for example" with "however" in the example? since "however" signals a contrast relation, options a) and d) both seem viable. specifically, option a) describes a situation in which she did not forget the writer's stuff. while option d), unlike option a), does not describe any example that signals a contrast, one may infer a contrast between option d) and the context: being forgetful is fine for some customers. nevertheless, option a) is arguably more plausible than option d) and should be chosen. the reason is that for d) to be sensible, one needs to assume that her forgetting the writer's stuff implies that she is in general forgetful. without this assumption, it may be strange for other customers to have an opinion on her forgetting the writer's stuff. in general, the most plausible option is the option that makes the smallest number of assumptions, and/or is the most coherent given the context and the discourse connective. considering the commonsense knowledge and the reasoning involved, it should not be difficult to see that this task is challenging. our contributions are four-fold. first, we create discosense, a new dataset aimed at testing lms' commonsense reasoning capabilities through discourse connectives. second, we employ a controlled text generation based adversarial filtering approach to generate compelling negatives. third, we establish baseline results on discosense with numerous state-of-the-art discriminator models and show that they struggle to perform well on dis-cosense, which makes our dataset an ideal benchmark for next-generation commonsense reasoning systems. finally, we show the efficacy of using discosense as a transfer learning resource through sequential fine-tuning of lms on dis-cosense followed by hellaswag and achieve near state-of-the-art results on the hellaswag test set. to stimulate work on this task, we make our code and data publicly available. 1 2 related work in this section, we discuss related work, focusing our discussion on the differences between dis-cosense and existing commonsense reasoning benchmarks. in addition, we present an overview of adversarial filtering, which will facilitate the introduction of the conditional adversarial filtering mechanism we propose in section 3. commonsense reasoning benchmarks. swag @xcite and hellaswag @xcite are arguably the most prominent commonsense reasoning benchmarks. in swag, given a partial description along with four candidate endings, the task is to predict the most plausible ending. the synthetic options (a.k.a. distractors) are generated through a process called adversarial filtering (af) (see below). hellaswag is an extension of swag that seeks to eliminate artifacts in the generated endings. unlike swag and hellaswag, discosense requires that the discourse connective be taken into account in the reasoning process, thus increasing the number of inference steps and potentially the task complexity. in addition, while the examples in swag and hellaswag come primarily from activitynet (a benchmark focused on dense captioning of temporal events), 1 for our code and data, see https://github.com/ prajjwal1/discosense/ . ## conclusion motivated in part by the fact that existing pretrained language models have surpassed human performance on numerous commonsense reasoning datasets, we introduced discosense, a challenging benchmark that concerns commonsense reasoning with discourse connectives to determine the most plausible ending of a sentence. this task was made difficult by the synthesis of high quality complex examples, which was made possible through coupling highly competitive conditionally trained models for language generation with adversarial filtering. the best performing model on dis-cosense only achieved an accuracy of 65%, significantly lagging behind humans. this makes dis-cosense an ideal benchmark for next-generation commonsense reasoning systems.