S em E val-2020 Task 4: Commonsense Validation and Explanation (2020)

Abstract: In this paper, we present SemEval-2020 Task 4, Commonsense Validation and Explanation (ComVE), which includes three subtasks, aiming to evaluate whether a system can distinguish a natural language statement that makes sense to humans from one that does not, and provide the reasons. Specifically, in our first subtask, the participating systems are required to choose from two natural language statements of similar wording the one that makes sense and the one does not. The second subtask additionally asks a system to select the key reason from three options why a given statement does not make sense. In the third subtask, a participating system needs to generate the reason automatically. 39 teams submitted their valid systems to at least one subtask. For Subtask A and Subtask B, top-performing teams have achieved results closed to human performance. However, for Subtask C, there is still a considerable gap between system and human performance. The dataset used in our task can be found at https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation .

## introduction in the past decades, computer' ability in processing natural language has significantly improved. however, its intelligence for understanding common sense expressed in language is still limited. for example, it is straightforward for humans to judge that the following sentence is plausible, or makes sense: "john put a turkey into a fridge" while "john put an elephant into the fridge" does not, but it is non-trivial for a computer to tell the difference. arguably, commonsense reasoning plays a central role in a natural language understanding system @xcite . it is essential to gauge how well computers can understand whether a given statement makes sense. in our task, we take an operational definition of making sense by asking human subjects to generate natural language statements that obey or violate their commonsense knowledge about the world. 1 many existing tasks embed the evaluation of commonsense understanding in other tasks such as coreference resolution @xcite , subsequent event prediction @xcite , ordinal common-sense inference @xcite , situations with adversarial generations @xcite , event validation @xcite , reading comprehension @xcite @xcite , dialogue @xcite and qa @xcite @xcite . they verify whether a system is equipped with common sense by testing whether the system can give a correct answer when the input does not contain such knowledge. the above tasks do not directly evaluate commonsense validation and they do not explicitly identify the key factor required in a commonsense validation process. the semeval-2020 task 4 includes three subtasks on testing whether a system can distinguish natural language statements that make sense from those that do not, and probe the reasons. in the first subtask, a system needs to choose the against-common-sense statement from two natural language statements of similar wordings, e.g., "john put an elephant into the fridge" and "john put a turkey into the fridge", respectively. the second task aims to find the key reason from three provided options why a given nonsensical statement does not make sense. for example, for the nonsensical statement, "john put an elephant into the fridge", the three options are "an elephant is much bigger than a fridge", "elephants are usually white while fridges are usually white", and "an elephant cannot eat a fridge." a system needs to identify the correct reason. in addition, the third task requires the participating systems to generate the reason automatically. we hope that the task and datasets can facilitate studies on commonsense validation, its interpretability, and the related natural language understanding and generation problems. there are 39 teams submitting valid systems to at least one subtask. in subtask a and subtask b, top-performing systems achieve performances closed to that of human subjects. however, for subtask c, there is still a relatively large between system and human performances. ## data construction our data construction is mainly performed on amazon mechanical turk, which consists of two steps: finally, each instance of the dataset have 8 sentences: ## systems and results in this section, we show the evaluation results of all the submitted systems for the three subtasks. since most systems share similar model architecture for subtasks a and b, we discuss the two subtasks together. ## related work commonsense reasoning in natural language has been studied in different forms of tasks and has recently attracted extensive attention. in the winograd schema challenge (wsc) @xcite , a model needs to solve hard co-reference resolution problems based on commonsense knowledge. for example, "the trophy would not fit in the brown suitcase because it was too big. what was too big (trophy or suitcase)?" the choice of plausible alternatives (copa) @xcite emphasizes on events and consequences. each question in copa aims to find the suitable cause or result of the premise from two given alternatives. all premises and alternatives are simple sentences. for example, the premise can be "the man broke his toe. what was the cause of this?" and the two candidate answers are "(1) he got a hole in his sock." and "(2) he dropped a hammer on his foot." several subsequent datasets are inspired by copa. the jhu ordinal common-sense inference (joci) @xcite aims to label the plausibility from 5 (very likely) to 1 (impossible) of human response after a particular situation. situations with adversarial generations (swag) @xcite request a system to choose the most likely-to-happen alternative after a specific situation. those datasets emphasize the pre-situations and/or the after-situations of certain situations, but not on the reasons why they occur or are caused. besides, our dataset is not limited to events or situations. it concerns a broader commonsense setting, which includes events, descriptions, assertion etc. some datasets are inspired by reading comprehension. the story cloze test and rocstories corpora @xcite aim to figure out the right ending from two candidate sentences after a four-sentence story. for a narrative text, mcscript @xcite gives various types of questions and pairs of answer candidates for each question. most questions require knowledge beyond the facts mentioned in the text. compared to those reading comprehension tasks, our benchmark encourages people to use any external resources they want. some other datasets evolve from qa problems and care more about factual commonsense knowledge. squabu @xcite provides a small hand-constructed test of commonsense and scientific questions. commonsenseqa @xcite asks crowd workers to create questions from conceptnet @xcite , which is a large graph of commonsense knowledge, where each question discriminates its answer candidates between three target concepts that all share the same relationship to a single source drawn from conceptnet. openbookqa @xcite provides questions and answer candidates, as well as thousands of diverse facts about elementary level science that are related to the questions. the ai2 reasoning challenge (arc) @xcite gives thousands of questions with different knowledge types, as well as a relevant 14m-sentence corpus, mixed with science facts and other narrative sentences. mutual provides a dataset for multi-turn dialogue reasoning in the commonsense area @xcite . those questions are not easy to answer without specializing certain domain knowledge, while our questions are based on daily common sense. some datasets focus on non-sentential eventual plausibility @xcite , such as "gorilla-ride-camel". in contrast, our dataset is based on statements which includes events, descriptions, assertion etc, not merely events, such as "china's territory is larger than japan's". and some datasets concentrate on limited attributes or actions of world knowledge, such as physics @xcite . our dataset concerns general commonsense knowledge beyond just physical common sense, the sentence in our task "tom's mom become (happy)/(upset) when tom gets high grades in the exam" is about social and emotional common sense. for our first task, those statements that conforms to commonsense can also be phrased as being plausible. thus our first task is similar to plausibility tests, despite that plausibility has a broader scope while our focus is on commonsense only. more importantly, compared with our work, the above tasks do not directly estimate general common sense or ask the logical reasons behind the correct answers and questions. in recent years, some large-scale commonsense inference knowledge resources have been developed, which may be helpful in commonsense reasoning tasks. atomic @xcite presents a large-scale everyday commonsense knowledge graph, which has nine if-then relations with variables, including causes, effects, and so on. event2mind @xcite proposes a new corpus and task, aiming to find out the mentioned/unmentioned people's intents and reactions under various daily circumstances. these datasets are not directly useful for our benchmark since they focus only on a small domain. conceptnet is a seminal knowledge graph that has been upgraded over time @xcite @xcite . conceptnet constructs triples using labeled edges as relations and various words and/or phrases as entities. it also has the sentences describing the corresponding triples. in contrast to these datasets, we investigate the evaluation of common sense, rather than building a resource. before organizing this shared-task, a pilot study @xcite has been performed, showing that there is still a significant gap between human and machine performance when no training data is provided, despite that the models have already been pretrained with over 100 million natural language sentences. in our task here, we also provide training data with human annotations. ## summary this paper summarizes semeval-2020 task 4: commonsense validation and explanation. in this task, we construct a dataset that consists of 11,997 instances and 83,986 sentences. the task attracted around 40 participating teams, out of which 31 teams submit their system papers. the pretrained models are shown to be very effective in subtask a and subtask b, but there is still a large room to improve system performances in subtask c. contextualized embedding such as roberta and bart play a central role in the success of the top-performing models, demonstrating that such methods contain commonsense information to a good extent. we attribute the high performance on subtask a and b to several main reasons: 1) subtask a is a relatively easy question by definition: a model needs only to detect a relatively less plausible content among the two candidate sentences. 2) pretrained models are obtained on billion-words large corpora such as wikipedia data, which help obtain commonsense knowledge @xcite , which helps achieve considerably better performance. 3) as described in the annotation process, we use the sentences from omcs to inspire crowd-sourcing workers. the top-3 systems also use omcs, which potentially help them to attain better performances. 4) for subtask b, as discussed in our data analysis section, the data has some flaws in the average length and common words, which reduces the difficulty. 5) some instances have obvious patterns. for example, there are tens of instances that contain "put xxx into yyy", and "xxx is bigger than yyy", making the problems simpler. 6) hundreds of crowd-sourcing