S em E val-2020 Task 5: Counterfactual Recognition (2020)

Abstract: We present a counterfactual recognition (CR) task, the shared Task 5 of SemEval-2020. Counterfactuals describe potential outcomes (consequents) produced by actions or circumstances that did not happen or cannot happen and are counter to the facts (antecedent). Counterfactual thinking is an important characteristic of the human cognitive system; it connects antecedents and consequent with causal relations. Our task provides a benchmark for counterfactual recognition in natural language with two subtasks. Subtask-1 aims to determine whether a given sentence is a counterfactual statement or not. Subtask-2 requires the participating systems to extract the antecedent and consequent in a given counterfactual statement. During the SemEval-2020 official evaluation period, we received 27 submissions to Subtask-1 and 11 to Subtask-2. Our data and baseline code are made publicly available at https://zenodo.org/record/3932442 . The task website and leaderboard can be found at https://competitions.codalab.org/competitions/21691 .

## introduction counterfactual statements describe events that did not happen or cannot happen, and the possible consequences had those events happened, e.g., "if kangaroos had no tails, they would topple over" @xcite . by developing a connection between the antecedent (e.g., "kangaroos had no tails") and consequent (e.g., "they would topple over"), based on the imagination of possible worlds, humans can naturally form some causal judgments; e.g., having tails can prevent kangaroos from toppling over. one can understand counterfactuals using knowledge and explore the relationship between causes and effects. although we may not be able to rollback the events which have happened or make impossible events occur in the real world, we can still think of potential outcomes of alternatives. counterfactual thinking is a remarkable ability of human beings and is considered by many researchers, to act as the highest level of causation in the ladder of causal reasoning. even the most advanced artificial intelligence system may still be far from achieving human-like counterfactual reasoning. counterfactual reasoning is an important component for ai systems in obtaining stronger capability in generalization @xcite . modeling counterfactuals has been studied in many different disciplines. for example, research in psychology has shown that counterfactual thinking can affect human cognition and behaviors @xcite . the landmark paper of @xcite gives a detailed analysis of counterfactual conditionals in philosophy and logistics. as another example, counterfactuals have also been investigated in epidemiology to reveal the relationship between certain diseases and potential risk factors for those diseases @xcite . we present a counterfactual recognition (cr) task, the task of determining whether a given statement conveys counterfactual thinking or not, and further analyzing the causal relations indicated by counterfactual statements. in our counterfactual recognition task, we aim to model counterfactual semantics and reasoning in natural language. specifically, we provide a benchmark for counterfactual recognition with two subtasks. subtask-1 requires systems to determine whether a given statement is counterfactual or not. the counterfactual detection task can serve as a foundation for downstream counterfactual analysis. subtask-2 requires systems to further locate the antecedent and consequent text spans in a given counterfactual statement, as the connection between an antecedent and consequent can reveal core causal inference clues. to build the dataset for counterfactual recognition, we extract over 60,000 candidate counterfactual statements by scanning through news reports in three domains: finance, politics, and healthcare. the first round of annotation focuses on labeling each sample as true or false, where true denotes a sample is counterfactual and false otherwise in subtask-1. a portion of samples labeled as true will be further used in subtask-2 to detect the text spans that describe the antecedent and consequent. specifically, we carefully select 20,000 high-quality samples from the 60,000 statements and use them in subtask-1, with 13,000 (65%) as the training set and the rest for testing. the dataset for subtask-2 contains 5,501 samples, among which we use 3,551 (65%) for training and the rest for testing. to achieve a decent performance in our shared task, we expect the systems should have a certain level of language understanding capacity in both semantics and syntax, together with a certain level of commonsense reasoning ability. in subtask-1, the top-ranked submissions all use pre-trained neural models, which appear to be an effective way to integrate knowledge learned from large corpus. all of these models use neural networks, which further confirms the effectiveness of distributed representation and subsymbolic approaches for this task. some top systems also successfully incorporate rules to further improve the performance, suggesting the benefits of combining neural networks with symbolic approaches. the first-place model also utilizes data augmentation to further improve system performance. in subtask-2, top systems take two main approaches: sequence labelling or question answering. same as systems in subtask-1, all of them benefit from pre-training. we will provide a more detailed analysis in the system and result section. we built a dataset for this shared task from scratch. our data, baseline code, and leaderboard can be found at https://competitions.codalab.org/competitions/21691 . the data and baseline code are also available at https://zenodo.org/record/3932442 . in general, our task here is a relatively basic one in counterfactual analysis in natural language. we hope it will intrigue and facilitate further research on counterfactual analysis and can benefit other related downstream tasks. ## task setup in this section, we detail the two counterfactual recognition subtasks and the metrics used to evaluate the performance. during the evaluation, participants can work on both subtasks or any one of them. ## data development we develop our dataset from news articles in the finance, politics, or healthcare domain. the data development consists of data collection and annotation. we use different approaches to ensure the quality of the data. ## related work modelling counterfactual thinking has started to attract more interest. one of the previous works closest to ours is @xcite , in which a small-scale counterfactual tweet dataset is collected from social media. there are three main differences between that dataset and ours. first, there are only 2,000 samples in the tweet dataset (including the supplement data mentioned in the paper), while our dataset for counterfactual detection in subtask-1 is ten times larger, which we believe is important for training deep learning based models. second, our benchmark provides evaluation for antecedents and consequents extraction, which are essential components of counterfactual analysis. third, our dataset includes statements from three different domains (finance, politics, healthcare). in contrast to the statements collected from tweets, which have a very large portion that are open-ended, vague thoughts, the counterfactuals in our dataset are more meaningful domain-related statements. there is another dataset timetravel proposed in @xcite for counterfactual story rewriting, in which given a short story and an alternative counterfactual event context, the story needs to be minimally revised to keep compatible with the intervening counterfactual event. the empirical results show that it is still challenging for current neural language models to perform well on the counterfactual story rewriting task due to the lack of counterfactual reasoning capabilities. in a broader viewpoint, counterfatuals are an important form of causal reasoning. researchers argue that the notion of counterfactuals is essential for causal reasoning, in which causal modeling is proposed to interpret counterfactual conditionals in natural language, and such work has been discussed since the possible worlds semantics developed in the 1970s @xcite . the more recent work renders useful insights by formulating causal inference as a three-level hierarchy, which are association, intervention, and counterfactual, respectively @xcite . the top of the hierarchy is counterfactual-if a model can correctly answer counterfactual queries like "what would happen if we had acted differently", it should also be able to answer association and intervention queries. the research in @xcite also made contributions to a general theory of causal inference, which is based on the structural causal model (scm), and counterfactual analysis is provided with a formal mathematical formalism. ## summary and future work we present a counterfactual recognition task that includes two basic subtasks. subtask-1 evaluates whether a given statement is counterfactual or not with 20,000 training and test statements. subtask-2 aims at recognizing antecedents and consequents in counterfactual statements. the official task received 27 submissions to subtask-1 and 11 submissions for subtask-2. the state-of-the-art performances achieved a 90% f1 score in subtask 1, as well as an 88.2% f1 and 57.5% exact match score in subtask-2. we hope this task and dataset will intrigue and facilitate further research on counterfactual analysis in natural language. the conjunctive normal form dictates a conditional conjunction is followed by a past tense subjective or past modal verb in the antecedent, and there is a past or present tense modal verb in the consequent @xcite . to filter sentences following this form, we first search for the word if in the sentence. a word with a past tense verb, modal verb, or past participle verb speech tag in the sentence must then follow. lastly, a word with a modal verb tag must come after.