BLCU - NLP at S em E val-2020 Task 5: Data Augmentation for Efficient Counterfactual Detecting (2020)

Abstract: Counterfactuals describe events counter to facts and hence naturally involve common sense, knowledge, and reasoning. SemEval 2020 task 5 is focusing on this field. We participate in the subtask 1 and we use BERT as our system. Our Innovations are feature extraction and data augmentation. We extract and summarize features of counterfactual statements, augment counterfactual examples in training set with the help of these features, and two general methods of data augmentation is experimented in our work. We demonstrate the effectiveness of our approaches, which achieves 0.95 of subtask 1 in F1 while using only a subset of giving training set to fine-tune the BERT model, and our official submission achieves F1 0.802, which ranks us 16th in the competition.

## introduction counterfactual statements describe events that did not actually happen or cannot happen, as well as the possible consequence if the events have had happened. detecting counterfactual statements involves common sense, knowledge, and reasoning, it is also the basis for all down-stream counterfactual related causal inference analysis in natural language. @xcite the problem of counterfactual have been studied in many domains, like the perspective of literally logical relations between the antecedent and consequent of counterfactual forms and the outcomes @xcite , and conducting counterfactual thought experiments for hypothetical tests on historical events, policies, or other aspects of a society and assess them by political scientists @xcite . @xcite use a combination of a rule-based approach and a supervised classifier to capture counterfactual statements from twitter, which is close to our problem. semeval 2020 task 5 focus on the problem of detecting counterfactual statements, as @xcite described in their work. subtask 1 of task 5 is a two-classification problem, in which we need to find out whether a statement is counterfactual or not. we choose to apply feature extraction and data augmentation to tackle this problem, for the biggest challenge exists when doing this task is unbalanced data: although the training set has 13,000 examples in total, seems to be enough to fine-tune a bert model, the number of both classes is highly unbalanced: there are only 1454 counterfactual examples, and 11546 non-counterfactual examples seriously dilute the proportion of counterfactual examples in the training set. by observing the training set, we conclude a set of feature words and rules to describe the occurrence of these words in counterfactual examples. in terms of data augmentation, we use two different methods, perform experiments on both augmented datasets to determine which method is more effective. we limit the number of non-counterfactual examples, which keeps the same as the number of counterfactual examples, in actual used training set to balance data. using our best model setting, we get f1 0.95 on our test set, and f1 0.802 on official test set, while we only use 4334 training examples to fine-tune our model. the rest of this paper is organized as follows. section 2 contains a description of our system. experiments and analysis of the results are introduced in section 3. we describe the conclusions in section 4. ## system description our system consists of two components: data augmentation, which includes 2 different augmentation methods, and bert model. since the data provided by the task organizers are highly unbalanced in numbers of examples in two classes, we believe that the key to solve the task is to expand the number of counterfactual examples in some way. we apply 2 data augmentation methods to solve this problem: back translation and a simple eda (easy data augmentation) system @xcite combine with wordnet and feature extraction. ## experiments in this section, we present experiments we conduct on our system and make detailed analysis. we compare the performance among different data augmentation and label balancing strategies. the model is evaluated on the freeze test set we keep untouched from the beginning. ## conclusion we use bert model and data augmentation approach to participate in semeval 2020 task 5 subtask 1. the goal of this task is classification between counterfactual and non-counterfactual statements. we demonstrate that using eda combined with wordnet and feature extraction can output relatively good result. our official submission achieves f1 0.802, which ranks us 16th in the competition. as for future work, we think data extension from external data sources based on feature words set and its pattern might be useful, since it might enlarge feature information of counterfactual statements, which is critical for increase our model's robustness.