Lee at S em E val-2020 Task 5: ALBERT Model Based on the Maximum Ensemble Strategy and Different Data Sampling Methods for Detecting Counterfactual Statements (2020)

Abstract: This article describes the system submitted to SemEval 2020 Task 5: Modelling Causal Reasoning in Language: Detecting Counterfactuals. In this task, we only participate in the subtask A which is detecting counterfactual statements. In order to solve this sub-task, first of all, because of the problem of data balance, we use the undersampling and oversampling methods to process the data set. Second, we used the ALBERT model and the maximum ensemble method based on the ALBERT model. Our methods achieved a F1 score of 0.85 in subtask A.

## introduction now, with the rapid development of internet technology, our society has entered a highly information era. life is full of different kinds of information. however, the messy information is likely to change or even become the wrong information during the non-stop transmission process. this will have some bad effects. in order to solve this problem, we have created some positions where the verification information is right or wrong. however, this will consume a lot of labor costs and cause a waste of resources. at the same time, there are usually some misjudgments in human detection. therefore, we need some new methods to deal. in the past two years, with the rapid development of artificial intelligence, some related problems have emerged in the field of deep learning. for example, common sense discrimination and natural language inference. the emergence of these problems provides some mechanical methods to solve the problem of error messages. our research is also carried out. semeval 2020 task 5 @xcite is to model causal inference in language: detect counterfactuals. this task is through the establishment of a model, and at the same time to detect whether it is a factual statement based on a large amount of text data. the generation of this task can help us to make a machine judgment of information correctness to a great extent. subtask a is to detect counterfactual statements. this subtask requires us to determine whether a given statement is counterfactual. a counterfactual statement describes an event that has not actually occurred or cannot occur, and the possible consequences of the event. more specifically, counterfactuals describe events that are contrary to facts, and therefore naturally contain common sense, knowledge, and reasoning. solving this problem is the basis of using natural language for all downstream counterfactual causal reasoning analysis. subtask b is to detect precedents and consequences. this subtask states that causal insights are inherent features of counterfactuals. for this task, we mainly participated in subtask a. in this task, we only participate in subtask a: detect counterfactual statements. for this task, we use a combination of research and deep learning methods to deal with related tasks. in the task, we mainly use the lightweight albert model based on the transform mechanism. according to the latest progress of related research, the albert model jointly researched by google research center and chicago toyota research center has become our preferred model. because it has fewer parameters and better performance than the bert model. during data preprocessing, we found that the data set of subtask a has serious data imbalance problems. in order to solve the problem of data imbalance, we used undersampling and oversampling methods for comparative experiments. after data processing, we input the generated word vector into the pre-trained model. in the model, we also adopted the method of maximum voting integration to optimize the performance of our model. in this task, our method is an effective way to get good performance. the rest of our paper is structured as follows. section 2 introduces related work. section 3 describes data preparation. methods are described in section 4. experiments and evaluation are described in section 5. the conclusions are drawn in section 6. ## related work in the field of natural language processing, common sense problems and natural inference problems are relevant to our task. these two problems have large data sets. by describing the data set, we can show the development of these two fields from the content of the data set. meanwhile, the enlightenment of related solutions can also be obtained. in the field of common sense, this question has some data sets. event2mind (rashkin et al., 2018) is a crowdsourced corpus of 25,000 event phrases covering a diverse range of everyday events and situations. situations with adversarial generations (swag) @xcite is a dataset consisting of 113k multiple choice questions about a rich spectrum of grounded situations. the winograd schema challenge (trinh and le, 2018) is a dataset for common sense reasoning. it employs winograd schema questions that require the resolution of anaphora: the system must identify the antecedent of an ambiguous pronoun in a statement. reading comprehension with commonsense reasoning dataset (record) @xcite is a large-scale reading comprehension dataset which requires commonsense reasoning. in the field of natural inference, there are also a large number of data sets. such as, the stanford natural language inference (snli) @xcite corpus contains around 550k hypothesis/premise pairs. the multi-genre natural language inference (multinli) @xcite corpus contains around 433k hypothesis/premise pairs. it is similar to the snli corpus, but covers a range of genres of spoken and written text and supports crossgenre evaluation. the scitail @xcite entailment dataset consists of 27k. in contrast to the snli and multinli, it was not crowd-sourced but created from sentences that already exist in the wild. ## data preparation in this part, we mainly introduce our processing of the data set. at the same time, for the problem of data imbalance, we will also introduce the methods of undersampling and oversampling. through two methods, we will solve the problem of data imbalance. ## methods in this part, we mainly introduce the albert model and the maximum ensemble method based on the albert model. ## conclusion in this task, we mainly use the albert model. based on the model, we also adopted the method of maximum ensemble. at the same time, due to the imbalance of the task data set, we also used the methods of undersampling and oversampling to deal with the problem of data imbalance. in this task, we found that using undersampling and maximum ensemble methods can achieve optimal performance. in the task, we did not think too much about why undersampling brought more improvements in results. in the future work, we will devote ourselves to finding the reason. however, in the rankings, the performance of our model is still not the best. we analyze that the reason for this situation may be that the model parameter fine-tuning has not reached the best and the data imbalance processing method is not good enough. in the future, we will improve our methods towards these reasons. we believe that we can get a good result. meanwhile, in order to solve the problem more comprehensively, we will try to participate in subtask b.