BY team at S em E val-2020 Task 5: Detecting Counterfactual Statements with BERT and Ensembles (2020)

Abstract: We participate in the classification tasks of SemEval-2020 Task: Subtask1: Detecting counterfactual statements of semeval-2020 task5(Detecting Counterfactuals). This paper examines different approaches and models towards detecting counterfactual statements classification. We choose the Bert model. However, the output of Bert is not a good summary of semantic information, so in order to obtain more abundant semantic information features, we modify the upper layer structure of Bert. Finally, our system achieves an accuracy of 88.90% and F1 score of 86.30% by hard voting, which ranks 6th on the final leader board of the in subtask 1 competition.

## introduction counterfactual statements describe events that did not happen or cannot happen, as well as the possible consequence if the events have had happened. more specifically, counterfactuals describe events counter to facts and hence naturally involve common sense, knowledge, and reasoning. tackling this problem is the basis for all down-stream counterfactual related causal inference analysis in natural language. in this work, we present our results for the semeval 2020 shared task 5 @xcite : modelling causal reasoning in language: detecting counterfactuals. subtask 1 focuses on the binary distinction if a sentence is counterfactual or not. our submission for subtask a ranks 6th. for subtask 1, we experiment with different neural network-based models such as recurrent neural network(rnn) @xcite , attention @xcite and bidirectional encoder representations from transformers (bert) @xcite . for the imbalanced data set, we find that the bert performs best on our development set. so, we combine the results by different bert models. the rest of the paper is organized as follows. section 2 briefly shows the related work. section 3 elaborates on our approach. it shows the pre-processing step and architecture of our model. section 4 describes the data set, hyper-parameters adopted research methodology and our results. finally, section 5 concludes our work. ## related work detecting the counterfactual statement is becoming more and more important. to build an effective classifier, one of the major problems is to find the appropriate features. normally, two types of features are applied: surface features like n-grams and word representations trained by the neural network. traditionally, the classifiers are trained on different types of surface features with approaches like svm, random forest, and logistic regression. using pre-trained word embeddings for feature extraction is effective in multiple nlp tasks. @xcite traditionally word embeddings are extracted from shallow neural networks trained on a large swathe of texts required to learn the contextual representations of words. for example, there are skip-grams @xcite and glove @xcite . however, these embeddings are learned from an aggregation of all possible word contexts, which may gloss over semantic nuances in representations. recently, models like elmo @xcite and bert significantly advance the state-of-the-art in language modeling by learning context-sensitive representations of words. elmo goes beyond word embeddings by learning representations that are functions of the entire input sentence. however, elmo is still considered shallow with two bidirectional lstm layers, and more recent transformer-based language models such as the openai generative pre-trained transformer (gpt) @xcite and bidirectional encoder representations from transformers (bert) have been extended to a depth of up to twelve layers. the bert is trained to be bidirectional with two novel prediction tasks, masked lm and next sentence prediction. the pre-trained bert model has been shown to achieve significant improvements in a series of downstream tasks over elmo and openai gpt. ## conclusion this paper introduces the overall idea and specific scheme of byteam in semeval-2020-task5. in the competition, we adopted multiple methods to modify the upper structure of bert model, perform stratified 5-fold cross-validation based on each model, and finally achieve good performance by hard voting. in the future, our work will focus on solving the problem of imbalanced data and how to make the model obtain abundant semantic information features. the code is available online. 5