Ferryman as S em E val-2020 Task 5: Optimized BERT for Detecting Counterfactuals (2020)

Abstract: The main purpose of this article is to state the effect of using different methods and models for counterfactual determination and detection of causal knowledge. Nowadays, counterfactual reasoning has been widely used in various fields. In the realm of natural language process(NLP), counterfactual reasoning has huge potential to improve the correctness of a sentence. In the shared Task 5 of detecting counterfactual in SemEval 2020, we pre-process the officially given dataset according to case conversion, extract stem and abbreviation replacement. We use last-5 bidirectional encoder representation from bidirectional encoder representation from transformer (BERT)and term frequency–inverse document frequency (TF-IDF) vectorizer for counterfactual detection. Meanwhile, multi-sample dropout and cross validation are used to improve versatility and prevent problems such as poor generosity caused by overfitting. Finally, our team Ferryman ranked the 8th place in the sub-task 1 of this competition.

## introduction counterfactual refers to a conditional statement in which the first clause is a past tense subjunctive statement expressing something contrary to fact, as in "if i had studied harder, i might have passed the exam". sometimes it can be used in practice to support the assumption that if events occurred differently in the past, what might have happened @xcite , as in "if rose had not accepted doctor's advice to cut the tumor, she might have been dead". since rose is still alive, and it is impossible to come back to ask rose to reject the advice. while in other cases, as in " if a drop in crude oil prices had been factored out, the stock price would not be so low", it is not the counterfactual that the drop in the price of crude oil, more than anything else, caused the stock market to plummet. judging counterfactual is hard for the reason that it involves so many aspects of life such as economy, medical treatment. besides, the exploration of causation is necessary. reasoning is a very important and challenge task in the fields of the natural language process and aims to analyze the internal causality of the sentence to find out whether the sentence is correct or not. in the causal relationship, the cause is partly responsible for the result, while the result partly depends on the cause. there is an inherent causal relationship among objective things. through grasping the causal relationship of things, people can understand the nature of things comprehensively. as an important research direction of causal reasoning, counterfactual reasoning has been employed in many cognitive processes. according to the order, counterfactual thinking can be divided into upward counterfactual thinking and downward counterfactual thinking. the former one is the assumption that something has happened in the past, that if certain conditions are met, there is possibility that there will be a better outcome than the real outcome. for example, "if we had been on the pitch before the game, we would not have lost the game". the latter one means that the alternative result is worse than the real one, like "fortunately, i went to the field for adaptation training before the game, otherwise i would have lost the game today". through counterfactual reasoning, people can not only have the joy of success, the regret of failure and other emotional values, but more importantly, they can enhance the confidence of future personal decisions, as well as enable us to find the source of wrong decisions and correct them. in addition, these counterfactual reasoning experience, for others in a similar situation, also have important reference significance. many researchers put in the effort and came up with a series of methods to explore the causation. uplift modelling and causal tree has been applied @xcite @xcite @xcite . in this paper, we propose a model combining bidirectional encoder representation from transformer (bert) with multi sample dropout. after adopting feature frequency-inverse document frequency (tf-idf) vectorizer, the model performs better. different models are compared and we get the best f1 score by combining bert with multi sample dropout, tf-idf vectorizer ensemble and last-5-[cls] token. our model is based on real situations, so when counterfactual occur, they can be compared and identified. multi sample dropout helps accelerate the training process and tf-idf vectorizer with linear regression improves the robustness of the model. rest of the paper are organised as follows. in section 2, we show the overview of the counterfactual. the details of our model are explained in section 3 and results are shown in section 4. we summarize the whole model in section 5. ## related work there are a lot of measures to do the reasoning, which mainly divided into two groups, statics learning and deep learning. the most basic assumption in statistical learning theory is that training data and test data come from the same distribution. however, in most practical cases, the test data is extracted from a distribution that is only relevant to, but not identical to, the distribution of the training data, which is a big challenge to the reasoning. besides, counterfactual distributions tend to differ from fact distributions. @xcite redefines the counterfactual problem as the problem of covariate transformation and later as domain adaptation problem, for the reason that actual distribution and the counterfactual result distribution are not the same. besides, the following points are considered: a) the minimum error rate of factual outcomes; b) the use of relevant factual results to guide counterfactual results is done by constraining the results of similar interventions; c) the distribution of the interventions is similar and is overcome by minimizing the discrepancy distance. here the discrepancy distance refers to the difference between the fact distribution and the counterfactual distribution in the representation space. later @xcite improves the discrepancy distance to be the joint distribution. the new model as a whole is similar to the previous one but overcomes the following limitation: a) the need of a two-step optimization and linear hypotheses of the learned representation and the lack of supporting deep neural networks, b) the loss of the treatment indicator due to high-dimensional learned representation. however, the weight compensating for the difference in treatment group size in the sample is given. @xcite improves the former weight to be learned and combines representation learning with re-weight. firstly, representation learning was used to minimize selection bias and ensure that the factual results were as correct as possible. re-weight could adjust the weight of samples to make the distribution of observed data and counterfactual data as consistent as possible. in contrast to previous approaches to domain adaptation, alaa (2017) considers counterfactual reasoning to be a multi task framework that mitigates selection bias through dropout bias scoring: each iteration has a certain dropout probability that depends on bias scoring. their model uses a deep multi task network with a series layers to model potential (factual and counterfactual) outcomes. ## result our best results for task5 are summarized in ## conclusion to detect counterfactual online effectively, we have applied the last-5-[cls] token and bert to deal with the multiple types of news in task 5. to reduce the computation cost, we have also applied the tf-idf to measure the importance of each word. in the training process, we have used the multi-sample dropout to avoid overfitting and accelerate training speed. over all, our work has show competitive results when comparing with the others. we use the multi sample dropout to improve the robustness compared with dropout. also, the combination of tf-idf vectorizer and lat-5-[cls] token is also a creative way to improve the accuracy rate by extracting more detailed information.