YNU -oxz at S em E val-2020 Task 5: Detecting Counterfactuals Based on Ordered Neurons LSTM and Hierarchical Attention Network (2020)

Abstract: This paper describes the system and results of our team’s participation in SemEval-2020 Task5: Modelling Causal Reasoning in Language: Detecting Counterfactuals, which aims to simulate counterfactual semantics and reasoning in natural language. This task contains two subtasks: Subtask1–Detecting counterfactual statements and Subtask2–Detecting antecedent and consequence. We only participated in Subtask1, aiming to determine whether a given sentence is counterfactual. In order to solve this task, we proposed a system based on Ordered Neurons LSTM (ON-LSTM) with Hierarchical Attention Network (HAN) and used Pooling operation for dimensionality reduction. Finally, we used the K-fold approach as the ensemble method. Our model achieved an F1 score of 0.7040 in Subtask1 (Ranked 16/27).

## introduction in recent years, in order to simulate counterfactual semantics and reasoning in natural languages, the introduction of counterfactuals in natural language understanding systems has received more and more research attention. according to the problem of counterfactual conditionals @xcite , counterfactual statements can be transformed into opposing statements with true premises and results. counterfactual statements describe events that did not occur or cannot occur, and the consequences that may occur after the event. more specifically, counterfactuals describe events that are contrary to facts and involve common sense, understanding, and reasoning. therefore, whether the system can correctly identify counterfactual natural language statements is very important. in semeval-2020 task 5: modelling causal reasoning in language-detecting counterfactuals @xcite . the organizer has designed two subtasks. subtask1 requires determining whether a given sentence is counterfactual. solving this problem is the basis for all downstream counterfactual related causal reasoning analysis using natural language. subtask2 aims to locate the premises and results in of counterfactuals, and to point out that causal insights are inherent characteristics of counterfactuals. this is to further test the causal knowledge conveyed in counterfactual statements. in this competition, we only participated in subtask1. we used a deep learning method to build a neural network model based on ordered neuron lstm (on-lstm) @xcite and hierarchical attention network (han) @xcite . the difference between ordered neurons lstm (on-lstm) and original lstm @xcite is that encodes the hierarchical structure of sentences into features to enhance the expression ability of lstm @xcite . the hierarchical attention network (han) model has two important characteristics. the first is layering, the word-level layer and the sentence-level layer, which conforms to the document structure; the second is the attention mechanism, which can give dynamic weight according to the content when weighting. in addition, we have added the pooling operation, and in the pooling layer, we used average pooling and max pooling operations to compress the amount of parameters and reduce overfitting. finally, considering the reason for the small dataset and data imbalance, we used the k-fold approach as the ensemble method. the rest of the paper is structured as follows. section 2 introduces related work to detect counterfactual statements. section 3 describes the dataset and our model. in section 4, we analyze the experimental results. we summarize and looked forward to future work in section 5. ## related work counterfactual statements that describe events that did not occur and their consequences. counterfactuals have been studied in many different fields. logicians and philosophers are concerned with the literal logical relationship between the cause and effect of counterfactual form and the result @xcite . in contrast, political scientists often conduct counterfactual thought experiments on policies, or other aspects of society and evaluate them @xcite . despite extensive research in the field of counterfactual thinking, counterfactuals have not been extensively studied in computational linguistics. first, the base rate of counterfactual statements is lower; we found that only 2% of status updates and 1% of tweets on facebook contain counterfactual statements. second, counterfactual statements can take many forms of natural language 1 . they can use or not use explicit if-or then-clauses (for example, consider "if i haven't met him, then i'll be better" rather than "i hope i haven't met him"). in this section, we review some studies and briefly discuss their findings. @xcite created an anti fact tweet dataset and explored the method of detecting anti fact using rule-based and supervised statistical methods. counterfactual examples for bias (ceb) is designed for non-experts to discover potential biases. using counterfactual examples is a technique for developing a fairer model @xcite @xcite . @xcite proposed a preliminary design of an interactive visualization tool, ceb, in order to reveal the bias in the neural network (nn), which is commonly used in artificial intelligence. ceb combines counterfactual examples with the abstraction of neural network decision-making processes, enabling non-experts to discover biases. in relational classification, counterfactual conditionals can be viewed as a subset of condition type of contingency class in the penn discourse treebank (pdtb) @xcite or the condition relation of rhetorical structure theory (rst) @xcite . moreover, many researchers have tried to use pdtb and rst for end-to-end discourse relationship analysis @xcite @xcite . ## conclusion this paper introduces the general idea and specific scheme of our participation in subtask1 in semeval-2020 task5, which aims to identify which of the two sentences is a counterfactual statement. we use the model based on the combination of on-lstm with han and pooling for classification, and used the k-fold approach to ensemble which could improve model performance. we found that the data is seriously unbalanced. undersampling or oversampling the data can effectively solve the problem of data imbalance, and transfer learning can effectively improve the performance of the model. we will explore these methods in the future.