HIT - SCIR at S em E val-2020 Task 5: Training Pre-trained Language Model with Pseudo-labeling Data for Counterfactuals Detection (2020)

Abstract: We describe our system for Task 5 of SemEval 2020: Modelling Causal Reasoning in Language: Detecting Counterfactuals. Despite deep learning has achieved significant success in many fields, it still hardly drives today’s AI to strong AI, as it lacks of causation, which is a fundamental concept in human thinking and reasoning. In this task, we dedicate to detecting causation, especially counterfactuals from texts. We explore multiple pre-trained models to learn basic features and then fine-tune models with counterfactual data and pseudo-labeling data. Our team HIT-SCIR wins the first place (1st) in Sub-task 1 — Detecting Counterfactual Statements and is ranked 4th in Sub-task 2 — Detecting Antecedent and Consequence. In this paper we provide a detailed description of the approach, as well as the results obtained in this task.

## introduction deep learning technologies have been truly remarkable and have caught many attentions of researchers. however, deep learning systems' lack of understanding of causal relations is perhaps the biggest roadblock to giving them human-level intelligence @xcite . causation is a fundamental concept in human thinking and reasoning, which indicates a special semantic relation between the cause with the effect @xcite . @xcite propose that there are three levels of causation, and the top level is the counterfactual analysis. counterfactual statements describe events that did not actually happen or cannot happen, as well as the possible consequence if the events have had happened @xcite . semeval 2020 task 5 consists of two subtasks which aim to detect counterfactual descriptions in sentences, this paper presents solutions to both of two subtasks. subtask1 -detecting counterfactual statements refers to determining whether a given statement is counterfactual or not in several domains. for example, given the sentence: "her post-traumatic stress could have been avoided if a combination of paroxetine and exposure therapy had been prescribed two months earlier.", the system is required to determine whether it is a counterfactual statement or not. this categorization task is evaluated by three evaluation indexes: precision, recall and f1. the challenges of this subtask include: a) positive and negative samples are unevenly distributed in the training data and test data. b) the patterns of counterfactual statements can be hardly defined by rules or learned by traditional word embedding based models. subtask2 -detecting antecedent and consequence aims to locate antecedent and consequent in counterfactuals. a counterfactual statement can be converted to a contrapositive with a true antecedent and consequent @xcite . consider the "post-traumatic stress" example discussed above; it can be transposed into "because her post-traumatic stress was not avoided, (we know) a combination of paroxetine and exposure therapy was not prescribed". such knowledge can be not only used for analyzing the specific statement but also be accumulated across corpora to develop domain causal knowledge (e.g., a combination of paroxetine and exposure may help cure post-traumatic stress). the results are evaluated by four indexes: precision, recall, f1 and exact match. unlike normal sequence labeling tasks, in some cases there is only an antecedent part while without a consequent part in a counterfactual statement. for instance, "thanks for the article on this new term that fits me so well, wish all your articles were worthy of praise.", there is only the antecedent part "wish all your articles were worthy of praise" in this sentence. recently, pre-trained language models have achieved great success in various nlp tasks @xcite @xcite . hence, we first explore multiple pre-trained language models (i.e., bert, xlnet and roberta) stacking methods with fine-tuned in counterfactual data. these models are basically trained in a supervised fashion with labeled data. however, there is not very sufficient labeled data in task 5. hence, we use a simple way of training neural networks in a semi-supervised fashion. basically, each pre-trained language model is trained in a supervised fashion with labeled data. for unlabeled data, pseudo-labels, created by just picking up the class which has more votes from pre-trained language models, are used as if they were true labels. ## related work subtask 1 is a text classification task. traditional machine learning systems, such as naive bayes @xcite and support vector machines (cortes and vapnik, 1995) perform well in this task. in recent years, text classification has made breakthroughs in deep learning. a few of studies @xcite @xcite made a series of improvements to the structure of deep neural networks. @xcite , a dynamic word vector based on the language model, has achieved very competitive performance in multiple domains of text classification. in addition, pre-trained language models such as xlnet @xcite , roberta @xcite , albert @xcite have also achieved high performances in various nlp tasks. subtask 2 is a sequence labeling task. traditional work uses statistical approaches such as hidden markov models (hmm) and conditional random fields (crf) @xcite @xcite . deep learning sequence labeling models utilize word-level long short-term memory (lstm) structures to represent global sequence information and a crf layer to capture dependencies between neighboring labels @xcite @xcite . in similar manner, fine-tuning the pre-trained model to complete the task of sequence labeling such as bert has achieved excellent performances. ## methodology for two subtasks (i.e., text classification task and sequence labeling task), we start by fine-tuning pretrained language models, including bert, roberta and xlnet. then, we enlarge the training set with pseudo-labeled sentences, which are predicted on the test set by voting ensemble of several single pre-trained models. finally, we use the ensemble model to classify counterfactual statements for subtask 1 and utilize a crf model to perform sequence labeling for extracting antecedent and consequence in subtask 2. we will introduce each module in details in the following sections. ## conclusion counterfactuals inference is a challenging problem in causation, which is crucial for today's ai system. in this paper, we designed and implemented an ensemble model for detecting counterfactuals. it achieved the top f1 score of 90.90% on the evaluation-subtask 1 leaderboard and 4th f1 score of 84.10% on the evaluation-subtask 2 leaderboard. task 5 mainly explores how to detect counterfactuals, which is the basis of counterfactuals inference. we believe that causation, especially counterfactuals will attract more attention from researchers driven by this task.