ISCAS at S em E val-2020 Task 5: Pre-trained Transformers for Counterfactual Statement Modeling (2020)

Abstract: ISCAS participated in two subtasks of SemEval 2020 Task 5: detecting counterfactual statements and detecting antecedent and consequence. This paper describes our system which is based on pretrained transformers. For the first subtask, we train several transformer-based classifiers for detecting counterfactual statements. For the second subtask, we formulate antecedent and consequence extraction as a query-based question answering problem. The two subsystems both achieved third place in the evaluation. Our system is openly released at https://github.com/casnlu/ISCASSemEval2020Task5 .

## introduction counterfactual statements describe events that did not actually happen or cannot occur, as well as the possible consequence if the events have had happened. counterfactual detecting aims to identify counterfactual statements in language and understand antecedents and consequents in these statements. for instance, the following sentence is a counterfactual statement, and the underlined term is the antecedent, while the italic term is the consequence: her post-traumatic stress could have been avoided if a combination of paroxetine and exposure therapy had been prescribed two months earlier. once understanding the statement, we can accumulate the causal knowledge for "post-traumatic stress", i.e., "a combination of paroxetine and exposure may help cure post-traumatic stress". to model counterfactual semantics and reason in natural language, semeval 2020 subtask 5 provides an english benchmark for two basic problems: detecting counterfactual statements and detecting antecedent and consequence @xcite . we build our evaluation systems that are built on pre-trained transformer-based neural network models, which have shown significant improvements over conventional methods in many nlp fields @xcite @xcite . specifically, in subtask 1, several transformer-based classifiers are designed to detect counterfactual statements. besides, because counterfactual antecedent expressions are usually expressed using some obvious conditional assumption connectives, such as if and wish. we also equip transformers with additional convolutional neural network to capture the above strong local context information. for subtask 2, we formulate antecedent and consequence extraction as a query-based question answering problem. specifically, to effectively model context information in counterfactual statements, we design two different kinds of input queries for antecedents/consequences and regard counterfactual statements as given paragraphs. the rest of this paper is organized as follows. section 2 introduces the background of pre-trained transformers. section 3 describes the overview of our system for two subtasks. in section 4-5, we describe the detailed experiment setup and the overall system performance on the two subtasks. finally, we conclude this paper in section 6. ## background different from the pre-trained word embedding in nlp @xcite , pre-trained contextualized models aim to learn encoders to represent words in context for downstream tasks. bert @xcite is a representative large-scale pre-trained transformer, which is trained using mask language modeling (mlm) and next sentence prediction (nsp) task. whole word masking model foot_0 (bert-wwm) is a simple but effective variant of bert. in this case, the pre-training stage always mask all of the tokens corresponding to a word instead of a single wordpiece token (sub-token). roberta @xcite further improves on bert's pre-training procedure and achieves substantial improvements. the improvements include training the model longer, with bigger batches over more data; removing the next sentence prediction objective; training on longer sequences; and dynamically changing the masking pattern applied to the training data. albert @xcite incorporates factorized embedding parameterization and cross-layer parameter sharing to reduce the number of parameters of bert. these two methods can significantly reduce the number of parameters of bert, thus improving the parameter efficiency and facilitating the learning of larger models. besides, albert also uses sentence ordering prediction (sop) self-supervised learning task to replace bert's nsp task, for it's helpful for the model to better learn sentence coherence. ## system overview given a candidate text @xmath0, w 2 , ..., w n }, our system needs to: 1) determine whether the candidate contains a counterfactual statement; 2) extract the antecedent and consequence from the counterfactual statement. for the example in section 1, we first detect it is a counterfactual statement, then extract "her post-traumatic stress could have been avoided" as its antecedent and "if a combination of paroxetine and exposuretherapy had been prescribed two months earlier" as its consequent. in the following, we describe our two sub-systems in detail. ## results we report the performance of subtask 1 and subtask 2, scored by the evaluation server foot_3 . in subtask 1, different models are evaluated using precision (p ), recall (r), and f1-score (f 1 ) for binary classification. while there are four metrics for subtask 2: exact match (em ), precision, recall, and f1-score. exact match measures the percentage of predictions that match the annotated antecedents and consequences exactly. note that, f1-score in subtask 2 is token level metric and will be calculated based on the offsets of predicted antecedent and consequence. ## conclusion in this paper, we propose a transformers-based system for counterfactual modeling. for counterfactual statements detection, we investigated a variety of advanced pretraining models and two efficient aggregation algorithms. for antecedent and consequent extraction, we framed it as a span-based question answering task, and then definition-enriched queries are designed to extract the required term from counterfactual statements. evaluation results demonstrate the effectiveness of our system. for future work, we plan to investigate how to inject extra-knowledge into counterfactual modeling systems, such as knowledge-enriched transformers.