BUT - FIT at S em E val-2020 Task 5: Automatic Detection of Counterfactual Statements with Deep Pre-trained Language Representation Models (2020)

Abstract: This paper describes BUT-FIT’s submission at SemEval-2020 Task 5: Modelling Causal Reasoning in Language: Detecting Counterfactuals. The challenge focused on detecting whether a given statement contains a counterfactual (Subtask 1) and extracting both antecedent and consequent parts of the counterfactual from the text (Subtask 2). We experimented with various state-of-the-art language representation models (LRMs). We found RoBERTa LRM to perform the best in both subtasks. We achieved the first place in both exact match and F1 for Subtask 2 and ranked second for Subtask 1.

## introduction one of the concerns of semeval-2020 task 5: modelling causal reasoning in language: detecting counterfactuals @xcite is to research the extent to which current state-of-the-art systems can detect counterfactual statements. a counterfactual statement, as defined in this competition, is a conditional composed of two parts. the former part is the antecedent -a statement that is contradictory to known facts. the latter is the consequent -a statement that describes what would happen had the antecedent held 1 . to detect a counterfactual statement, the system often needs to posses a commonsense world knowledge to detect whether the antecedent contradicts with it. in addition, such a system must have an ability to reason over consequences that would arise had the antecedent would have been true. in some cases, the consequent might not be present at all, but instead a sequence resembling consequent, but with no consequential statement, might be present. figure 1 shows a set of examples drawn from the data. ## results and analysis for subtask 1, we adapted the baseline provided by the task organizer to asses how more classical machine learning approaches perform on the dataset. after seeing the subpar performance, we turned our attention to pre-trained lrms, namely roberta and albert. the results of the best run of each model can be found in our final submission is an ensemble of roberta-large models since we found that this lrm performs better than albert for this task. we trained a number of models on the train set and computed f1 scores on the validation part. 10 best (in terms of f1) single models were selected, and the output probabilities were averaged for all the possible combinations of these models. the combination with highest f1 score was selected as a final ensemble. then we trained new models with the same parameters as the models in the ensemble, but using the whole training data, including the part that was previously used for validation. finally, for our submitted ensemble, we used checkpoints saved after the same number of updates as the best checkpoints for the systems trained only on part of the training data. the statistics were collected from 10 runs. #θ denotes the number of model's parameters. we also measured em/f1 for the extraction of antecedent/consequent separately; denoted as a em , a f 1 and c em , c f 1 respectively. at last acc no-c denotes no-consequent classification accuracy. for subtask 2, the results are presented in ## related work closest to our work, @xcite created a counterfactual tweet dataset and built a pipeline classifier to detect counterfactuals. the authors identified 7 distinct categories of counterfactuals and firstly attempted to classify the examples into one of these categories using a set of rules. then for certain categories, they used a linear svm classifier @xcite to filter out tricky false positives. a large effort in computational linguistics was devoted to the specific form of counterfactuals -so-called what-if questions. a recent paper by @xcite presents a new dataset for what-if question answering, including a strong, bert-based baseline. the task is to choose an answer to a hypothetical question about cause and an effect, e.g. do more wildfires result in more erosion by the ocean?. each question is accompanied by a paragraph focused on the topic of the question, which may or may not contain enough information to choose the correct option. the authors show that there is still a large performance gap between humans and state-of-the-art models (73.8% accuracy for bert against 96.3% for a human). this gap is caused mainly by the inability of the bert model to answer more complicated questions based on indirect effects, which require more reasoning steps. however, the results show that the bert model was able to answer a large portion of the questions even without accompanying paragraphs, indicating that the lrm models have a notion of commonsense knowledge. ## conclusions we examined the performance of current state-of-the-art language representation models on both subtasks and we found yet another nlp task benefits from unsupervised pre-training. in both cases, we found roberta model to perform slightly better than other lrms, while its results also being more stable. we have ended up first in both em and f1 on subtask 2 and second in subtask 1.