Yseop at S em E val-2020 Task 5: Cascaded BERT Language Model for Counterfactual Statement Analysis (2020)

Abstract: In this paper, we explore strategies to detect and evaluate counterfactual sentences. We describe our system for SemEval-2020 Task 5: Modeling Causal Reasoning in Language: Detecting Counterfactuals. We use a BERT base model for the classification task and build a hybrid BERT Multi-Layer Perceptron system to handle the sequence identification task. Our experiments show that while introducing syntactic and semantic features does little in improving the system in the classification task, using these types of features as cascaded linear inputs to fine-tune the sequence-delimiting ability of the model ensures it outperforms other similar-purpose complex systems like BiLSTM-CRF in the second task. Our system achieves an F1 score of 85.00% in Task 1 and 83.90% in Task 2.

## introduction a counterfactual can be defined as something that is contrary to the truth or that did not actually occur. it refers to an event that did not or cannot happen, as well as the possible consequences if it had happened. in the sentence "if dogs had no ears, they could not hear", the statement "if dogs had no ears" is an example of a counterfactual because dogs do have ears. task 5 of semeval-2020 @xcite focuses on identifying these specific sentence types amongst sentences delivering close semantic similarities. this implies understanding and disambiguating the causal link between two sentence fragments. we approached this task as an opportunity to test the effectiveness of disambiguating at a grammatical level against traditional baseline systems. this paper describes a parallel approach in deriving some meaning from text to leverage the influence of context and relevance of structure in recognizing counterfactual statements. specifically, we will explore how many expressions of such statements a mapping of grammatical types can cover before falling short to high-performing models, most notably bert @xcite , which performed well on both tasks with an f1 score of 85.00% on task 1 and 83.90 % on task 2. ## related work although the task of detecting counterfactuals is relatively new, @xcite proposes using modal logic to form rule-based determination methods from social media posts. these methods are supplemented by a statistical classifier (linear svm) that is retrained to tackle more challenging counterfactual forms. previous work on causal identification by @xcite studied the contribution of verbs in the determination of causal relations. by analyzing closely different formulations, they concluded that similarities in meaning can be derived from very different syntactic structures as for causality relation extraction, different deep learning systems built on the success of neural-based models have been proposed. the linguistically informed cnn model @xcite leverages the use of word embeddings and other linguistic features to detect causal patterns and outperforms rule-based classification. @xcite introduces a multi-level causal detector that makes use of multi-head self-attention to capture semantic features at word level and infer causality at segment level. this engineered system has rivaled state-of-the-art models in terms of performance and thorough understanding of complex semantic information such as discourse relations and transitivity rules. finally, @xcite presents a self-attentive bilstm-crf model that makes use of transfer learning to overcome the problem of data insufficiency and extract causal relations in natural language text. this solution transfers a trained embedding from a large corpus and uses the causality tagging scheme to identify dependencies between cause and effect. experimental results prove the effectiveness of this model, but its major limitation is the insufficiency of high-quality annotated data to learn from. ## dataset the datasets used are those provided by the shared task organisers. the data is described in @xcite . as per official task instructions, no additional data was used. the proposed baseline by the task organisers is a support vector machine (svm) classifier. after a preliminary phase based on exploring grammatical features engineering, we discuss two different approaches and compare them to the baseline: a classical approach using popular machine learning classifiers with semantic and additional grammatical features, a combination that has shown to perform well on information retrieval tasks relying on text understanding @xcite , and a deep learning approach using a bert linear classifier. the objective of this competitive approach is to determine whether a supplement of linguistic features can be sufficient to correctly recognise counterfactual structures as opposed to running heavier models integrating broader contextual knowledge. ## conclusion in this paper, we presented our experiments for identifying counterfactual statements. modeling features derived from a linguistic analysis such as specific grammar structures for counterfactual statements and coupling them with established machine learning or deep learning models did not perform as well as context-learning models, as our hybrid bert-mlp solution outperforms even complex combinations of deep learners and displays a better level of understanding and handling challenging counterfactual forms. future work could explore the impact of graph knowledge in accommodating systems and rendering them more perceptive of implicit and ambiguous textual meanings. feature pattern example verb @xmath1.*> <@xmath2.*> <@xmath3.*> <@xmath4.*> <@xmath5.*> <@xmath6.*> <@xmath7.*> <@xmath8.*> <@xmath9.*> <nn.*> <@xmath10.*> <@xmath11.*> <@xmath12.*> <@xmath13.*> <@xmath14.*> <@xmath15.*> <@xmath16.*> <@xmath17.*> <nn.*> <@xmath18.*> <@xmath19.*> <@xmath20 and transforming text to lowercase. we also replace multiple white spaces with a single space. examples of these transformations are provided in for the morpho-syntactic phase, we apply pos-based stemming and lemmatisation for the bow and tf-idf embeddings. we also remove words with frequency less than 5 for these embeddings. this effectively decreases the dimensions of bow and tf-idf vectors. our final feature list consists of 2000 features that are the unique lemmatized vocabulary words and word groups curated from the input text. however, despite their simplicity and low time complexity, bow and tf-idf have two major drawbacks. first, as the size of the data and the number of unique words in the training text increase, the length of vectors becomes much larger. moreover, in these two approaches, only words and their repetitions are important and the order of the words in the text is not considered in the model. this is why we also consider both word2vec and bert embedding approaches in our experiments.