IITK - RSA at S em E val-2020 Task 5: Detecting Counterfactuals (2020)

Abstract: This paper describes our efforts in tackling Task 5 of SemEval-2020. The task involved detecting a class of textual expressions known as counterfactuals and separating them into their constituent elements. Our final submitted approaches were an ensemble of various fine-tuned transformer-based and CNN-based models for the first subtask and a transformer model with dependency tree information for the second subtask. We ranked 4-th and 9-th in the overall leaderboard. We also explored various other approaches that involved classical methods, other neural architectures and incorporation of different linguistic features.

## introduction counterfactuals @xcite are statements that describe events that did not (or cannot) happen, which implies that their truth cannot be empirically verified. they may be divided into two major categories @xcite . in the first category, the statement discusses possible outcomes had the event in question actually happened, and thus is called a counterfactual conditional. specifically, it contains a logical segment equivalent to "if a, then b" where a did not occur. the "if a" part is called the antecedent and the "then b" is called the consequent. an example of this is "if we had changed the battery on time, the car wouldn't have broken down." a statement of the second type is called a counterfactual wish. they also describe alternate realities, but do not talk about outcomes and do not display a conditional structure. they are called so because they often display 'wish' vocabulary e.g. "i wish there was an easy way to categorise counterfactuals!" here, the segment referring to the fictional event is called the antecedent, and there is no consequent. while there are other ways to categorise these logical forms, we believe that this system covers most of the examples seen in the dataset and provides clear intuition about their general structure. readers interested in a more rigorous treatment may refer to @xcite . the ability to reason via counterfactuals is considered a higher form of intellect, as one must extrapolate the truth using causal relationships, a commonsense understanding of the world and other related events. hence, for artificially intelligent systems, counterfactual reasoning is critical for generalisation and explainability. in terms of more concrete applications, it has been hypothesised to be important for natural language understanding, goal determination (a major challenge in modern reinforcement learning) and error analysis @xcite . @xcite have worked on systems that generate counterfactual explanations for why classifiers behave the way they do on given examples. we tried a variety of approaches on this task, including classical machine learning, cnn-based models, bilstm-based models and transformer-based models. we found that the transformer-based neural models (introduced by @xcite ) consistently outperformed their competitors. we obtained an f1 ## previous and related work research on counterfactuals is a relatively new area in the nlp community, and consequently, limited research literature exists. the closest work is by @xcite which aimed to detect counterfactual elements in tweets. they identify a set of rules to filter such examples based on grammatical form (sentences containing "wish", "should have", etc.) and train an svm to classify filtered sentences on the basis of pos tags and n-gram features. while their method works well on tweets, we found that these rules had limited coverage and were not effective in filtering long sentences like those in our dataset. the filtering also removes several examples, which makes it harder to train further classifiers in an already data-constrained setting. @xcite provides a thorough treatment of counterfactuals in terms of grammar. however, the features and verb forms they identify and use are quite complex, like fake tenses. such morphology is currently undetectable by even state-of-the-art libraries like spacy foot_1 , which makes this system hard to implement with computational models without appropriate data. nevertheless, the paper does allow one to construct certain useful grammatical rules for inference. counterfactual implication and causality are closely linked @xcite . counterfactual conditionals often talk about direct results of the presence (or absence) of a particular factor e.g. "if the gas hadn't run out, we would have been able to cook dinner." even when they do not directly encode such relationships -one may consider sentences like "even if you were the last person on the planet, i wouldn't go out with you!" -they often imply the existence of other factors that drive the relationship. thus, from a linguistic perspective, the vocabulary and structure seen in causal sentences and that seen in counterfactual sentences have a high degree of overlap. hence, we believe systems to detect and parse causal relationships would perform well on this task as well. with respect to the first subtask, fine-tuning neural models using information-rich word embeddings seems to form the state-of-the-art @xcite and forms the backbone of our submitted approach as well. the second subtask bears a resemblance to relation-entity extraction (of which cause-effect may be considered a specific case), which has also been recently dominated by neural models @xcite . we found span-based @xcite and discourse parsing-based @xcite approaches particularly interesting with regards to our task. we tried svms and gradient-boosted random forests on linguistic feature-based (pos tags, verb tense and aspect information, etc.) representations of the samples. the baseline provided by the organisers was an svm classifier on tf-idf features with stop words removed. we felt that this would result in the removal and down-weighting of terms critical for identifying counterfactuality, like 'should have' and 'would have'. indeed, we found that performance improved remarkably by discarding tf-idf features and allowing stop words. this implies that the way that closed class words are used in a sample is particularly informative in this task. even using simple unigram word features provided decent results, which suggests that there are specific words whose presence in itself is useful for detecting counterfactuality. thus, we used these features in our neural model as well (see 4.1.3 c)). even though the results improved by the progressive addition of linguistic features, the f1 score eventually stagnated at around 65%. we then added a regex-based filtering step, where we divided the samples based on certain typical counterfactual forms (inspired by the work of @xcite ) and classified them separately. we used four primary forms -sentences containing an 'if' then a modal verb, sentences containing a modal verb then an 'if', sentences containing the word 'wish', and the leftovers. in the first two, we wished to capture verb forms like 'would have' and 'could have', which we found to have a strong correlation with counterfactuality. we found the modal-if form the hardest to classify, with an f1 of around 52%. the most notable improvements were on the 'wish' form where the f1 was around 90%, which hints that counterfactual wishes are the easiest to identify. classifiers for the other two categories obtained an f1 of around 62-65%, similar to what we obtained before filtering. ## implementation details and results all the scores reported in the paper are on the validation set created by us by randomly splitting the dataset in a ratio of 75%-25%. we used the same training and validation sets across models to make sure the results were comparable. please note we only report results of the key experiments -more details can be found at the github repository. the results are reported in tables 3 and 4 . the highly imbalanced nature of the dataset posed a persistent challenge. we explored four methods to tackle it -simple oversampling of the minority class, smote @xcite , undersampling the majority class, and using a weighted cross-entropy loss. the sampling was done such that the final number of positive and negative samples in the training set were equal. the weights in the cross-entropy loss were set to be the inverse of the class proportions i.e. mistakes on the minority class were penalised much more than those on the majority class. out of all of these, we found that using the weighted cross-entropy loss provided the best results. all of the methods reported use this loss to tackle data imbalance. for classical methods, we used the scikit-learn toolbox foot_2 and spacy to generate linguistic features. a linear svm with the regularisation parameter set as 1.0 yielded the best results. the linear svm has been observed to perform the best in sentence classification tasks @xcite . the implementation for the cnn based models in 4.1.2 was kept similar to the original implementation by @xcite . the convolutional kernel of sizes {3,4,5} was used with relu activation after each convolution and a max-pool layer after the final convolution. a fully-connected layer with the output being the number of classes (2 in our case) completed the model. a dropout @xcite layer was added before the final layer. an adamw @xcite optimiser was used with an initial learning rate of 1 × 10 -3 . as for the transformer models with a cnn head, the word-embedding matrix simply replaced the glove embeddings that were used in a stand-alone cnn model, with the rest of the model remaining the same. the transformer-based models described in 4.1.3 were implemented with the help of the transformers library in pytorch by huggingface foot_3 . we used an adamw optimiser for all of our models with an initial learning rate of 1 × 10 -5 over all the parameters and an epsilon of 1 × 10 -8 . we trained the models for a total of 20 epochs and saved the model after each epoch if it provided the best f1 score on the validation set so far. we found that performance peaked around 16 epochs. we decayed the learning rate linearly to 0 over the training period. the choice of these involved quite a bit of fine-tuning as we found that results were sensitive to these hyper-parameters. the algorithm would not converge outside a specific range, and even within this range, the f1 could vary by 4-5 points upon small changes. as we mentioned earlier, the 'large' versions of the transformers outperformed their 'base' versions by 1-2% in the case of bert and 4-5% in the case of the others. as shown, adding the cnn layer did actually improve performance in the relatively less-pretrained bert, but slightly worsened performance on the sophisticated roberta (large). similarly, our experiments with linguistic features (lf) improved performance on base-sized transformers, but could not beat the performance of a fine-tuned large roberta. the results for the discourse parsing-based (dp) method are also shown here -as mentioned earlier, they perform marginally (0.3%-0.5% f1) better than the models of 4.1.3 b) and c). for subtask 2, the transformer-based model in 4.2.2 was implemented with the help of the simple transformers library foot_4 in pytorch. we tried bert and roberta as the models and used an adamw optimiser with an initial learning rate of 4 × 10 -5 over all the parameters and an epsilon of 1 × 10 -8 , and were trained for 5 epochs. we used spacy for generating and visualising the dependency tree in 4.2.2. ## qualitative analysis of results here we present a brief analysis of the classification results for subtask 1. we try to identify some of the major reasons why our models perform the way they do, mostly by analysing the training data and the false predictions made by our best models. identifying whether a sentence is a counterfactual or not can be a difficult task even for humans -even we were often unable to agree upon whether a particular example in the training data was a counterfactual or not. however, one pattern is quite evident -many counterfactuals are conditionals e.g. of the form "if... then...". hence, the model learns to give a high weight to the embeddings corresponding to these words. unfortunately, this means that a sentence having a similar structure is often falsely classified as a counterfactual. for example, the following statements were falsely classified as counterfactuals: but the major issue in all the models was a low recall, i.e. a high number of false negatives. a low recall value was the major reason why we kept the voting threshold low (less than 50%) for a statement to be classified as a counterfactual in our ensemble models. we believe that this was due to the models not being able to capture certain types of counterfactuals that do not follow a specific sentence structure. for example, consider the sentence: it is possible that the complexity and length of the sentence and the presence of many clauses that were clearly not counterfactual overshadowed the one clause that was a counterfactual and brought down the final score. for subtask 2, the major challenge was that the antecedent and consequent which contributed to counterfactuality were often only parts of the longer sentence and hence it was difficult to conclude where the contributing bits ended. this problem was easier to solve in some cases where there was a pattern to the sentences as in the case of sentences containing 'if'. there we saw that the dependency tree could be relied upon as there was a clear structure to the start of the antecedent, which depended on the main noun/verb of the clause. ## conclusion and future work the identification and the analysis of counterfactual statements are essential for any computational natural language task, be it knowledge extraction or question answering. we attempt to identify such statements from a diverse corpus and produce competitive results using several sota language models. we modify some of the models and also use linguistic features to make our models task-specific, and with an ensemble of these models, we were able to rank 4th on the leaderboard in the first subtask. there is still scope for further improvements, such as hardcoding more specific rules in the model related to the various kinds of counterfactuals. we also attempt to analyse the counterfactual statements by identifying the antecedent and the consequent. we model the problem as a sequence prediction problem and try out multiple transformer-based models as well as probabilistic ones. we ranked 9th on the leaderboard for this subtask. there are some other methods that one can experiment with for this task, such as separating the sentence into clauses and focusing on individual clauses, span-based models, analysing patterns involving other parts-of-speech, and graph-based networks that can leverage the power of the dependency tree correlation that we identified. we leave these methods for future work.