CNRL at S em E val-2020 Task 5: Modelling Causal Reasoning in Language with Multi-Head Self-Attention Weights Based Counterfactual Detection (2020)

Abstract: In this paper, we describe an approach for modelling causal reasoning in natural language by detecting counterfactuals in text using multi-head self-attention weights. We use pre-trained transformer models to extract contextual embeddings and self-attention weights from the text. We show the use of convolutional layers to extract task-specific features from these self-attention weights. Further, we describe a fine-tuning approach with a common base model for knowledge sharing between the two closely related sub-tasks for counterfactual detection. We analyze and compare the performance of various transformer models in our experiments. Finally, we perform a qualitative analysis with the multi-head self-attention weights to interpret our models’ dynamics.

## introduction causal reasoning is a process of detecting cause-effect relationships and is increasingly being used in artificial intelligence for improving generalization and interpretability. modelling causal reasoning in language involves detecting such cause-effect relationships from natural language texts. a cause-effect relationship can be modelled as: event a causes event b. counterfactuals describe events counter to facts and hence naturally involve common sense and causal reasoning. a counterfactual can be modelled as a cause-effect relationship of the form: event a could have caused event b (event a did not occur). semeval-2020 task-5 @xcite ) consists of two independent sub-tasks: a binary-classification task for detecting counterfactual statements and a span-detection task for detecting antecedent (cause)consequent (effect) spans of given counterfactual statements ( ## background early work on causal reasoning and related tasks in natural language was based on various statistical and linguistic approaches @xcite . recent work for causal reasoning related tasks involves deep learning based approaches. causal reasoning can be achieved through extraction of cause-effect relations with crf and lstm based sequence labelling tasks @xcite . counterfactuals can contain implicit causal relations ( ## experiments for all our experiments, we use binary cross entropy loss (for counterfactual detection) and smooth l1 loss (for antecedent-consequent span regression) with adam optimizer (with weight decay) to train our models. we use the pytorch implementations of the smallest base variants of pre-trained transformer models by hugging face foot_1 @xcite in our base architecture. we use a 90-10 data split for training and development purposes respectively. the data distribution across the splits is shown in ## discussion for the final submission, we use roberta and bert as the transformer model in our base architecture for sub-task 1 and sub-task 2 respectively. on the final task leaderboard, our system ranks 13 th (0.845 f1) for the counterfactual detection sub-task and 7 th (0.688 f1) for the antecedent-consequent detection sub-task. since we treat antecedent-consequent detection as a regression task, we do not monitor the exact match score between the predicted and ground truth spans, which is more significant for token-level sequence labelling based approaches. we analyse the performance of various transformer models with hyperparameter tuning post evaluation ( the superscripts here represent the most attending attention-head(s) for the corresponding word. the same can be confirmed by a visualization (figure 2 ) of the head-wise color coded self-attention weights. for example 3 and 4, we have no consequent part in the text. our system detects (0,0) as consequent span start and end locations for such counterfactual statements, indicating the absence of the consequent. the conjunctions (but / and) in such counterfactual statements are ignored by the attention-heads. but the conjunctions (if ) in counterfactual statements with a consequent part (example 1,2 and 5) are highly attended by the attention-heads through the tokens from entire sequence. this shows the ability of the model to differentiate the causal connectives from the non-causal ones in the text. punctuation play an important role here as they are usually present near the boundaries of antecedent-consequent spans (example 2 and 3). auxiliary verbs (would, wouldn't, could, have) are assigned maximum attention across all the examples as they directly correspond to the conditional nature of counterfactual statements. ## conclusion our proposed approach uses multi-head self-attention weights from transformer models to detect causal relations for counterfactual detection in text. through our experiments, we find that roberta overall shows the best performance for counterfactual detection task and bert performs the best for the antecedent-consequent detection task. we show that even smaller transformer models like distilbert perform counterfactual detection tasks effectively. with knowledge sharing between the two sub-tasks, our system detects antecedent-consequent spans in counterfactual statements with good efficiency by a simple regression over the spans. this can possibly be further improved by post-inference processing on the predicted spans or replacing the regression module with a token-level sequence labelling module. further, we show that through our approach, the attention-heads attain a property of assigning linguistically selective-attention with respect to the conditional nature of the counterfactual statements.