L ang R esearch L ab_ NC at F in C ausal 2020, Task 1: A Knowledge Induced Neural Net for Causality Detection (2020)

Abstract: Identifying causal relationships in a text is essential for achieving comprehensive natural language understanding. The present work proposes a combination of features derived from pre-trained BERT with linguistic features for training a supervised classifier for the task of Causality Detection. The Linguistic features help to inject knowledge about the semantic and syntactic structure of the input sentences. Experiments on the FinCausal Shared Task1 datasets indicate that the combination of Linguistic features with BERT improves overall performance for causality detection. The proposed system achieves a weighted average F1 score of 0.952 on the post-evaluation dataset.

## introduction 1 the understanding of cause-effect relation is an important nlp task because it appeals to human perception, reasoning, and decision-making. it has vast applications in the field of information extraction @xcite , question answering @xcite , and event prediction @xcite , among others. however, modeling causality relations between events is a non-trivial task because it requires a deeper analysis of the discourse and sometimes external knowledge to forge the relationship between separate events and entities. in the present work, the focus is on detection of causal relationships in a given text, which is modeled as a binary classification task. sometimes the presence of causal connectives, such as causes, because of, leads to, after, due to indicates causality. however, there may be cases when the causal relation is more implicit making the task of causality detection more challenging. a causal relationship in a sentence involves the presence of a cause and an effect, where the cause triggers the effect. in other words, two events x and y are considered to be causally related if the occurrence of x is triggered by the occurrence of y, or vice versa. for illustration, consider the following: (1) fluctuations in exchange rates added to the risk factors. past studies revealed that two major approaches for causality detection involve the use of handcrafted features @xcite or deep neural networks @xcite . the proposed work integrates syntactic and semantic features of the input text with pre-trained embedding vectors to train a supervised neural network for the classification of causal relations. the rest of the paper is organized as follows. the proposed system is described in section 2, section 3 contains implementation details, and experimental results are presented in section 4. ## model architecture the proposed model aims to supplement pre-trained bert @xcite embeddings with linguistic features in order to induce knowledge about the syntactic and semantic peculiarities of the this work is licensed under a creative commons attribution 4.0 international license. license details: http://creativecommons.org/licenses/by/4.0/ input in the model. the model architecture is presented in figure 1 . the linguistic feature vector and the bert embeddings are concatenated together to generate a linguistically informed representation of the input text. the enhanced representations are processed using two identical layers of fully connected feed-forward network before applying a softmax classifier. the details of linguistic features are presented in sec 2.1, and bert features is described in sec 2.2. ## implementation details pre-processing: in the pre-processing step, spacy foot_2 library is used to perform tokenization, lemmatization, sentence segmentation, part-of-speech (pos) tagging and dependency parsing. spacy's named entity recognizer is employed to identify entities belonging to cardinals, monetary amounts, and percentages. constituency parsing is derived using the benepar_en2 model @xcite . ## results and analysis in this section we present the results of our experiments on the blind evaluation-task1 dataset. the evaluation metrics are precision, recall and weighted f1 score. weighted f1 score is calculated by multiplying the class wise f1-scores with the class support, i.e. the number of examples in that class. the results corresponding to different subsets of the feature space is given in predictions of the proposed model for samples taken from the validation set are described in ## conclusion causality detection in a text is a challenging task due to the semantic peculiarities of the english language and also because it requires a deeper domain understanding. in the present work, semantic and structural knowledge of the input text is induced on the top of input embeddings to generate enhanced representation. the results indicate that the enhanced representations improve the performance across all the evaluation metrics. in future work we would like to experiment with more complex architectures such as lstms and transformers. additionally, we would also like to experiment with cnn and max-pooling based dimensional reduction for treatment of the sparse linguistic feature space. 8 best results are obtained for sampling seed 5050 9 https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/earlystopping