Knowledge-Augmented Language Models for Cause-Effect Relation Classification (2022)

Abstract: Previous studies have shown the efficacy of knowledge augmentation methods in pretrained language models. However, these methods behave differently across domains and downstream tasks. In this work, we investigate the augmentation of pretrained language models with knowledge graph data in the cause-effect relation classification and commonsense causal reasoning tasks. After automatically verbalizing triples in ATOMIC2020, a wide coverage commonsense reasoning knowledge graph, we continually pretrain BERT and evaluate the resulting model on cause-effect pair classification and answering commonsense causal reasoning questions. Our results show that a continually pretrained language model augmented with commonsense reasoning knowledge outperforms our baselines on two commonsense causal reasoning benchmarks, COPA and BCOPA-CE, and a Temporal and Causal Reasoning (TCR) dataset, without additional improvement in model architecture or using quality-enhanced data for fine-tuning.

## introduction automatic extraction and classification of causal relations in text has been an important yet challenging task in natural language understanding. early methods in the 80s and 90s @xcite @xcite ) mainly relied on defining hand-crafted rules to find cause-effect relations. starting 2000, machine learning tools were utilized in building causal relation extraction models @xcite @xcite @xcite @xcite . word-embeddings and pretrained language models (plms) have also been leveraged in training models for understanding causality in language in recent years @xcite @xcite . investigating the true capability of pretrained language models in understanding causality in text is still an open question. more recently, knowledge graphs (kgs) have been used in combination with pretrained language models to address commonsense reasoning. two examples are causal-bert @xcite for guided generation of cause and effect and the model introduced by @xcite for commonsense story generation. atomic 2020 motivated by the success of continual pretraining of plms for downstream tasks (gururangan et al., 2020), we explore the impact of common sense knowledge injection as a form of continual pretraining for causal reasoning and cause-effect relation classification. it is worth highlighting that even though there are studies to show the efficacy of knowledge injection with continual pretraining for commonsense reasoning @xcite , performance of these techniques is very dependent on the domain and downstream tasks @xcite . and, to the best of our knowledge, there are limited studies on the effect of commonsense knowledge injection with knowledge graph data on cause-effect relation classification @xcite . our contributions are as follows: the overview of our method is shown in figure 1 . 1 we first convert triples in atomic 20 20 @xcite knowledge graph to natural language texts. then we continually pretrain bert using masked language modeling (mlm) and evaluate performance of the resulting model on different benchmarks. samples in atomic 20 20 are stored as triples in the form of (head/subject, relation, tail/target) in three splits including train, development, and test. atomic 20 20 has 23 relation types that are classified into three categorical types including commonsense relations of social interactions, physicalentity commonsense relations, and event-centric commonsense relations. in the rest of the paper, we refer to these three categories as social, physical, and event, respectively. ## results and discussion results of our experiments on tcr are shown in model acc (%) joint system @xcite 77.3 bert-large (baseline) ❈ 75.0 atomic-bert-large m lm ❈ 91.0 results of experiments on copa-test are shown in ## conclusion we introduced a simple framework for augmenting plms with commonsense knowledge created by automatically verbalizing atomic 20 20 . our results show that commonsense knowledge-augmented plms outperform the original plms on causeeffect pair classification and answering commonsense causal reasoning questions. as the next step, it would be interesting to see how the previously proposed model improvement methods or using unbiased fine-tuning datasets can potentially enhance the performance of our knowledgeaugmented models.