Knowledge-Augmented Language Models for Cause-Effect Relation Classification
Abstract
Previous studies have shown the efficacy of knowledge augmentation methods in pretrained language models.
However, these methods behave differently across domains and downstream tasks.
In this work, we investigate the augmentation of pretrained language models with knowledge graph data in the causeeffect relation classification and commonsense causal reasoning tasks.
After automatically verbalizing triples in ATOMIC 2020 , a wide coverage commonsense reasoning knowledge graph, we continually pretrain BERT and evaluate the resulting model on cause-effect pair classification and answering commonsense causal reasoning questions.
Our results show that a continually pretrained language model augmented with commonsense reasoning knowledge outperforms our baselines on two commonsense causal reasoning benchmarks, COPA and BCOPA-CE, and a Temporal and Causal
Reasoning (TCR) dataset, without additional improvement in model architecture or using quality-enhanced data for fine-tuning.
Introduction
Automatic extraction and classification of causal relations in text has been an important yet challenging task in natural language understanding.
Early methods in the 80s and 90s (Joskowicz et al., 1989;
Kaplan and Berry-Rogghe, 1991; Garcia et al., 1997; Khoo et al., 1998) mainly relied on defining hand-crafted rules to find cause-effect relations.
Starting 2000, machine learning tools were utilized in building causal relation extraction models (Girju, 2003; Chang and Choi, 2004, 2006; Blanco et al., 2008; Do et al., 2011; Hashimoto et al., 2012;
Hidey and McKeown, 2016).
Word-embeddings and Pretrained Language Models (PLMs) have also been leveraged in training models for understanding causality in language in recent years (Dunietz et al., 2018; Pennington et al., 2014;
Dasgupta et al., 2018; Gao et al., 2019).
Investigating the true capability of pretrained language models in understanding causality in text is still an open question.
More recently, Knowledge Graphs (KGs) have been used in combination with pretrained language models to address commonsense reasoning.
Two examples are CausalBERT (Li et al., 2020) for guided generation of Cause and Effect and the model introduced by Guan et al.
(2020) for commonsense story generation.
[Figure]
Motivated by the success of continual pretraining of PLMs for downstream tasks (Gururangan et al., 2020), we explore the impact of common sense knowledge injection as a form of continual pretraining for causal reasoning and cause-effect relation classification.
It is worth highlighting that even though there are studies to show the efficacy of knowledge injection with continual pretraining for commonsense reasoning (Guan et al., 2020), performance of these techniques is very dependent on the domain and downstream tasks (Gururangan et al., 2020).
And, to the best of our knowledge, there are limited studies on the effect of commonsense knowledge injection with knowledge graph data on cause-effect relation classification (Dalal et al., 2021).
Our contributions are as follows:
• We study performance of PLMs augmented with knowledge graph data in the less investigated cause-effect relation classification task.
• We demonstrate that a simple masked language modeling framework using automatically verbalized knowledge graph triples, without any further model improvement (e
g., new architecture or loss function) or quality enhanced data for fine-tuning, can significantly boost the performance in cause-effect pair classification.
• We publicly release our knowledge graph verbalization codes and continually pretrained models.
Method
The overview of our method is shown in Figure 1.
We first convert triples in ATOMIC 2020 (Hwang et al.
, knowledge graph to natural language texts.
Then we continually pretrain BERT using Masked
Language Modeling (MLM) and evaluate performance of the resulting model on different benchmarks.
Samples in ATOMIC 2020 are stored as triples in the form of (head/subject, relation, tail/target) in three splits including train, development, and test.
ATOMIC 2020 has 23 relation types that are classified into three categorical types including commonsense relations of social interactions, physicalentity commonsense relations, and event-centric commonsense relations.
In the rest of the paper, we refer to these three categories as social, physical, and event, respectively.
We remove all duplicates and ignore all triples in which the target value is none .
Moreover, we ignore all triples that include a blank.
Since in masked language modeling we need to know the gold value of masked tokens, a triple that already has a blank (masked token/word) in it may not help our pretraining.
For instance, in the triple: [PersonX affords another ___, xAttr, useful] it is hard to know why or understand what it means for a person to be useful without knowing what they afforded.
This preprocessing step yields in 782,848 triples with 121,681, https:// 177,706, and 483,461 from event, physical, and social categories, respectively.
Distribution of these relations is shown in Figure 2. x R e a s o n C a u s e s D e s i r e s
N o t D
e
s i r e s
M a d e U p O f
H
a s P r o p e r t y
C a p a b l e O f
H a s S u b E v e n t
i s A f t e r
i s B e f o r e
A t L o c a t i o n o R e a c t o E ff e c t o W a n t x I n t e n t
x
R e a c t x E
ff e c t
x N ee d
H i n d e r e d B y x W a n t x A tt r O b j e c t U s e
[Figure]
[Figure]
We verbalize ATOMIC2020 knowledge graph
2. Converting Triples
Each relation in ATOMIC 2020 is associated with a human-readable template.
For example, xEffect ’s and HasPrerequisite ’s templates are as a result,
PersonX will and to do this, one requires , respectively.
We use these templates to convert triples in ATOMIC 2020 to sentences in natural language by concatenating the subject, relation template, and target.
Examples of converting triples to text are shown in Figure 3.
When we convert triples to natural language text, ideally we want to have grammatically correct sentences.
Human readable templates provided by ATOMIC 2020 are not necessarily rendered in a way to form error-free sentences when concatenated with subject and target in a triple.
To address this issue, we use an open-source grammar and spell checker, LanguageTool, 2 to double-check our converted triples to ensure they do not contain obvious grammatical mistakes or spelling errors.
Similar approaches that include deterministic grammatical transformations were also previously used to convert KG triples to coherent sentences (Davison et al., 2019).
It is worth pointing out that the DataTo-Text generation (KG verbalization) for itself is a separate task and there have been efforts to address this task (Agarwal et al., 2021).
We leave investigating the effects of using other Data-To-Text and grammar-checking methods to future research. 2. Continual Pretraining
As mentioned earlier, we use MLM to continually pretrain our PLM, BERT-large-cased (Devlin et al., 2018).
We follow the same procedure as BERT to create the input data to our pretraining (e.g., number of tokens to mask in input examples).
We run the pretraining using ATOMIC 2020 ’s train and development splits as our training and evaluation sets, respectively, for 10 epochs on Google Colab TPU v2 using PyTorch/XLA package with a maximum sequence length of 30 and batch size of 128. 3 To avoid overfitting, we use early stopping with the patience of 3 on evaluation loss.
We select the best model based on the lowest evaluation loss at the end of training.
Experiments 3. Benchmarks
We chose multiple benchmarks of commonsense causal reasoning and cause-effect relation classification to ensure we thoroughly test the effects of our newly trained models.
These benchmarks include: 1) Temporal and Causal Reasoning (TCR) dataset (Ning et al., 2018), a benchmark for joint reasoning of temporal and causal relations; 2)
Choice Of Plausible Alternatives (COPA) (Roemmele et al., 2011) dataset which is a widely used and notable benchmark (Rogers et al., 2021) for commonsense causal reasoning; And 3) BCOPACE (Han and Wang, 2021), a new benchmark inspired by COPA, that contains unbiased token distributions which makes it a more challenging benchmark.
For COPA-related experiments, since
COPA does not have a training set, we use COPA’s development set for fine-tuning our models and testing them on COPA’s test set (COPA-test) and
BCOPA-CE.
For hyperparameter tuning, we randomly split COPA’s development set into train (%90) and dev (%10) and find the best learning rate, batch size, and number of train epochs based on the evaluation accuracy on the development set. Then using COPA’s original development set and best set of hyperparameters, we fine-tune our models and evaluate them on the test set.
In all experiments, we report the average performance of models using four different random seeds.
For TCR, we fine-tune and evaluate our models on train and test splits, respectively.
3. Models and Baseline
We use bert-large-cased pre-trained model in all experiments as our baseline.
For COPA and BCOPACE, we convert all instances to a SWAG-formatted data (Zellers et al., 2018) and use Huggingface’s
BertForMultipleChoice –a BERT model with a multiple-choice classification head on top.
And for
TCR, we convert every instance by adding special tokens to input sequences as event boundaries and use the R-BERT 5 model (Wu and He, 2019).
We chose R-BERT for our relation classification since it not only leverages the pretrained embeddings but also transfers information of target entities (e.g., events in a relation) through model’s architecture and incorporates encodings of the targets entities.
Examples of COPA and TCR are shown in Figure 4.
BCOPA-CE has the same format as COPA.
[Figure]
𝐻
[Figure] O
P A T C R The death toll <e1> climbed </e1> to 99 on Sunday after a suicide car bomb <e2> exploded </e2> Friday in the middle of a group of men playing volleyball in northwest Pakistan, police said.
Cause-Effect(e1,e2) [CLS] The computer crashed.
[SEP] I backed up my files. [SEP] [CLS]
The computer crashed.
[SEP] I downloaded a virus.
[SEP]  Results and Discussion
Results of our experiments on TCR are shown in
Table 1.
As can be seen, our model significantly outperforms both our baseline and the joint infer- ence framework by Ning et al.
(2018) formulated as an integer linear programming (ILP) problem.
[Table]
Results of experiments on COPA-test are shown in Table 2.
We initially observed that a continually pretrained model using all three types of relations has a lower performance than our baseline.
By taking a closer look at each relation type, we decided to train another model, this time only using the event relations.
The reason is that event-centric relations in ATOMIC 2020 specifically contain commonsense knowledge about event interaction for understating likely causal relations between events in the world (Hwang et al., 2021).
In addition, event relations have a relatively longer context (# of tokens) than the average of all three relation types combined which means more context for a model to learn from.
Our new pretrained model outperformed the baseline by nearly %5 which shows the effect of augmented pretrained language model with commonsense reasoning knowledge [Table]
For a fair comparison, we report BERT-SocialIQA’s average performance.
We further experiment on the Easy and Hard question splits in COPA-test separated by Kavumba et al.
(2019) to see how our best model performs on harder questions that do not contain superficial cues.
Results are shown in Table 3.
As can be seen, our ATOMIC-BERT model significantly outperforms both the baseline and former models on
Hard and Easy questions.
[Table]
It is worth mentioning three points here.
First, our model, BERT-large, has a significantly lower number of parameters than state-of-the-art models,
Google T5-11B ( ∼ 32x) and DeBERTa-1.5B ( ∼ 4x) and it shows how smaller models can be competitive and benefit from continual pretraining.
Second, we have not yet applied any model improvement methods such as using a margin-based loss introduced by Li et al.
and used in CausalBERT (Li et al., 2020), an extra regularization loss proposed by Han and Wang (2021), or fine-tuning with quality-enhanced training data, BCOPA, introduced by Kavumba et al.
As a result, there is still great room to improve current models that can be a proper next step.
Third, we achieved a performance on par with BERT-SocialIQA (Sap et al., 2019) 6 while we did not use crowdsourcing or any manual re-writing/correction, which is expensive, for verbalizing KG triples to create our pretraining data. [Table] [Table]
[Table]
[Table]
4. BCOPA-CE: Prompt vs. No Prompt
Results of experiments on BCOPA-CE are shown in Table 4.
As expected based on the results also reported by Han and Wang (2021), we initially observed that our models are performing nearly as random baseline.
Since we do not use the type of question when encoding input sequences, we decided to see whether adding question type as a prompt to input sequences will improve the performance.
We added It is because and As a result, as prompt for asks-for="cause" and asks-for="effect" , respectively.
Interestingly, the new model outperforms the baseline and Han and Wang (2021)’s b-l-aug model that is fine-tuned with the same data as ours, when question types are added as prompts to input sequences of correct and incorrect answers in the test set.
We also ran a similar experiment on COPA-test (Table 5) in which adding prompt did not help with performance improvement.
[Table]
Conclusion
We introduced a simple framework for augmenting
PLMs with commonsense knowledge created by automatically verbalizing ATOMIC 2020 .
Our results show that commonsense knowledge-augmented
PLMs outperform the original PLMs on causeeffect pair classification and answering commonsense causal reasoning questions.
As the next step, it would be interesting to see how the previously proposed model improvement methods or using unbiased fine-tuning datasets can potentially enhance the performance of our knowledgeaugmented models.