CIKQA: Learning Commonsense Inference with a Uniﬁed Knowledge-in-the-loop QA Paradigm
Abstract
Recently, the community has achieved substantial progress on many commonsense reasoning benchmarks.
However, it is still unclear what was learned from the training process: the knowledge, how to do inference, or both?
We argue that due to the large scale of commonsense knowledge, it is infeasible to annotate a large enough training set for each task to cover all commonsense for learning.
Thus we should separate the commonsense knowledge acquisition and inference over commonsense knowledge as two separate tasks.
In this work, we focus on investigating models’ commonsense inference capabilities from two perspectives: (1) Whether models can know if the knowledge they have is enough to solve the task; (2) Whether models can learn commonsense inference capabilities, that generalize across commonsense tasks. We ﬁrst align commonsense tasks with relevant knowledge from commonsense knowledge bases and ask humans to annotate whether the knowledge is enough or not. Then, we convert different commonsense tasks into a uniﬁed question answering format to evaluate models’ generalization capabilities.
We name the benchmark as Commonsense Inference with knowledgein-the-loop Question Answering ( CIKQA ).
Introduction
Understanding human language requires both the language knowledge (e g., grammar and semantics) and world knowledge, which can be further divided into factual and commonsense knowledge (Katz and Fodor, 1963).
Recently, the community has made great progress on helping machines acquire and apply language and factual knowledge.
However, how to help machines acquire and inference over commonsense is still unclear.
To answer this question, many commonsense reasoning datasets (Roemmele et al., 2011; Sakaguchi et al., 2019; Talmor et al., 2019; Zellers et al., 2019; Lin et al., 2020) have been proposed.
Even though they target different knowledge types, modalities, and come in different formats, they often follow a standard supervised learning setting, and aim at helping machines to solve a speciﬁc task with the training data.
However, two limitations of this learning paradigm have limited the development of commonsense reasoning systems.
First, there is no clear separation between knowledge and inference.
As discussed in (Elazar et al., 2021), a common phenomenon is that larger training data will lead to better performance, mainly because richer knowledge is covered.
However, due to the large scale of commonsense knowledge, it is infeasible to annotate a large enough training data for each task, and the responsibility of the training data should be teaching models how to do inference rather than acquiring the commonsense knowledge.
Several recent works have explored using structured knowledge for commonsense reasoning tasks (Lin et al., 2019;
Lv et al., 2020; Paul and Frank, 2020).
However, as these works did not clearly analyze the coverage of the structured knowledge (i.e., knowledge graphs), it is still unclear what the performance means, better knowledge coverage or better inference capability.
To dig into what is behind this learning process, we propose to equip each question with supporting knowledge. By doing so, we could evaluate whether models can know if there provided knowledge is sufﬁcient or not and how well they can conduct inference over the provided knowledge to solve the task. Second, the supervised learning may force the model to learn the distribution of the training data rather than a universal inference model. As a result, the model may perform well on the test set that follows the same distribution but fail on other tasks (Kejriwal and Shen, 2020).
Previously, as different tasks have different formats, it is hard to evaluate the generalization ability of commonsense reasoning models.
Motivated by existing trends of using a uniﬁed format (i.e., question answering) for different tasks (Khashabi et al., 2020), we propose to convert various commonsense reasoning tasks into a uniﬁed QA format such that we can easily and fairly evaluate the generalization ability of learned commonsense reasoning models.
Combining these two lines of effort, we propose a new commonsense inference evaluation benchmark Knowledge-based Commonsense Inference with QA ( CIKQA ).
An example is shown in Figure 1.
We equip several popular commonsense reasoning tasks with the supporting knowledge from existing commonsense knowledge graphs and convert them into a uniﬁed QA format.
We leverage human annotation to label whether the provided knowledge is enough to answer the question.
With CIKQA , we are interested in answering three questions: (1) Whether current models can distinguish the knowledge is gold or not; (2) Whether current models can learn to conduct inference over provided knowledge; (3) Can current commonsense inference models generalize across different commonsense reasoning tasks.
Experiments with several recent knowledgebased commonsense reasoning models and a proposed baseline JointI , which jointly encodes the knowledge and question with a single model, show that even though inference over commonsense knowledge is challenging, models can learn to conduct simple inference after training with a few examples and better answer the questions than not using the knowledge.
As a comparison, learning to distinguish gold knowledge is still a more challenging task.
Last but not least, even though current models demonstrate the encouraging generalization ability across three relatively simple tasks, they still cannot learn complex inference (i.e., compare multiple paths) very well.
We hope that our benchmark could motivate more advanced commonsense inference methods in the future.
Related works
To help machines understand commonsense, the community has devoted great efforts to constructing commonsense knowledge bases with either crowdsourcing (e.g., ConceptNet (Liu and Singh, 2004) and ATOMIC (Sap et al., 2019)) or information extraction techniques (e.g., ASER (Zhang et al., 2020)).
Typically, crowd-sourced knowledge bases have higher quality, but the autoconstructed ones have larger coverage.
Besides acquiring the commonsense knowledge, the community also developed many commonsense reasoning datasets to test models’ commonsense reasoning abilities.
Even though these datasets may have different formats (e.g., slot ﬁtting in Winogrande (Sakaguchi et al., 2019) and question answering in CommonsenseQA (Talmor et al., 2019)), knowledge types (e.g., causal commonsense in COPA (Roemmele et al., 2011) and numerical commonsense in NumerSense (Lin et al., 2020)), or modalities (e.g, visual commonsense in
VCR (Zellers et al., 2019) and textual commonsense in many others), they all follow a standard supervised learning setting, and aim at helping machines to solve a speciﬁc commonsense task in an end-to-end manner.
Given this setting, it is often difﬁcult to tell what has been learned during the training process.
Was it used to acquire commonsense knowledge, learn to conduct commonsense inference, or both?
Such ambiguity limits our progress in solving these commonsense reasoning tasks.
In this work, we connect the efforts on commonsense acquisition and inference by creating a commonsense inference benchmark
CIKQA , where the models can focus on learning to do the inference over the supporting commonsense knowledge graph (KG).
Answering questions in natural language based on a knowledge base (KB) has been a mature research topic in the NLP community, which is also known as the KBQA problem (Clark et al., 1999; Yih et al., 2015, 2016; Usbeck et al., 2017;
Cui et al., 2017).
Previous works mainly focus on factual knowledge, which is stored in the format of triplets, and the main challenge is how to parse the question and then precisely and effec- tively identify the correct path over a large-scale
KB to do the inference.
Compared with inference over factual knowledge, inference over commonsense knowledge brings the following unique challenges : (1) Commonsense is a kind of preference rather than ﬁxed knowledge, which typically involves the comparison of several candidates.
As a result, the ideal commonsense reasoning process could involve the comparison of multiple paths; (2) Commonsense is about daily events or objects rather than named entities, and thus it is difﬁcult to ﬁnd an exact node from the commonsense KB that matches the question and we may need to conduct inference based on the partial match (i e., the extracted nodes are relevant but not identical).
Task Formulation
In CIKQA , to encourage a generalizable commonsense inference model, we equip each question with a supporting knowledge graph G . and follow previous works (Khashabi et al., 2020; Cohen et al., 2020; Wu et al., 2020; Du and Cardie, 2020) to unify all selected tasks as a binary question answering problem ( Q , A 1 , A 2 ).
As the autoextracted knowledge could contain noise or may not cover all the essential knowledge for answering the question and humans are capable of saying “I do not know” when they do not know how to answer a question, we leverage human annotators to annotate whether the knowledge is gold (i.e., accurate and enough) for answering the question and test whether current models have the same commonsense reasoning capability of distinguishing the gold knowledge as humans.
Details about task selection, format uniﬁcation, support knowledge extraction, and annotation are as follows.
3. Task Selection and format Uniﬁcation
In CIKQA , we select the following four popular commonsense reasoning tasks:
HardPCR: The hard pronoun coreference resolution (HardPCR) task is one of the most famous commonsense reasoning tasks.
For each question, a target pronoun and two candidate mentions are provided, and the task is to select the correct mention that the pronoun refers to.
Careful expert annotations are conducted to get rid of the inﬂuence of all simple linguistic rules and ask the models to solve the problem with commonsense reasoning.
In CIKQA , we include instances from WSC (Levesque et al., 2012), DPR (Rahman and Ng, 2012), and WinoGrande (Sakaguchi et al., 2020).
To create a question regarding the target pronoun, we ﬁrst ﬁnd the sentence that contains the target pronoun and then determine whether the pronoun refers to a person or an object.
If it is a person, we will ask who participates.
Otherwise, we will ask what participates.
CommonsenseQA (Talmor et al., 2019): CommonsenseQA is a commonsense question answering dataset.
For each question-answer pair, four relevant but wrong concepts are used as the other candidates, and the models are required to select the correct one out of ﬁve candidates.
In CIKQA , we randomly sample a negative answer to make it a binary choice task, which is consistent with other datasets.
COPA (Roemmele et al., 2011): COPA focuses on evaluating whether models can understand the causality between events or not.
For each head event, two candidate tail events are provided, and models are asked to predict the one caused by or the reason for the head event.
ATOMIC (Sap et al., 2019):
The last one is the commonsense knowledge base completion.
Given a head concept (e.g., “eat food”) and a relation (e.g., “cause”), we want to predict the tail concept.
In CIKQA , we focus on predicting edges of ATOMIC.
For COPA and ATOMIC, as they are essentially predicting the relations between two events or states (e.g., “PersonX eats”- Causes -“PersonX is full”), for each edge, we randomly sample another event or state (e.g., “PersonX is hungry”) as the negative tail and ask the model to select the correct one.
To make the task challenging and avoid sampling irrelevant events or states, we require the sampled negative event or state to be connected with the head event or state with a different edge.
For each type of relation, we write a simple pattern to generate the question.
For example, for the “Causes” relation, we will ask “What can be caused by ‘PersonX is hungry’?”
Demonstrations of instances in original datasets and their transformed questions and candidate answers are presented in Appendix Section C.
3. Supporting Knowledge Extraction
As mentioned in Section 1, a limitation of existing commonsense reasoning benchmarks is that there is no clear boundary between knowledge and inference such that we are unclear about what has been learned from the training process, the knowledge, or how to do inference.
To address this issue and encourage models to learn inference rather than knowledge from the training data, we propose to equip each question with supporting knowledge.
Only if we can ﬁnd supporting knowledge for a question will the question be selected to form the dataset.
This section introduces the selected commonsense knowledge graphs and then introduces how we extract the corresponding commonsense knowledge for each question.
3.2. Commonsense KG Selection
Many commonsense knowledge graphs have been developed to enhance machines’ commonsense reasoning abilities.
Several representative ones are
ConceptNet (Liu and Singh, 2004), ATOMIC (Sap et al., 2019), GLUCOSE (Mostafazadeh et al., 2020), and ASER (Zhang et al., 2020).
Among these four, ConceptNet, ATOMIC, and GLUCOSE are constructed via crowd-sourcing while
ASER is constructed automatically with information extraction techniques.
Besides ATOMIC, which is used as one of the tasks, we use the other
KBs as supporting knowledge resources.
3.2. Supporting Graph Extraction
Here we introduce how to extract the supporting knowledge from external commonsense knowledge bases.
For each question, we need to obtain a sub-graph from supporting knowledge graphs such that it contains the relevant commonsense knowledge about the question.
The sub-graph extraction process includes the following three steps: (1) Pre-processing: Convert each question into several key sentences; (2) Matching: Match the sentences into nodes in the KG; (3) Extraction:
Retrieve the supporting sub-graph for the overall knowledge graph.
Data Pre-processing : For each question and the associated candidate answers, we ﬁrst replace the question words (e.g., “What”) with the two candidate answers such that it becomes two declarative sentences.
For instance, if the question is “The ﬁsh ate the worm.
It was hungry.
Who is hungry?”
and the candidates are “Fish” and “Worm,” we will convert the question into the declarative sentence: “The ﬁsh is hungry” and “The worm is hungry.”
As a result, we will get three sentences for this question: “The ﬁsh ate the worm,” “The ﬁsh is hungry,” and “The worm is hungry.”
KG Matching :
After getting the declarative sentences that contain the question and key answers, to extract the relevant knowledge, we map them to nodes in knowledge graphs.
Considering that each sentence may have multiple words and it is often hard to ﬁnd an exact match, we adopt an embedding-based matching technique.
For each sentence and node in the KG, we treat them as a sentence and get the corresponding representations with SimCSE (Gao et al., 2021).
For each input sentence, SimCSE encodes the sentence in a vector.
A close distance between two vectors indicates that the two sentences are similar to each other.
We use cosine similarity on the obtained representations to measure the similarity between two sentences.
Since there are 287 thousand nodes in GLUCOSE and 194 million nodes in
ASER, it is computationally infeasible to compute the cosine similarity between sentences pair by pair.
Thus for each extracted sentence, we ﬁrst apply Faiss (Johnson et al., 2017), a large-scale similarity-based matching algorithm that ﬁrst clusters all KG nodes in the vector space to increase the matching efﬁciency when ﬁnding the top N nodes in the KG.
After that, we sort the N nodes based on the cosine similarity to ﬁnd the top K similar nodes.
In our implementation, we set N and K to be 60 and 1.
On average, it takes 25 seconds to retrieve relevant nodes for each question.
Graph Extraction : In the next step, we construct the sub-graph.
We denote the extracted m nodes as n 1 , n 2 , ..., n m , and for each of them, we ﬁnd
K similar nodes from KG.
The resulting matched node sets are denoted as N 1 , N 2 , ..., N m .
For any pair of eventualities n ∈ N i and n (cid:48) ∈ N j
( i (cid:54) =
j ), if there exist a path in the KG between n and n (cid:48) , we will keep that path.
After merging all paths together, we will get the ﬁnal sub-graph.
On average, it takes less than two seconds to construct a graph for each question.
Knowledge Quality Annotation : We annotate whether the extracted knowledge is accurate and enough.
For each question, we invite ﬁve annotators to provide the annotation.
The average Interannotator agreement (Cohen’s kappa statistic) is 0.83, which indicates the high-quality of our annotation.
In the end, we apply a strict standard (at least four of ﬁve annotators need to vote for gold) [Table] to select the gold knowledge.
More annotation details could be found in Appendix Section A.
3. CIKQA Statistics
We report the dataset statistics in Table 1.
In total, we collect 14,599 instances, and among which Hard PCR and ATOMIC provide the most questions because their original datasets are much larger than others.
According to the annotation, 16.69% of the supporting knowledge graphs are gold knowledge.
Based on our analysis, annotators hold a very strict standard for selecting the gold knowledge.
For each task, we randomly split the dataset into training, development, and testing set with a standard 8:1:1 splitting.
As a result, we get 11,678 training, 1,459 development, and 1, testing instances.
More detailed statistics, and examples of CIKQA are presented in Appendix Section B and C, respectively.
The JointI Model
We introduce a transformer-based commonsense inference model as a strong baseline for CIKQA .
Unlike previous works that acquire question and knowledge representations separately, we propose to combine them ﬁrst and then acquire the representation jointly.
As a result, we name our method as Joint Inference ( JointI ).
As shown in
Figure 2, given a question Q , two answers Aand A 2 , and a supporting knowledge graph G
=
( h 1 , r 1 , t 1 , w 1 ) , ..., ( h n , r n , t n , w n ) , where h , r , t , w indicates the head, relation, tail, weight respectively, and n is the number of edges, our goal is predict which answer is the correct one.
Here, all questions, answers, heads and tails in the KG are list of tokens.
JointI consists of two main components (i.e., knowledge sampling and joint inference).
Details are as follows.
Knowledge Sampling:
As current language models require the input to be in a sequence format rather than a graph, we ﬁrst conduct a weighted random walk over G to convert it into several knowledge paths P that are in the format of sequence.
During our sampling, the weight of an edge determines the possibility of it being sampled.
As a result, an edge with a larger weight is more likely to appear in the sampled path and has a bigger impact on the prediction.
Another point worth mentioning is that, following previous work (Lv et al., 2020), we convert all the relations into natural language according to the relation template (e.g., “IsA” to “is a”).
As shown in
Figure 2, each P ∈ P can be viewed as a long sentence, where nodes in G are connected with connectives.
An example is “I sleep because I am tired so I rest on a bench...”.
Joint
Inference:
The key difference between
JointI and previous works is that we jointly acquire the representation of the knowledge, question, and answer rather than acquiring them separately and then combining.
Many previous works have demonstrated the superiority of such an approach on other NLP tasks (Huang et al.
; Sakaguchi et al., 2020).
Speciﬁcally, if we want to predict the plausibility score for A given Q , for each knowledge path P , we ﬁrst concatenate it with the question Q and candidate answer A :
[Equation]
where [ · ] indicates the concatenation.
We follow previous works to insert a special token between
P and Q and Q and A .
Once obtaining a concatenated input of P , Q and A , we encode it using a transformer module T rans and get a prediction score with a multi-layer perceptron module MLP for a particular question and answer:
[Equation]
After that, we will get the ﬁnal prediction with the average of all sampled paths:
[Equation]
In the end, the candidate answer with a higher score will be predicted.
Since the task is formulated as a binary classiﬁcation problem, we adopt the cross-entropy loss and optimize the model with
Adam (Kingma and Ba, 2015).
Experiments
In this section, we present the performance of current commonsense inference models on CIKQA .
Besides JointI , we also show the performance of the following baseline methods:
(1) Vanilla LM :
We use the language model (LM) based multiple-choice (MC) model as the basic baseline.
For each candidate answer, we follow the standard ﬁnetuning procedure to concatenate it with the question and then feed it to a pre-trained language model.
After getting the sentence representation, a linear layer is used to obtain a score and trained with a cross-entropy loss.
(2) KagNet :
As one of the pioneering works that utilized structured knowledge for solving commonsense reasoning tasks, KagNet (Lin et al., 2019) ﬁrst uses a graph convolution network to encode the knowledge graph and then apply an LSTM based hierarchical attention mechanism to encode the knowledge path that starts with the concepts corresponding to the question and end with concepts corresponding to the answer.
At the same time, KagNet encodes the question and answers with pre-trained LMs.
In the end, it concatenates all representations for the ﬁnal prediction.
(3) Graph Based Reasoning (GBR) : Instead of only encoding paths starting with the question concepts and ending with answer concepts, the follow-up work GBR (Lv et al., 2020) proposes to conduct a depth-ﬁrst algorithm over the knowledge graph to generate a sequence of paths as the supporting knowledge paths.
(4) Multi-Head Knowledge Attention (MHKA) :
To further utilize the knowledge, MHKA (Paul and Frank, 2020) uses a transformer network to model the paths from the question concepts and answer concepts, then concatenates the knowledge and context representation for the ﬁnal prediction.
We implement all experiments with Huggingface (Wolf et al., 2019).
We select BERT-base (Devlin et al., 2019) as the base language model for all models.
The batch size is set to be 16.
All models are trained for 10,000 steps 2 , and the best-performing checkpoints on the dev set are evaluated.
For our model, we set both the number of random walk paths and walk length to be ﬁve.
Considering that the auto-extracted knowledge could contain noise or miss certain knowledge, we add a “gold knowledge” setting, where only examples with the gold knowledge are used for training and testing, for all models as the upper bound of their model.
All other hyper-parameters are the same as the base language model.
All models are trained with GTX 2080 and the average running time is 12 hours.
5. Results
For each model, we train it with different numbers of training instances and report the average performance and standard deviation 3 of ﬁve trails in Figure 3, from which we can observe that with the help of knowledge, all inference models outperform the baseline model without knowledge, especially JointI .
When the auto-extracted knowledge and gold knowledge are provided, JointI outperforms the baseline Vanilla LM model by 4. and 15.34, respectively.
It supports our assumption that it is hard to learn all knowledge from the limited training data and external structured knowledge could help.
Moreover, we also notice that when the knowledge is provided, JointI could learn to answer the questions with only a small number of examples.
This suggests that if we only want to learn to do the inference over commonsense, we may only need a few training examples.
Besides that, the comparison between autoextracted knowledge and gold knowledge also shows that current commonsense knowledge base construction and retrieval methods are still not optimal and we may need to devote more effort to these two directions in the future.
Last but not least, we can see that JointI outperforms other inference models among most settings, which shows that jointly encoding question and knowledge is not just more efﬁcient but also a more effective strategy than acquiring them separately, and could serve as a stronger baseline for future works.
Due to the simplicity and efﬁciency of JointI , we will conduct the rest analysis experiments with JointI .
5. Distinguishing the Gold Knowledge
Humans have the capability of saying “I do not know” when they ﬁnd out that they cannot answer a question with their knowledge.
To investigate whether current deep models have a similar capability, we use JointI as an example to test whether these deep models can distinguish the gold knowledge.
For each (question, answer, and knowledge) triplet, we train and test JointI with annotated knowledge quality label.
To address the imbalanced distribution problem, we randomly select the same number of “Not Gold” examples as the “Gold” ones to make the dataset balanced.
From the results in Figure 4, we can see that the performance of JointI could be improved slightly with the increase of training data.
However, after seeing thousands of examples, it still can only achieve 0.65 accuracy on a binary classiﬁcation problem.
It shows that knowing when to say “I do not know” is still a challenging task for current deep models.
Generalization Ability
An important assumption and motivation behind
CIKQA is that even though the commonsense could be enormous, the inference rules over commonsense knowledge should be limited.
As a result, even though we could not learn all the commonsense from limited training data, we can learn how to conduct inference with several tasks and then generalize to others.
In this section, we conduct experiments with both the “Without Knowledge” and “With Knowledge” models to show that with our uniﬁed formulation, we can gain such generalization ability across different tasks.
To clearly show the effect of the supporting common- [Table] [Table] sense KB, we conduct experiments on two settings: (1) Gold Subset: We only train and test the model on questions, where the supporting graph is annotated as gold; (2) Full Set: We train and test the model with the whole dataset.
We train the model with questions from a speciﬁc task and test it on all tasks.
The results are presented in Table 2.
From the results, we can see that the knowledge can help models to generalize well among CommonsenseQA, COPA, and ATOMIC.
The only exception is HardPCR.
This is mainly because the inference needed for solving HardPCR is more complex than the other tasks, where we do not only need to ﬁnd the relevant knowledge but also need to replace the target pronoun with the entity in the provided knowledge.
How to train a model that can learn to conduct such complex reasoning is a problem worth exploring in the future.
In general, the observed generalization ability is encouraging because if we can learn a good model on CIKQA , based on the assumption that there exists limited types of inference, potentially we can solve any commonsense reasoning tasks as long as the needed inference type is covered by CIKQA .
At the same time, we also notice that current models still cannot learn complex inference (i.e., compare multiple paths) with few examples, and we leave how to solve that problem as the future work.
Conclusion
In this paper, we present CIKQA , a uniﬁed commonsense inference benchmark.
Speciﬁcally, we ﬁrst convert several popular commonsense tasks into a uniﬁed QA format and equip each question with a supporting commonsense knowledge graph.
During the training on CIKQA , models do not need to worry about the commonsense knowledge and can thus focus on learning to do the inference.
Experiments show that models can better learn how to do commonsense inference with a few examples and signiﬁcantly outperform the baseline method that does not use structured knowledge in the data-scarce setting.
More interestingly, with our uniﬁed formulation, models demonstrate the encouraging generalization ability across tasks.
As both the format uniﬁcation and supporting graph extraction are automatic, we can easily extend to other commonsense reasoning tasks in the future.
All used code and data are submitted as part of the appendix.
Acknowledgements
The authors of this paper were supported by the Ofﬁce of the Director of National Intelligence (ODNI), Intelligence Advanced Research
Projects Activity (IARPA), via IARPA Contract
No. 2019-19051600006 under the BETTER Program, and by contract FA8750-19-2-1004 with the US Defense Advanced Research Projects Agency (DARPA). The views expressed are those of the authors and do not reﬂect the ofﬁcial policy or position of the Department of Defense or the U.S. Government.
This paper was also supported by the NSFC Fund (U20B2053) from the NSFC of
China, the RIF (R6020-19 and R6021-20) and the
GRF (16211520) from RGC of Hong Kong, the
MHKJFS (MHP/001/19) from ITC of Hong Kong with special thanks to HKMAAC and CUSBLT,
B Statistics
We report the number of questions that a supported graph can be ﬁnd, the average size of supporting graph, and the number of helpful instances of
CIKQA in Table 4.
In total, we collect 14,599 instances with the average supported graph size of 2.75.
C Case Study
Demonstration of how we convert the original dataset into the uniﬁed format is presented in Table 3.
For each task, we use a template to automatically convert it into the uniﬁed QA format.
Besides that, we also present several questions along with knowledge in Figure 6.
From the example we can see that, the reasoning over HardPCR is more challenging than other tasks.
In the HardPCR example, two paths can be found relevant to question: (1) “I am drunk” → Co Occurrence → “I hit someone”; (2) “I am drunk” → Co Occurrence → “That is not fair” → Co Occurrence → “You kick me”.
For the correct inference, we need to know when there is a conﬂict, we should trust the one-hop inference more because the additional node in the two-hop path may introduce extra noise.
As a comparison, for other tasks, the main inference we need is to ﬁnd the relevant paths, which is relatively easy.
D Detailed Experimental Results
Detailed experimental results are presented in Table 5.
[Table] [Table]
Other eventualities are white.
Edge weights are in brackets.
We only show the relevant part of the graph for the clear representation.
All extracted eventualities are lemmatized, we recover them for the ease of understanding.
[Table]