Identifying relevant common sense information in knowledge graphs (2022)

Abstract: Knowledge graphs are often used to store common sense information that is useful for various tasks. However, the extraction of contextually-relevant knowledge is an unsolved problem, and current approaches are relatively simple. Here we introduce a triple selection method based on a ranking model and find that it improves question answering accuracy over existing methods. We additionally investigate methods to ensure that extracted triples form a connected graph. Graph connectivity is important for model interpretability, as paths are frequently used as explanations for the reasoning that connects question and answer.

## introduction for models to be able to reason about situations that arise in everyday life, they must have access to contextually appropriate common sense information. this information is commonly stored as a large set of facts from which the model must identify a relevant subset. one approach to structuring these facts is as a knowledge graph. here, nodes represent high-level concepts, and typed edges represent different kinds of relationship between concepts. in practice, a subset of facts that are thought to be contextually relevant are extracted from the graph, as using all facts in each instance is unnecessary, noisy, and computationally expensive. prior work has focused on different ways to encode these facts, including by inputting them into a graph neural network (gnn) or into a transformer @xcite . however, the question of how to identify useful information has been under-explored, particularly in work that uses gnn encoders. if contextually important information is not retrieved then performance could be dramatically reduced, a potential result of the use of overly simplistic retrieval methods. in this paper we explore methods to extract highquality subgraphs containing contextually relevant information. 1 we approach this as a ranking task across triples in a knowledge graph, and propose two methods that use the scores to extract a subgraph. the first is a weighted pathfinding approach which extends prior work @xcite , while the second builds a minimum spanning tree that includes the highest-ranked triples (figure 1 ). both approaches ensure that all or most nodes in the subgraph are reachable from each other, which is important for two reasons. first, it means that the gnn can update node embeddings with information from most other nodes, which would not be possible if the graph were disconnected. second, it allows paths of reasoning to be extracted from the subgraph, which are often used as explanations for model behaviour @xcite @xcite . there are also situations when specific concepts need to be included in order for a subgraph to be of high enough quality. for example, in question answering, a full explanation must include one or more concepts mentioned both in the question and in a candidate answer. this requires robustness towards how concepts identified because the knowledge repository might express the concept in a slightly different lexical form from the question and/or answer. we therefore experiment with a embedding-based method to identify these concepts, and compare it with existing lexical methods. our contributions are as follows foot_2 : many prior approaches to retrieving relevant common sense triples from a knowledge graph start by identifying relevant nodes. simple lexical overlap between a concept and the context (e.g. question text) is often used for this @xcite . however, this entity linking approach is likely to only retrieve simple concepts, as the idiosyncratic phrasing of some node names in knowledge graphs like conceptnet @xcite are unlikely to show up in text. @xcite investigate this in detail and propose a series of pre-processing steps that allow lexically-based linking without exact phrase matches. for the same reason, the heuristics used by @xcite for lexical matching are employed by a series of later works @xcite @xcite . although lexical matching is a frequent approach with common sense knowledge graphs, in other domains embedding-based approaches are more popular @xcite . these work by embedding the candidate text and finding the nearest neighbour in the space of entity embeddings. in question answering, lin et al. ( 2019 ) split these concepts into those identified in the question and in the answer, and find additional concepts for the relevant subgraph by iteratively finding shortest paths between the two sets. this process continues until a maximum number is collected, or the path lengths exceed a threshold. the final subgraph used as input to models is constructed from this set with all valid edges added. some approaches score nodes and triples that have been identified. @xcite score multiple paths for each question and answer and choose the answer with the highest mean path score. @xcite extract a subgraph following @xcite , and additionally score each node for relevance to a question using roberta @xcite . ranking is also common with prose facts, particularly when they are input into transformer-based models that have limits on input size @xcite . ## methodology in this section we introduce our methods for extracting a contextually-relevant subgraph g for a question answering task. the graph should contain triples that are useful in distinguishing the correct answer from a set of distractors. for each instance, we represent the question text as q and the ith candidate answer as a i , and the set of concepts extracted from each as c q and c a i respectively. ## results our results on openbookqa are presented in the tuned baseline for openbookqa beats the proposed methods in all cases, although there is reasonable variation in accuracy between the baselines of different sizes (see ## analysis we observe that, in the majority of cases, using methods to increase connectivity within the extracted subgraph improves performance over simply including the top rated facts. the minimum spanning tree (mst) approach has the advantage of including these facts, unlike the weighted path method which may not. however, to ensure that the graph is connected the mst approach may have to include nodes and edges that are less relevant to the context. one might expect a weighted approach to counterbalance this, however this also results in a larger subgraph being constructed which may be detrimental (see appendix b). indeed, with lexical grounding the weighted approach adds an average of 37 nodes and 83 edges to the extracted subgraph, compared with 26 nodes and 71 edges in the unweighted case. the weighted pathfinding approach has the advantage of avoiding edges which are not relevant to the query. additionally, the subgraph is extracted in way that is closer to c q and c a i than the mst approach, which considers these nodes only after selecting the top-ranked triples. as a result, the question and answer nodes are connected in a larger variety of ways, which may help increase performance. for openbookqa, the increase in score between lexical and embedding-based entity linking with an unweighted mst suggests that the concepts identified by the latter method are particularly useful. the same magnitude of increase is not seen in com-monsenseqa. one possible reason for this is that commonsenseqa was constructed directly using conceptnet, which may increase the relevance of concepts obtained with lexical methods. similarly to with lexical grounding, the weighted mst with embedding grounding adds more nodes and edges on average (153 nodes, 217 edges) than the unweighted one (112 nodes, 172 edges). in both cases, the resulting subgraph is substantially larger than the equivalent ones built from lexicallylinked entities. this is likely due to the kinds of nodes identified by entity linking -we observe that concepts identified by the embedding-based method are more specific, and so are less connected within the overall graph. conversely, concepts that are identified lexically are likely to be simpler and more general, and so better connected within the graph, meaning fewer additional nodes and edges are required to build the mst. ## conclusion we present a method for extracting relevant information from a common sense knowledge graph, casting it as a ranking problem. we show that scores obtained from a ranking model can be used to select triples containing useful information for a question answering task, improving performance over a commonly-used approach. as it is undesirable for extracted subgraphs to have low connectivity, particularly when using paths within them for model interpretation, we use an algorithm for calculating minimum spanning trees over a supplied set of nodes and edges to ensure the graph is connected. we find that this helps performance; in particular, the models with highest accuracy on commonsenseqa use a weighted version of this. we additionally find that using an entity linking approach that uses embeddings rather than lexical matching improves performance in some cases. we distribute the contextually-relevant subgraphs to facilitate future work; these drop in to existing models with no further processing required. future work might investigate the influence of the fact ranker, as our results suggest that it can transfer from the science to general common sense domain successfully. further training of the ranker using higher-quality negative samples from e-qasc @xcite ) may yield better performance, as noted by @xcite .