Open-Domain Contextual Link Prediction and its Complementarity with Entailment Graphs (2021)

Abstract: An open-domain knowledge graph (KG) has entities as nodes and natural language relations as edges, and is constructed by extracting (subject, relation, object) triples from text. The task of open-domain link prediction is to infer missing relations in the KG. Previous work has used standard link prediction for the task. Since triples are extracted from text, we can ground them in the larger textual context in which they were originally found. However, standard link prediction methods only rely on the KG structure and ignore the textual context that each triple was extracted from. In this paper, we introduce the new task of open-domain contextual link prediction which has access to both the textual context and the KG structure to perform link prediction. We build a dataset for the task and propose a model for it. Our experiments show that context is crucial in predicting missing relations. We also demonstrate the utility of contextual link prediction in discovering context-independent entailments between relations, in the form of entailment graphs (EG), in which the nodes are the relations. The reverse holds too: context-independent EGs assist in predicting relations in context.

## introduction a knowledge graph (kg) is constituted by a set of (subject, relation, object) triples such as (apple, acquire, beats). kgs have entities (subjects and objects) as nodes and relations as labeled edges. manually-built kgs such as freebase @xcite , wikidata @xcite , or dbpedia @xcite have a known set of hand-built relations. in contrast, the relation-labels of open-domain kgs are obtained from text rather than fixed. open-domain kgs can be constructed by applying parsers or openinformation extraction methods to text @xcite . the contextual link prediction task predicts relations that hold between the entitypair in a grounded triple. for example, we predict that the relation own should be added between apple and beats. c) an example eg of type organization, organization. the contextual link prediction and eg learning tasks are complementary. for example, @xmath0. open-domain link prediction is the task of adding relation edges that are missing from the graph because the corresponding triple was not found in the text @xcite . figure 1a shows part of an example open-domain kg, in which the triple (apple, own, beats) is missing, but can be inferred using link prediction over all entities in the complete kg. previous work has applied standard link prediction methods such as transe @xcite , conve @xcite , or tucker @xcite to open-domain triples. these methods have been shown to be effective in learning the kg structure, but they are sub-optimal for open-domain link prediction because they ignore the textual context of the triples. since the triples are extracted from text, they can be automatically grounded back to their contexts. hence, in addition to the kg structure, the triple contexts can be used as input to the link prediction task. figure 1b shows the context sentences that have given rise to the partial kg in figure 1a . 1 there are multiple clues in the contexts such as deal, $, cash, stock, and financial times, that could be used in addition to the triples in the rest of the kg, to predict that the triple (apple, owns, beats) should be added. this is because these clues could have been seen around occurrences of other entity-pairs of the same type (e.g., facebook and whatsapp) that are connected by acquire, 's purchase of, and own relations. in this paper, we propose the new task of contextual link prediction for such open-domain graphs: given a triple (e 1 , r, e 2 ) grounded in context with the relation r holding between the entities e 1 and e 2 , our goal is to predict all the other relations that hold between the two entities. we present a model that uses contextualized relation embeddings to predict new relations. we start with bert @xcite pre-trained embeddings and fine-tune them with a novel unsupervised contextual link prediction objective function. after training the contextual link prediction model, we can add missing relations to the kg (e.g., own in in figure 1a ) by predicting the relations that hold between the entities of triple mentions in context (e.g., the context c 1 in figure 1b ). our experiments show that the proposed model for the contextual link prediction task significantly outperforms standard link prediction in open-domain kg completion. in addition, we investigate the interplay between contextual link prediction and context-independent entailments between relations, in the form of entailment graphs (eg). an eg has typed relations as nodes and entailment relation as directed edges @xcite @xcite @xcite . the type of each relation is determined by the types of its two entities. egs are by definition context-independent, but they use relation types as a proxy of the context. figure 1c shows a fragment of an eg showing that for example acquire entails own. similar to opendomain kgs, egs are constructed based on extracted triples from text. the entailment between two relations is predicted by computing a directional entailment score between them. it has been recently shown that the two tasks of open-domain link prediction and eg learning are complementary @xcite . egs suffer from sparsity since many correct entailment relations are not directly supported by the extracted triples from the text. the egs can be improved by augmenting the extractions with novel triples from standard link prediction models. conversely, explicit entailments from egs are shown to be useful in predicting missing links in the kg. we show a similar relationship between contextual link prediction and the eg learning tasks. as in the previous work, we augment the set of extracted triples with novel predictions, but we use contextual link prediction instead of standard link prediction. we define a new entailment score which we use to build new state-of-the-art egs when tested on a challenging relation entailment dataset. our results show that contextual link prediction produces higher quality triples for augmentation than standard link prediction. conversely, we also show that egs in turn contain complementary information that can be combined with contextual link prediction to further improve the open-domain kg completion results. 2 our main contributions are the following. mckenna et al. ( 2021 ) extend egs to include entailments of different valencies, e.g., a binary relation entailing a unary one. we only consider binary relations, but the method can be extended to multiple valencies in the future. @xcite use temporal information for entailment graph learning by constraining the context of each relation to entity-pairs observed in a temporal window around it. our method could be adapted similarly. schmitt and schütze (2021) have proposed a supervised model that fine-tunes pre-trained lms directly on a training portion of entailment datasets. they report better results than egs, but our focus is different. unlike their method, our approach is unsupervised and is not capable of learning potential artifacts from datasets @xcite . in addition, we explicitly build egs by doing machine-reading over large text corpora, and hence can explain the basis for the beliefs captured in them. pre-trained lms for link prediction. kg-bert @xcite uses contextual representations for kg completion. however, they form synthetic token sequences by concatenating entity descriptions and relation tokens, whereas we use the natural text associated with the triples. extracting factual knowledge from pre-trained language models. these works form a prompt where an entity is missing (e.g., apple acquire [mask] ), and ask the language models to predict the masked entity @xcite @xcite @xcite . these models do not probe for relations because a) they face technical challenges in processing multi-token relations; and b) relations can be expressed in many different ways. the matching-the-blank (mtb) model @xcite learns relation embeddings by encouraging relations that share the same entity-pairs to have similar embeddings. this is similar to our training, but has two main differences: first, our contextual link prediction model outputs a directional score between relations in context (e.g., acquire in figure 1 ) and hypothesis relations (e.g., own), while mtb learns a symmetric similarity score. second, we can predict a score for any hypothesis relation as long as the relation is previously observed somewhere in the corpus with any other entities ( §3.2). ## contextual link prediction in this section, we first discuss the notation and define the contextual link prediction task. we then present our model and training for the task. ## scoring entailment between relations in this section, we describe our new entailment score. we augment the set of input triples with novel triples from the contextual link prediction model. we compute entailment scores 0 ≤ w @xmath1. we propose an entailment score, named cnce markov chain (mc), similar to the conve mc score of @xcite , but based on the contextual link prediction score, as follows. we form a bipartite graph with relations on one side and triple mentions on the other side (figure 3 ). 7 we define the entailment score as the probability that a random walk (with length 2) from one relation ends in another. in particular, we define a markov chain with relation states r as well as triple mention states m as its nodes. each relation r has directed edges to its triple mentions @xmath2, where d(r) is defined as the set of all triple mentions of r. on the other hand, each mention @xmath3, r, e 2 , c, s) has directed edges to a set of relations r(m) = r(e 1 , e 2 ) ∪ u(m), where r(e 1 , e 2 ) is the set of observed relations for the entity-pair and u(m) ⊆ r(t 1 , t 2 ) \ r(e 1 , e 2 ) contains a set of relations with high contextual link prediction scores. for a relation @xmath4, the triple (e 1 , u, e 2 ) is unobserved in the text corpus, but is likely to be correct (e.g., pose with in previous work forms a bipartite graph with relations on one side and entity-pairs on the other side. where pr q|m is defined in eq 1. we define the cnce mc entailment score as: in order to form the markov chain (figure 3 ), we first connect each triple mention m to its observed relations r(e 1 , e 2 ). if the number of observed relations |r(e 1 , e 2 )| is less than @xmath5, we augment the markov chain by connecting the mentions to k-|r(e 1 , e 2 )| highest scoring relations (i.e., u(m)) corresponding to unobserved triples. we then use entailment scores as the local entailment scores and apply the global soft constraints of @xcite to learn global entailment scores. we build egs, whether local or global, by applying a threshold on the entailment scores,. for a threshold δ > 0, we build egs where nodes are relations @xmath6, and edges include (r, q) with w @xmath7. 9 in our experiments, we slide the threshold in the range [0, 1] to build and evaluate egs with varying degrees of confidence. ## experimental setup we discuss the details of the corpus and training. ## results and discussion we first evaluate our proposed method for the opendomain contextual link prediction task ( §6.1). we then investigate the complementarity of contextual link prediction and egs ( §6.2 and §6.3). ## conclusions we have introduced the contextual link prediction problem and proposed a model (cnce) for it. we trained cnce on a corpus of triple mentions. @xcite we tuned β = 0.05 using the newsspike dev set.