On Masked Language Models for Contextual Link Prediction (2022)

Abstract: In the real world, many relational facts require context; for instance, a politician holds a given elected position only for a particular timespan. This context (the timespan) is typically ignored in knowledge graph link prediction tasks, or is leveraged by models designed specifically to make use of it (i.e. n-ary link prediction models). Here, we show that the task of n-ary link prediction is easily performed using language models, applied with a basic method for constructing cloze-style query sentences. We introduce a pre-training methodology based around an auxiliary entity-linked corpus that outperforms other popular pre-trained models like BERT, even with a smaller model. This methodology also enables n-ary link prediction without access to any n-ary training set, which can be invaluable in circumstances where expensive and time-consuming curation of n-ary knowledge graphs is not feasible. We achieve state-of-the-art performance on the primary n-ary link prediction dataset WD50K and on WikiPeople facts that include literals - typically ignored by knowledge graph embedding methods.

## introduction large-scale knowledge graphs (kgs) have gained prominence over the past several decades as a means for representing complex structured data at scale, leading to the development of machine learning models designed to predict new or unknown information from a kg @xcite . a subclass of such models deals with link prediction, i.e. inferring new facts from a given kg consisting of (subject, relation, object) triples. for instance, a link prediction model might reason from a kg containing the triple (usa, electedpresident, jfk) to infer that the triple (jfk, bornincountry, usa) also likely exists (i.e. jfk was born in the country usa). the triple format is often too restrictive to represent a query effectively. for instance, the query ) is augmented with an auxiliary link for qualifer information @xcite . each entity or relationship is represented by a unique identifier. qualifiers require the use of specialised encoder architectures; literal qualifiers like 1960 typically cannot be used at all. (b) we instead represent the query in a templated language model, where the qualifier detail can be directly appended. who was elected president of the united states in 1960? permits multiple correct answers when simplified to the triple format (usa, electedpresident, @xcite ), in the absence of the context 1960 (also referred to as a qualifier @xcite ). recently, several kg completion models have been developed aimed specifically at link prediction in the presence of qualifiers, collectively referred to as hyper-relational or n-ary link prediction models @xcite @xcite @xcite @xcite . usage of qualifiers becomes particularly difficult when they include literals, i.e. values that cannot be efficiently represented as discrete graph entities. examples of literals include years (like 1960), times, or numerals. existing kg completion algorithms typically remove literals @xcite or use specialised techniques to leverage them @xcite . the need for new models to leverage qualifiers and literals reveals some fundamental weaknesses in discrete, triple-based knowledge graph represen-tations. unlike graphs, written languages clearly permit the use of qualifiers and literals to represent facts and queries. pre-trained language models like bert @xcite have already shown competitive performance compared to existing kg link prediction approaches on triple-based kgs @xcite . as such, it is natural to ask whether language models (lms) present a better alternative for inferring facts with qualifiers and literals compared to n-ary kg inference models. apart from their ability to represent qualifiers and literals, using lms with novel pre-training methodologies on vast corpora also presents opportunities to enable n-ary link prediction without access to any n-ary training set. the need to construct large, partially complete n-ary knowledge graphs in new domains is an expensive and timeconsuming requirement of link prediction @xcite . here, we present hyper relational link prediction using an auxiliary entity linked corpus (hyper-elc), the first fully natural-language-based approach applied to kg link prediction benchmarks containing qualifiers and literals. we make use of model pre-training to leverage the large corpora directly available to language models, applying a simple entity-linking approach to prime the model for later inference on named kg entities and to enable link prediction without access to any nary training set. to our knowledge, this is the first approach to link prediction without kg supervision. we also use fine-tuning to specifically focus hyper-elc on the types of queries represented in the training set. by using kg link prediction datasets, we can directly compare language models to kg models specifically designed to take advantage of additional context in form of qualifiers and literals. our results show competitive performance compared to these link prediction models, suggesting that language models provide a performant and practical alternative to kg models for link prediction beyond triple-based datasets. ## definitions a hyper-relational (n-ary) graph, made up of hyperrelational facts, can be defined as @xmath0, r, e), where v is the set of vertices (entities), r is the set of relations, and e is a set (e 1 , . . . , e n ) of edges with e @xmath1. here, p denotes the power set. a hyper-relational fact e @xmath2, r, o, q), with s, @xmath3. here, q is the set of qualifier pairs (qr i , qv i ) with qualifier relations qr @xmath4. an example of a fact in this representation would be (stephenhawking (s), awardreceived (r), edding-tonmedal (o), (pointintime (qr 1 ), 1975 (qv 1 ))). ## methods our approach consists of three stages: ## experiments throughout this section we compare to the following external baselines developed for n-ary link prediction: (i) nalp-fix @xcite , (ii) hinge @xcite , (iii) stare @xcite , (iv) hy-transformer @xcite , and (v) gran @xcite . nalp-fix is an improved version of the original nalp model @xcite . none of these methods make predictions over natural language and none of them encode literals. the metrics that we use are based on predicting only the subject and object of the hyper-relational facts. we follow the filtered setting introduced by @xcite as discussed in section 5.2 to ensure that corrupted facts are not valid facts from the rest of the dataset. for each test example, we filter from the model's predicted ranking all of the entities that appear in the same position in otherwise identical examples in either the training, validation or test set (except the test entity of interest). we consider mean reciprocal rank (mrr) and hits at 1 and 10 (h@1 and h@10 respectively). ## discussion and future work here, we presented hyper-elc, the first purely natural language-based approach to n-ary link prediction and the first model to leverage literals in n-ary qualifiers. the natural language-based approach allows us to take advantage of pre-training on massive entity-linked corpora and easily leverage the detail present in hyper-relational facts. hyper-elc matched state-of-the-art performance on wd50k and established state-of-the-art on a version of wikipeople containing only literal qualifiers. however, it did not reach the performance of existing kg models on the full wikipeople dataset. as shown in in addition, we expect performance improvements by increasing coverage of relevant information for the entities of interest in the pre-training dataset. the wd50k and wikipeople pre-training datasets only have 88.2% and 85.3% coverage of the wd50k and wikipeoople entities, respectively. this could be achieved by improving the quality of the entity linking methodology used. simple improvements could be made to our regex method, such as including the wikidata surface forms in the regex dictionaries. even greater improvements could likely be made with feature based or neural entity linking methodologies. finally, we found that hy-transformer had the best performance on wd50k (100), though hyper-elc performed similarly to or better than the other kg baselines. @xcite propose that hy-transformer's auxiliary masked qualifier prediction task allows it to better leverage the train set, which could explain why hy-transformer performs well on the smaller train set in wd50k (100). a similar qualifier prediction task could also be investigated in the context of a language model, which we leave for future work. overall, our results show how a language model can leverage weakly relevant data (an entity-linked corpus) to reach strong performance on a complex link prediction task. in particular, we note that many practical relational inference problems do not exist in isolated domains where only a structured kg model is available; rather, they are loosely informed by massive, readily available unstructured natural language datasets. in these cases, the sheer quantity and variety of data available to language models, combined with their inherent flexibility in representing context, may swing the balance in their favour. 7 : 37 of the roles/relationships that occur most frequently in the wd50k (100) train dataset were considered and some were altered to make templates more similar to natural language -for example improving grammar.