Integrating Lexical Information into Entity Neighbourhood Representations for Relation Prediction (2021)

Abstract: Relation prediction informed from a combination of text corpora and curated knowledge bases, combining knowledge graph completion with relation extraction, is a relatively little studied task. A system that can perform this task has the ability to extend an arbitrary set of relational database tables with information extracted from a document corpus. OpenKi[1] addresses this task through extraction of named entities and predicates via OpenIE tools then learning relation embeddings from the resulting entity-relation graph for relation prediction, outperforming previous approaches. We present an extension of OpenKi that incorporates embeddings of text-based representations of the entities and the relations. We demonstrate that this results in a substantial performance increase over a system without this information.

## introduction curated knowledge repositories such as knowledge bases and relational databases provide powerful tools for many practical knowledge related tasks. they require, however, substantial effort to create and maintain. many applications deal with knowledge that is continuously changing, presenting prohibitive maintenance costs and limiting the utility of explicit knowledge representation technologies. the new knowledge is often available in text based formats such as reports, news items and memos. in this work, we use the term "proposition" to describe a triple (e 1 , r, e 2 ) that indicates that a relation r holds between two entities e 1 and e 2 . work in the field has largely focussed on either extracting propositions directly from text or inferring missing propositions by examining knowledge graphs. what we are interested in here combines the two in a single model, utilising information from the knowledge base and collections of text together to infer relations, both mentioned in the text and implied by the text in combination with existing knowledge. previous work following this approach draws on patterns in the curated knowledge graph in combination with the graph of entity mentions in texts, allowing prediction of new knowledge base relations @xcite @xcite . @xcite extend this work by incorporating text predicates connecting entity mentions extracted using openie tools @xcite and introducing the concept of "entity neighbourhoods" consisting of the binary openie predicates and knowledge base relations foot_0 that occur with a given entity as their subject or object. drawing on the success of text based representations incorporated into entity recognition tasks @xcite , we extend zhang et.al.'s model by incorporating text based embeddings of entities and relations into the entity neighbourhood representations. texts are drawn from knowledge base metadata and occurrences in source texts. we use fasttext @xcite word embeddings and bert @xcite to obtain text embeddings. the resulting models achieve state of the art results on two knowledge base extension data sets. ## related work open information extraction (openie) attempts to find relations expressed in collections of texts through identification of entity and relation spans @xcite . our work can be taken as an approach to incorporate this extracted information into an existing knowledge base. relation extraction, the identification of relations expressed in text between given entity mentions, has received much attention in recent years (e.g.: @xcite @xcite foot_1 ) including the creation of many annotated data sets (e.g.: @xcite @xcite ). these tasks consider only the recognition of knowledge directly expressed in individual texts, whereas we seek to utilise the combined knowledge from both a collection of texts and a knowledge base, allowing implicit and automatic association between expressions in texts and knowledge base relations and inference of propositions not directly expressed in individual texts. a number of works present a distant supervision approach that utilises entity pairs in texts as a signal for the presence of propositions that may be incorporated in a knowledge base. this signal is inherently noisy, and several approaches have been devised do deal with this (e.g.: @xcite @xcite ). closer to what we propose, @xcite propose a neural attention mechanism between a knowlege graph and supporting texts, outperforming previous approaches. these approaches do not utilise graph information in the form of connections between the texts and can only extract relations explicitly mentioned in the texts. we note that the openki model @xcite , which we use as a baseline, outperforms these models (see ## enhanced entity neighbourhood model we build on the entity neighbourhood encoding (ene) model proposed by @xcite . we then combine our enhanced neighbourhood encodings with the more complex "dual attention" model coined as "openki". input data consists of a knowledge base or "kb" (a curated collection of proposition triples) and a collection of texts with entities identified and linked to knowledge base entities (where possible). in addition, text predicates linking entity mentions in source texts may be extracted (for example with openie tools such as reverb @xcite or ceres @xcite ). alternatively, sentences can be used as proxies for text predicates. the task then is to decide whether a query proposition (e 1 , r, e 2 ) with kb relation r is true and should be added to the kb. a graph of the propositions drawn from both the knowledge base and source texts is constructed. here the entities are nodes and kb relations and text predicates are directed links from the subject entity to the object entity. "neighbourhoods" of entities are then defined as the set of outward links (subject neighbourhoods) and inward links (object neighbourhoods) from/to an entity. each relation and predicate p is associated with two unique, trainable embeddings v subj p , v obj @xmath0. we combine these learned relation/predicate embeddings with embeddings v text p derived from associated texts to obtain enhanced representations v •••:t as follows. zhang et.al. @xcite posit that the aggregated representations v agg subj (e) and v agg obj (e) provide ultra-fine grained type information about entities when playing the respective roles and observe that including entity type information into their models does not notably improve performance, suggesting that type information is already present. taking inspiration from that, we propose combining these aggregated representations with text based entity embeddings v text @xmath1. we then obtain association scores for a candidate predicate p, candidate subject entity s and candidate object entity o via a vector similarity measure (dot product in our case). these scores are then passed through sigmoid functions with trainable temperatures a subj , a @xmath2, b @xmath3, then summed with trainable mixing weights α subj , α @xmath4. the mixing weights are passed through the relu function to ensure that the raw scores can only contribute positively to the final score without canceling each other out. score(s, p, o) the resulting score, trained with a max-margin loss, allows us to rank propositions, with true propositions ranked higher. the full openki model incorporates a third scoring component that combines aggregated neighbour representations (equation 2 ) with a "query attention mechanism" similar to @xcite -see @xcite for details. for text enhanced models we replace the neighbour representations with equation 3. ## data sets following @xcite we test our models on two data sets: 1) english language extractions from the new your times (nyt) @xcite consisting of sentences with named entities identified and linked to freebase (fb) and 2) re-verb @xcite (an openie tool) extractions from clueweb @xcite (english language web texts) as preprocessed by openki authors foot_2 also with entities linked to freebase. for the nyt data, we use sentences as proxies for text predicates and for predicate texts we use whole sentences (including the entity mentions). texts for freebase relations are derived the second data set consists of reverb @xcite (an openie tool) extractions from clueweb with entities linked to freebase @xcite , as provided by openki authors foot_3 . text predicates in this data are provided in text form, which are used directly. we obtain texts for free-base relations in a similar way to the nyt data. note that original sentences from clueweb are not readily available for this data. to obtain entity texts for both data sets, we use the property type.object.name of the associated freebase entity, where present, or the entity span in the nyt source text in other cases 5 . most freebase entities also include a longer description text (the common.topic.description property). we concatenate the entity names and their descriptions to obtain a second text representation, used in the ". . . + desc" columns in ## experiments we follow the experiments presented in @xcite for effective comparison. in preliminary experiments, we additionally trained all model variants using only text representations (effectively to zero vectors), and found performance to be substantially degraded in all cases. similarly, the sota knowledge base completion model tucker @xcite performed very poorly when applied to the combined text predicate + kb relation graph for both data sets. source code for our experiments including data download links is available on github foot_5 . we use text embeddings derived from fasttext word embeddings @xcite and bert-small @xcite . for fasttext, we average the embeddings for words in the text. for bert we use two strategies: the average of token representations and the representation of the special "[cls]" token appended to all texts during standard bert pre-processing. we use 100 dimensional learned embeddings, learning rate 0.005 with the radam optimiser @xcite and batch size 128 for 150 epochs (clueweb data) and 70 epochs (nyt data). we train with max-margin loss with a margin of 1.0, using 16 negative examples for each positive example. negative samples consist of the entity pair and a uniformly sampled (no-positive) relation or predicate. for evaluation, with the nyt data we use the area under the precision-recall graph (auc-pr) for relation prediction over entity pairs. with the openie data we use mean average precision (map) on the task of ranking entity pairs. reported results are from the best of 5 runs for each configuration (as measured by development set performance). ## discussion: in we observe that including entity texts performs better than relation/predicate texts, even when entity texts are included as well (mostly ~3% improvement). this can probably be explained by the paucity of the predicate text representations for kb relations and that whole sentences contain extraneous information not relevant to the relationship between entities. future work with, for example, contextual bert representations of predicate spans and excluding kb relation texts may perform better. including entity descriptions is either similar or better than not including them (up to ~4%), in particular for bert. it is not surprising that bert can leverage the long-form entity descriptions effectively. average bert token embeddings perform better than the cls token embedding in most cases. the most surprising result is the relative performance between bert and fasttext, with fasttext outperforming bert with entity only text enhancement and providing the best performing models. it is not clear to us why this is the case. one hypothesis is that the fully conected layers projecting text representations to the learned embedding dimension may do better with a different, lower learning rate, and that this effect may be more pronounced with the larger bert representations. we plan to explore this in future work. it is worth noting that using sentences as proxies for text predicates is a rather weak setup. the majority of sentences contain a single entity pair, meaning that the sentences (as a predicate proxy) only appears in one subject and one object neighbour list. this provides little graph information for the model to utilise. the small proportion that do overalap appear to provide benefit however. openki identifies a compatibility between relations and entities through their co-occurrences in a graph. though a strong signal, our results indicate that this information is further enhancded by the detailed and nuanced information that can be found in both task source texts and entity and relation descriptions. text based information alone, however, has not been found to provide sufficient informa- @xcite 0.369 pycnn + att. @xcite 0.341 tion for good performance on these tasks, as seen in both our preliminary experiments without learned graph-based embeddings and previous work that relies on text based inference ( ## conclusion we investigated the task of integrating new information in the form of a collection of texts such as news articles into a knowledge base (kb), building on previous models that utilised information from the combined graph of knowledge base relations and predicates extracted from the texts using openie tools. we propose a mechanism for incorporating text representations of entities, kb relations and text predicates into the state of the art openki model, providing a substantial improvement in performance. from this we can conclude that source texts and entity and relation descriptions contain nuanced information useful to the task beyond that contained in graph structures in the knowledge base and extracted predicate propositions. our models represent a new state of the art on two data sets for this task.