Simultaneously Self-Attending to Text and Entities for Knowledge-Informed Text Representations (2021)

Abstract: Pre-trained language models have emerged as highly successful methods for learning good text representations. However, the amount of structured knowledge retained in such models, and how (if at all) it can be extracted, remains an open question. In this work, we aim at directly learning text representations which leverage structured knowledge about entities mentioned in the text. This can be particularly beneficial for downstream tasks which are knowledge-intensive. Our approach utilizes self-attention between words in the text and knowledge graph (KG) entities mentioned in the text. While existing methods require entity-linked data for pre-training, we train using a mention-span masking objective and a candidate ranking objective – which doesn’t require any entity-links and only assumes access to an alias table for retrieving candidates, enabling large-scale pre-training. We show that the proposed model learns knowledge-informed text representations that yield improvements on the downstream tasks over existing methods.

## introduction self-supervised representation learning on large text corpora using language modeling objectives has been shown to yield generalizable representations that improve performance for many downstream tasks. examples of such approaches include bert @xcite , roberta @xcite , xlnet @xcite , gpt-2 @xcite , t5 @xcite etc. however, whether such models retain structured knowledge in their representation is still an open question @xcite @xcite which has led to active research on knowledge-informed rep- * equal contribution resentations @xcite @xcite . models that learn knowledge-informed representations can be broadly classified into two categories. the first approach augments language model pretraining with the aim of storing structured knowledge in the model parameters. this is typically done by augmenting the pre-training task, for example by masking entity mentions @xcite or enforcing representational similarity in sentences containing the same entities @xcite . while this makes minimal assumptions, it requires memorizing all facts encountered during training in the model parameters, necessitating larger models. the second approach directly conditions the representation on structured knowledge, for example fusing mention token representations with the mentioned entity's representation @xcite . in this paper we consider the latter approach to learning knowledge-informed representations. conditioning on relevant knowledge removes the burden on the model parameters to memorize all facts, and allows the model to encode novel facts not seen during training. however, existing methods typically assume access to entity-linked data for training @xcite , which is scarce and expensive to annotate, preventing large scale pre-training. moreover, these methods don't allow for bi-directional attention between both the text and the kg when representing text. we propose a simple approach to incorporate structured knowledge into text representations. this is done using self-attention @xcite to simultaneously attend to tokens in text and candidate kg entities mentioned in the text, in order to learn knowledge-informed representations after multiple layers of self-attention. the model is trained using a combination of a mention-masking objective and a weakly-supervised entity selection objective, which only requires access to an alias table to generate candidate entities and doesn't assume any entity-linked data for training. we show that this objective allows the model to appropriately attend to relevant entities without explicit supervision for the linked entity and learn representations that perform competitively to models trained with entity-linked data. we make the following contributions: (1) we propose knowledge-informed transformers (knit), an approach to learn knowledge-informed text representations which does not require entity-linked data for training, (2) we train knit on a large corpora curated from the web with wikidata as the knowledge graph, (3) we evaluate the approach on multiple tasks of entity typing and entity linking and show that it performs competitively or better than existing methods, yielding large improvements even while @xmath0. ## knowledge-informed transformers (knit) in this section, we describe the knit model as well as its training procedure. knit makes use of the mention-masking objective for training and conditions the encoder on both text as well as mentioned entities but does not assume any entity-linked data for training. figure 1 shows the overall model. ## conclusion we propose a simple approach to learn knowledgeinformed text representations using self-attention between text and mentioned entities. our approach does not rely on any entity-linked data for training, enabling large-scale pre-training. we show that the method learns better representations than competing approaches and also learns entity-linking without explicit linking supervision. in the future, it will be interesting to explore how such methods can be used to condition the text encoder on structured kg facts about entities. 3 were done using random seeds. three different sample datasets were collected for each of ope-nent(4%), figer(0.5%) and figer(0.05%). each sample would comprise an equal number of examples per entity type but randomised across the three runs. numbers reported in