KEPLET : Knowledge-Enhanced Pretrained Language Model with Topic Entity Awareness (2023)

Abstract: In recent years, Pre-trained Language Models (PLMs) have shown their superiority by pre-training on unstructured text corpus and then fine-tuning on downstream tasks. On entity-rich textual resources like Wikipedia, Knowledge-Enhanced PLMs (KEPLMs) incorporate the interactions between tokens and mentioned entities in pre-training, and are thus more effective on entity-centric tasks such as entity linking and relation classification. Although exploiting Wikipedia’s rich structures to some extent, conventional KEPLMs still neglect a unique layout of the corpus where each Wikipedia page is around a topic entity (identified by the page URL and shown in the page title). In this paper, we demonstrate that KEPLMs without incorporating the topic entities will lead to insufficient entity interaction and biased (relation) word semantics. We thus propose KEPLET, a novel Knowledge-Énhanced Pre-trained LanguagE model with Topic entity awareness. In an end-to-end manner, KEPLET identifies where to add the topic entity’s information in a Wikipedia sentence, fuses such information into token and mentioned entities representations, and supervises the network learning, through which it takes topic entities back into consideration. Experiments demonstrated the generality and superiority of KEPLET which was applied to two representative KEPLMs, achieving significant improvements on four entity-centric tasks.

## introduction pre-trained language models (plms) @xcite @xcite have shown their effectiveness on many natural language understanding tasks. to exploit the rich syntactic and semantic information in the pre-training mentioned entities she released crazy in love and baby boy . words crazy in love boy baby she released crazy in love and baby boy . words topic entity crazy in love boy baby she released crazy in love and baby boy . beyoncé mentioned entities words h"ps://en.wikipedia.org/wiki/beyoncé title: beyoncé there are three levels of interactions among words, mentioned entities and topic entity. we use to stand for the entity and words linkage, @xmath0. in general, keplms have achieved superior performance compared with plms in entity-centric tasks @xcite , like named entity recognition, entity typing etc. keplms have gained their efficacy by modeling entities mentioned in wikipedia-like textual corpus. unfortunately, they still did not fully exploit the entity knowledge there, in the sense that they neglected another important role entities named, topic entities, which are page titles in the corpus. for example, in fig. 1 , those sentences are actually from the page of beyoncé and around this celebrity as a topic. however, conventional keplms simply neglect such linkages, and treat those sentences independently, which will lead to both insufficient entity interaction and biased (relation) word semantics. take the sentence "she released crazy in love and baby boy." for example, if ignoring its topic entity beyoncé, keplms can no longer rely on this sentence to capture the interaction between crazy in love and beyoncé. moreover, the semantics of word "released" will be biased since it is between the above pair of entities, not between crazy in love and a common word "she". based on the above discussions, topic entities are indeed important to sentence semantics in wikipedia-like textual corpus. however, they are non-trivial to model with simple revisions to existing keplms. readers may wonder whether they can be treated similarly as mentioned entities, e.g., assign them certain position embeddings. however, we note that it is impractical since most topic entities do not explicitly appear in the sentences like mentioned entities2. readers may also think of using a co-reference resolution model to replace words like "she" with the topic model's mention before feeding them to keplms. however, embedding such a model to keplms will not only introduce resolution noise, but also be insufficient to cover cases where topic entity information can clarify the local semantics of non-pronoun positions, e.g., entity mentions with ambiguous names3. in this paper, we seek to fix the systematic neglect of topic entities in existing keplm efforts. we develop képlet, a general knowledge-énhanced pre-trained language model with topic entity awareness, which applies to most keplms. to exploit mature keplms as its base while filling in the gap that topic entities do not have explicit positions, képlet features a topic entity fusion module. to integrate topic entities into keplms, képlet first identifies potential fusing positions in sentences where topic entities can clarify local ambiguity including but not limited to co-references, through a gated neural network function @xcite . it then fuse the topic entity features into the hidden representations of those positions in a soft manner. finally, képlet trains the fusion module with the base keplm in an end-to-end manner, through a specially designed topic-entity-aware contrastive loss. to validate the generality and effectiveness of képlet, we conducted comprehensive experiments based on two representative keplms, i.e., luke (yamada et al., 2020) and ernie @xcite , on entity-centric benchmarks. the results demonstrate that képlet consistently improves the performance of these two keplms across the tasks. we summarize our contributions as follows: 1) we identify the systematic neglect of topic entities in existing keplm efforts. 2) we propose képlet with a novel topic entity fusion module and a topic-entity-aware loss, forcing existing keplms to fully exploit entity knowledge in wikipedia-like corpus. 3) képlet achieves the best performance among existing keplms on several entity-centric benchmarks. ## preliminaries in this section, we define two types of entities essential in wikipedia-based keplm pre-training, i.e., mentioned entities and topic entities. we also give an overview of how conventional keplms enhance plms by incorporating mentioned entities, but neglect topic entities. ## képlet: integrating topic entities in keplms so far, we have motivated the fusion of topic entities into keplms. the task then boils down to finding proper places in keplm training to fuse topic entities information. for mentioned entities, existing keplms @xcite @xcite @xcite fuse their information by considering them as special tokens spanning multiple positions in a sentence. however, topic entities are non-trivial to incorporate in a similar manner, as they do not directly appear in the sentence. naively assigning them artificial positions and doing insertions/replacements will potentially break the semantics of the sentences, thus deteriorating the lm's training. in this section, we detail how képlet integrates topic entities. képlet features two modules, fusing position identification and entity feature fusion, and we customize the transformer encoders to accommodate the two modules. besides, it employs a topic-entity-aware contrastive learning loss. fig. ## experiment settings we evaluate the effectiveness of képlet on extensive entity-centric tasks: entity typing, relationship classification, named entity recognition and extractive qa. to ensure a fair comparison, we follow the experiment settings of previous work @xcite @xcite . ## experiment results in this section, we aim to answer the following experimental questions: eq1. which feature fusion method can better integrate the topic entity with keplms? eq2. can topic entity fusion improve the performance of keplms on downstream tasks? and eq3. how should the topic entity fusion module of képlet be added to keplms to achieve optimal performance? ## computational overhead discussion our model, trained on knowledge-enhanced plms (keplms) for a single epoch, requires 128 hours on an a10 gpu-far less than the 11,520 hours needed for a keplm. assuming each minibatch holds m documents of c tokens each, the added computational burden over keplms is computed as m × c + m × c × c + m × (m -1). here, m × c represents the topic entity fusion position identification, m × c × c corresponds to attention-based entity fusion, and m×(m-1) pertains to the topic-entity-aware contrastive loss. this overhead is minimal as both m and c are small constants. furthermore, during inference, képlet bypasses topic entity position and fusion identification, keeping its computational demands in line with standard knowledge-enhanced language models. we group the related keplms works based on what context information is used when encoding sentences with entity knowledge (i.e., mentions), which are depending on sentential context only or additional non-sentential context. sentential context only keplms. most ke-plms pre-trains by encoding sentences with mentioned entities and aligning the encoded spans with their ground truth entities. ernie (zhang et al., 2019) integrates static entity embeddings into their plm with a novel fusion layer to fuse spans' representations with entities'. luke (yamada et al., 2020) treats both words and entities as tokens but uses different embeddings, self-attention layers, and masked token/entity prediction heads. it averages position embeddings of words in a span as the position embedding of the entity. different from ernie and luke, which require mention spans to be available, knowbert (liu et al., 2020) and eae (févry et al., 2020) incorporate entity linking modules to identify the spans, which are trained in an end-to-end manner. to deal with entity co-references in sentences, corefbert (ye et al., 2020) and tome (de jong et al., 2021) propose extending the alignments to not only explicit entity mentions but also implicit co-reference spans. wklm (xiong et al., 2020) pre-trains by distinguishing the ground truth entity's embeddings from those of randomly corrupted ones. k-adaptor (wang et al., 2021a) utilizes plug-in adapters @xcite to inject factual and linguistic knowledge without updating the lms. ## conclusion we propose képlet to address the oversight of topic entities in keplm learning. képlet performs topic entity fusion by identifying the fusing positions and fusing topic entity features through concatenation-or attention-based fusion. furthermore, we design a new pre-training task, i.e., topicentity-aware contrastive learning, for better topic entity fusion. experiment results on several entitycentric tasks prove the effectiveness of képlet. potential directions of the future work include: 1) adopting képlet to multi-language setting. since topic entities are language invariant @xcite , we can expand the topic-entity-aware contrastive learning for the same topic entity under different languages; 2) applying képlet to domain-specific knowledge-centric tasks like fact checking and fake news detection.