A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models (2022)

Abstract: Pre-trained language models (PLMs) cannot well recall rich factual knowledge of entities exhibited in large-scale corpora, especially those rare entities. In this paper, we propose to build a simple but effective Pluggable Entity Lookup Table (PELT) on demand by aggregating the entity’s output representations of multiple occurrences in the corpora. PELT can be compatibly plugged as inputs to infuse supplemental entity knowledge into PLMs. Compared to previous knowledge-enhanced PLMs, PELT only requires 0.2%-5% pre-computation with capability of acquiring knowledge from out-of-domain corpora for domain adaptation scenario. The experiments on knowledge-related tasks demonstrate that our method, PELT, can flexibly and effectively transfer entity knowledge from related corpora into PLMs with different architectures. Our code and models are publicly available at https://github.com/thunlp/PELT

## introduction recent advance in pre-trained language models (plms) has achieved promising improvements in various downstream tasks @xcite . some latest works reveal that plms can automatically acquire knowledge from largescale corpora via self-supervised pre-training and then encode the learned knowledge into their model parameters @xcite @xcite . however, due to the limited capacity of vocabulary, existing plms face the challenge of recalling the factual knowledge from their parameters, especially for those rare entities @xcite . to improve plms' capability of entity understanding, a straightforward solution is to exploit @xcite 5.0m an external entity embedding acquired from the knowledge graph (kg) @xcite @xcite , the entity description @xcite , or the corpora @xcite . in order to make use of the external knowledge, these models usually learn to align the external entity embedding @xcite to the their original word embedding. however, previous works ignore to explore entity embedding from the plm itself, which makes their learned embedding mapping is not available in the domain-adaptation. other recent works attempt to infuse knowledge into plms' parameters by extra pre-training, such as learning to build an additional entity vocabulary from the corpora @xcite , or adopting entity-related pre-training tasks to intensify the entity representation @xcite @xcite . however, their huge pre-computation increases the cost of extending or updating the customized vocabulary for various downstream tasks. in this paper, we introduce a simple but effective pluggable entity lookup table (pelt) to infuse knowledge into plms. to be specific, we first revisit the connection between plms' input features and output representations for masked language modeling. based on this, given a new corpus, we aggregate the output representations of masked tokens from the entity's occurrences, to recover an elaborate entity embedding from a well-trained plm. benefiting from the compatibility and flexibility of the constructed embedding, we can directly insert them into the corresponding positions of the input sequence to provide supplemental entity knowledge. as shown in we conduct experiments on two knowledgerelated tasks, including knowledge probe and relation classification, across two domains (wikipedia and biomedical publication). experimental results show that plms with pelt can consistently and significantly outperform the corresponding vanilla models. in addition, the entity embedding obtained from multiple domains are compatible with the original word embedding and can be applied and transferred swiftly. ## methodology in this section, we first revisit the masked language modeling pre-training objective. after that, we introduce the pluggable entity lookup table and explain how to apply it to incorporate knowledge into plms. ## conclusion in this paper, we propose pelt, a flexible entity lookup table, to incorporate up-to-date knowledge into plms. by constructing entity embeddings on demand, plms with pelt can recall rich factual knowledge to help downstream tasks.