Entities as Experts: Sparse Memory Access with Entity Supervision (2020)

Abstract: We focus on the problem of capturing declarative knowledge about entities in the learned parameters of a language model. We introduce a new model—Entities as Experts (EaE)—that can access distinct memories of the entities mentioned in a piece of text. Unlike previous efforts to integrate entity knowledge into sequence models, EaE’s entity representations are learned directly from text. We show that EaE’s learned representations capture sufficient knowledge to answer TriviaQA questions such as “Which Dr. Who villain has been played by Roger Delgado, Anthony Ainley, Eric Roberts?”, outperforming an encoder-generator Transformer model with 10x the parameters on this task. According to the Lama knowledge probes, EaE contains more factual knowledge than a similar sized Bert, as well as previous approaches that integrate external sources of entity knowledge. Because EaE associates parameters with specific entities, it only needs to access a fraction of its parameters at inference time, and we show that the correct identification and representation of entities is essential to EaE’s performance.

## introduction neural network sequence models, pre-trained as language models, have recently revolutionized text understanding @xcite @xcite , and recent work has suggested that they could take the place of curated knowledge bases or textual corpora for tasks such as question answering @xcite . in this paper, we focus on developing neural sequence models that capture the knowledge required to answer questions about real world entities. to this end, we introduce a new model architecture that can access distinct and independent representations of the entities mentioned in text. unlike other efforts to inject entity specific knowledge into sequence models @xcite @xcite our model learns entity representations from text along with all the other model parameters. we call our model entities as experts (eae), since it divides the parameter space according to entity identity. this name also reflects eae's similarities with the massive mixture of experts @xcite , as well as other work that integrates learned memory stores into sequence models @xcite . to understand the motivation for distinct and independent entity representations, consider figure 1 . a traditional transformer @xcite needs to build an internal representation of charles darwin from the words "charles" and "darwin", both of which can also refer to different entities such as the charles river, or darwin city. conversely, eae can access a dedicated representation of "charles darwin", which is a memory of all of the contexts in which this entity has previously been mentioned. this memory can also be accessed for other mentions of darwin, such as "charles robert darwin" or "the father of natural selection". retrieving and re-integrating this memory makes it easier for eae to find the answer. we train eae to predict masked-out spans in english wikipedia text @xcite ; to only access memories for entity mention spans; and to access the correct memory for each entity mention. mention span supervision comes from an existing mention detector, and entity identity supervision comes from wikipedia hyperlinks. by associating memories with specific entities, eae can learn to access them sparsely. the memory is only accessed for spans that mention entities, and only the mentioned memories need to be retrieved. we evaluate eae's ability to capture declarative knowledge using the lama knowledge probes introduced by @xcite , as well as the open-domain variants of the triviaqa and web-questions question answering tasks @xcite . on both tasks, eae outperforms related approaches with many more parameters. an in-depth analysis of eae's predictions on triviaqa shows that the correct identification and reintegration of entity representations is essential for eae's performance. we further demonstrate that eae's learned entity representations are better than the pre-trained embeddings used by @xcite ; @xcite at knowledge probing tasks and the ta-cred relation extraction task @xcite . we show that training eae to focus on entities is better than imbuing a similarsized network with an unconstrained memory store, and explain how eae can outperform much larger sequence models while only accessing a small proportion of its parameters at inference time. ## approach let @xmath0. . . e n } be a predefined set of entities, and let @xmath1, w 1 . . . w m } be a vocabulary of tokens. a context @xmath2. . . x l ] is a sequence of tokens x @xmath3. each context comes with the list of the mentions it contains, @xmath4. . . m m ], where each mention m @xmath5, s m i , t m i ) is defined by its linked entity e m i , start token index s m i and end token index t m i . entity mentions might not be linked to a specific entity in e, thus e m @xmath6, where e ∅ refers to the null entity. ## knowledge probing tasks we follow previous work in using cloze tests and question answering tasks to quantify the declarative knowledge captured in the parameters of our model @xcite . ## comparison to alternative entity representations eae is novel in learning entity representations along with the parameters of a transformer model. here, we provide a direct comparison between eae's learned representations and the externally trained, fixed representations used by ernie @xcite and @xcite . from three different sources and two different training scenarios. the initial embedddings are either the transe-wikidata embeddings used by ernie @xcite ; the deep-ed embeddings used by @xcite ; or the random embeddings used by eae. the embeddings are either frozen, following ernie and knowbert, or trained along with all other network parameters. 8 along with the knowledge probing tasks from section 5, we report performance on the on the 9-way entity typing task from @xcite . it is clear that learning entity representations is beneficial. however, initializing the entity memory with deep-ed embeddings leads to gains on most of the knowledge probing tasks, suggesting that there are potential benefits from combining training regimens. meanwhile, the best entity typing results come from initializing with wikidata embeddings, possibly because wikidata has high quality coverage of the types ('person', 'organization', 'location', etc.) used. finally, we point out that both ernie and knowbert differ from eae in other ways (see appendix a). knowbert in particular incorporates wordnet synset embeddings as well as entity embeddings, leading to entity typing accuracy of 76.1. future work will explore different ways of combining learned embeddings with knowledge from diverse external sources. as discussed in section 3, there are significant differences between eae and knowbert other than the choice of entity representation. in particular, knowbert has an explicit entity-entity attention mechanism. to determine whether this has a significant effect on a model's ability to model entity-entity relations, we evaluate eae on tacred @xcite dataset introduced by @xcite . 9 ## conclusion we introduced a new transformer architecture, eae, which learns entity representations from text along with other model parameters. our evaluation shows that eae is effective at capturing declarative knowledge and can be used for a wide variety of tasks -including open domain question answering, relation extraction, entity typing and knowledge probing tasks. our entity representations influence the answer predictions for open-domain question answering system and are of high quality, compared to prior work such as knowbert. our model learns representations for a pre-fixed vocabulary of entities, and cannot handle unseen entities. future work can explore representations for rare or unseen entities, as well as developing less memory-intensive ways to learn and integrate entity representations. furthermore, integrating information from knowledge-bases can further improve the quality of entity representation.