Adaptable and Interpretable Neural M emory O ver Symbolic Knowledge (2021)

Abstract: Past research has demonstrated that large neural language models (LMs) encode surprising amounts of factual information: however, augmenting or modifying this information requires modifying a corpus and retraining, which is computationally expensive. To address this problem, we develop a neural LM that includes an interpretable neuro-symbolic KB in the form of a “fact memory”. Each element of the fact memory is formed from a triple of vectors, where each vector corresponds to a KB entity or relation. Our LM improves performance on knowledge-intensive question-answering tasks, sometimes dramatically, including a 27 point increase in one setting of WebQuestionsSP over a state-of-the-art open-book model, despite using 5% of the parameters. Most interestingly, we demonstrate that the model can be modified, without any re-training, by updating the fact memory.

## introduction neural language models (lms) @xcite @xcite that have been pre-trained by self-supervision on large corpora contain rich knowledge about the syntax and semantics of natural language @xcite , and are the basis of much recent work in nlp. pretrained lms also contain large amounts of factual knowledge about the world @xcite @xcite . however, while large lms can be coerced to answer factual queries, they still lack many of the properties that knowledge bases (kbs) typically have. in particular, it is difficult to distinguish answers produced by memorizing factual statements in the pre-training corpus from lower-precision answers produced by linguistic generalization @xcite . it is also difficult to add or remove factual information without retraining the lm, an expensive pro-cess foot_0 . the difficulty of updating knowledge in neural lms contrasts with symbolic kbs, where it is very easy to add or modify triples, and is a major disadvantage of using a lm "as a kb"-as in many domains (news, product reviews, scientific publications, etc) the set of known facts changes frequently. symbolic kbs thus remain practically important @xcite , especially for nlp applications where text is hard to automatically process (e.g., scientific, technical, or legal) or tasks rich in information that exists only in structured form (e.g., technical specifications of a new product, where no product page or review text discussing it yet exists). motivated by this, past work has sought to combine the benefits of neural lms with the large, broad-coverage kbs that now exist @xcite @xcite . this paper continues this research program with a new knowledge-augmented lm called fact injected language model (film). film is a masked lm, where masks can be filled either from the token vocabulary or an entity vocabulary. the vector representation of each entity in a kb is jointly learned alongside other parameters of a transformer lm, and stored in a separate entity memory. film also includes a fact memory where each element is derived from a triple of vectors, representing a kb entity or relation. since these triples are defined compositionally from (representations of) entities and relations, they have an interpretable symbolic meaning: e.g., if e mtv is the vector representation of kb entity "mountain view, ca" and e google and r hq similarly correspond to "google inc" and the relation "headquartered in", these vectors can be used to construct a memory element f (e google , r hq , e mtv ) for the kb assertion "google, inc is headquartered in mountain film uses the contextually encoded mask token as a query to the fact memory. in this case, the contextual query chooses the fact key (charles darwin, born_in) which returns the set of values {united kingdom} (the value set can be multiple entity objects such as the case from calling the key [united kingdom, has_city]) . the returned object representation is incorporated back into the context in order to make the final prediction. note that the entity representations in the facts (both in keys and values) are shared with the entity memory. the portion within the dashed line follows the procedure from févry et al. (2020). view, ca". this means that the fact memory can be easily extended with new facts. in analysis on four benchmark question answering datasets we show that film improves significantly, and sometimes dramatically, over several strong baselines (e.g. bart @xcite and t5 @xcite ) and this improvement is even larger when removing train-test overlap. in one setting of webquestionssp, we outperform the next best performing model (rag @xcite ) by 27 points despite using only 5% of the number of parameters. most interestingly, we demonstrate that film models can be updated without any re-training, by modifying the fact memory. specifically, in §4.1, we show we can inject new fact memories at inference time, enabling film to correctly answer questions about pairs of entities that were never observed in the training (either during pre-training or fine-tuning). in §4.2 we also evaluate updating the model by inserting contra-positive facts that contradict facts mentioned in the pretraining data, and we show that film can correctly answer novel questions in this scenario as well. to summarize, this paper's contributions are: ## fact injected language model model the fact injected language model (film) model (see figure 1 ) extends the transformer @xcite architecture of bert @xcite with additional entity and facts memories. these memories store semantic information which can later be retrieved and incorporated into the representations of the transformer. similar to the approach in févry et al. (2020), entity embeddings will (ideally) store information about the textual contexts in which that entity appears, and by inference, the entity's semantic properties. the fact memory encodes triples from a symbolic kb, constructed compositionally from the learned embeddings of the entities that comprise it and implemented as a key-value memory which is used to retrieve entities given their kb properties. this combination results in a neural lm which learns to access information from a symbolic kb. ## modifying the knowledge base because our model defines facts symbolically, it can in principle reason over new facts injected into its memory, without retraining any parameters of the model. since existing datasets do not directly test this capability, we elected to construct variants of freebaseqa and webquestionssp where we could simulate asking questions that are answerable only from newly injected kb facts. the approach we used was to ( 1 ) identify pairs of entities that occur in both a question and answer of some test example; (2) filter out such pairs from the kb as well as all pre-training and fine-tuning data; and (3) test the system trained on this filtered data, and then manually updated by injecting facts about those entity pairs. this filtering procedure is reminiscent of that used by @xcite , but also addresses pretraining / test-set overlap. ## conclusion we presented film, a neural lm with an interpretable symbolically bound fact memory. we demonstrated the effectiveness of this method by outperforming many state-of-the-art methods on four benchmark knowledge intensive datasets. we used the model's symbolic interface to change the output of the lm by modifying only the nonparametric memories, without any additional training. we showed film could incorporate newly injected facts unseen during training. additionally, we can modify facts, such that they contradict the initial pre training text, and our model is still largely able to answer these questions correctly.