R elational W orld K nowledge R epresentation in C ontextual L anguage M odels: A R eview (2021)

Abstract: Relational knowledge bases (KBs) are commonly used to represent world knowledge in machines. However, while advantageous for their high degree of precision and interpretability, KBs are usually organized according to manually-defined schemas, which limit their expressiveness and require significant human efforts to engineer and maintain. In this review, we take a natural language processing perspective to these limitations, examining how they may be addressed in part by training deep contextual language models (LMs) to internalize and express relational knowledge in more flexible forms. We propose to organize knowledge representation strategies in LMs by the level of KB supervision provided, from no KB supervision at all to entity- and relation-level supervision. Our contributions are threefold: (1) We provide a high-level, extensible taxonomy for knowledge representation in LMs; (2) Within our taxonomy, we highlight notable models, evaluation tasks, and findings, in order to provide an up-to-date review of current knowledge representation capabilities in LMs; and (3) We suggest future research directions that build upon the complementary aspects of LMs and KBs as knowledge representations.

## introduction knowledge bases (kbs) are data structures that connect pairs of entities or concepts by semantically meaningful symbolic relations. decades' worth of research have been invested into using kbs as tools for relational world knowledge representation in machines @xcite @xcite @xcite @xcite . most large-scale modern kbs are organized according to a manually engineered schema that specifies which entity and relation types are permitted, and how such types may interact with one another. this explicit enforcement of relational structure is both an advantage and a drawback @xcite . on one hand, schemas support complex queries over the data with accurate, consistent, and interpretable answers. on the other hand, schemas are "ontological commitments" @xcite that limit flexibility in how knowledge is stored, expressed, and accessed. handcrafted schemas also require significant human engineering effort to construct and maintain, and are therefore often highly incomplete @xcite . language models as kbs? the tension between structured and unstructured knowledge representations is not new in natural language processing @xcite . however, only recently has an especially promising solution emerged, brought about by breakthroughs in machine learning software, hardware, and data. specifically, deep contextual language models (lms) like bert @xcite and @xcite have shown to be capable of internalizing a degree of relational world knowledge within their parameters, and expressing this knowledge across various mediums and tasks-in some cases, without the need for a predefined entity-relation schema @xcite . consequently, some have begun to wonder whether lms will partially or even fully replace kbs, given sufficiently large training budgets and parameter capacities. present work in this review, we summarize recent compelling progress in machine representation of relational world knowledge with lms. we propose to organize relevant work by the level of kb supervision provided to the lm (figure 1 ): for each supervision level, we provide notable examples in terms of methodology and/or findings, and compare the benefits and drawbacks of different approaches. we conclude in § 6 with our vision of the future, emphasizing the complementary roles of lms and kbs as knowledge representations. related work as this topic is relatively nascent, few related surveys exist. closest to our own work, colon-hernandez et al. (2021) cover methods for combining contextual language representations with graph representations, albeit with a comparatively narrow scope and no discussion of implicit knowledge. liu et al. (2021a) survey promptbased learning in lms, which overlaps with our discussion of cloze prompting in § 3.1, although relational world knowledge is not their main focus. ## preliminaries we briefly review preliminaries and assumptions necessary for our survey. ## word-level supervision the standard language modeling task is to predict the n-th word in a sequence of n words-that is, a conditional probability estimation task @xcite . while many variants of this task have been proposed to allow lms to condition their predictions on different inputs @xcite @xcite , a notable feature of all such approaches is that they operate at the word (and subword) level. if these supervision techniques do not incorporate kbs at all, how are they relevant when considering lms as relational knowledge representations? the answer is simple. typical language prompt handcrafting @xcite automatic prompt engineering @xcite @xcite adversarial prompt modification @xcite @xcite varying base prompts @xcite @xcite symbolic rule-based prompting @xcite statement scores ( § 3.2) single-lm scoring @xcite ) dual-lm scoring @xcite modeling corpora like wikipedia are known to contain kb-like assertions about the world @xcite . lms trained on enough such data can be expected to acquire some kb-like knowledge, even without targeted entity-or relation-level supervision. therefore, in order to motivate the necessity (if at all) of kb supervision, it is crucial to first understand what relational world "knowledge" lms acquire from word-level pretraining. in this section, we cover strategies to extract and utilize this knowledge under the cloze prompting ( § 3.1) and statement scoring ( § 3.2) protocols. ## entity-level supervision we next review entity-level supervision strategies for lms, most often toward improving performance in knowledge probes like lama ( § 3.1) and canonical nlp tasks like entity typing, entity linking, and question answering. we roughly categorize approaches from "least symbolic" to "most symbolic." on the former end of the spectrum, the lm is exposed to entity mentions in text but not required to link these mentions to an external entity bank ( § 4.1). on the latter end, the lm is trained to link mentions to the kb using late ( § 4.2) or mid-to-early fusion approaches ( § 4.3). ## relation-level supervision finally, we consider methods that utilize kb triples or paths to supervise lms for complex, often compositional tasks like relation classification, text generation, and rule-based inference. we again organize methods in the order of less to more symbolic. in this context, less symbolic approaches treat triples and paths as fully natural language ( § 5.1, 5.2). by contrast, more symbolic approaches learn distinct embeddings for relation types in the kb ( § 5.3). ## conclusion and vision in this review, we provide an overview of how lms may acquire relational world knowledge during pretraining and fine-tuning. we propose a novel taxonomy that classifies knowledge representation methodologies based on the level of kb supervision provided to an lm, from no explicit supervision at all to entity-and relation-level supervision. in the future, we envision a stronger synergy between the perspectives and tools from the language modeling and knowledge bases communities. in particular, we expect powerful and expressive lms, which are actively being developed in nlp, to be increasingly combined with large-scale kb resources to improve their knowledge recall and reasoning abilities. on the converse, we expect such kb resources to be increasingly generated directly by lms. within both of these directions, we hope that future work will continue to explore the themes discussed in this paper, in particular that of delineating and testing kb-level memorization versus generalization in lms. we also expect that more standardized benchmarks and tasks for evaluating lm knowledge will be developed, a direction that has recently seen some progress @xcite . as research at the intersection of lms and kbs is rapidly progressing, we look forward to new research that better develops and combines the strengths of both knowledge representations.