KALM : Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding (2023)

Abstract: With the advent of pre-trained language models (LMs), increasing research efforts have been focusing on infusing commonsense and domain-specific knowledge to prepare LMs for downstream tasks. These works attempt to leverage knowledge graphs, the de facto standard of symbolic knowledge representation, along with pre-trained LMs. While existing approaches leverage external knowledge, it remains an open question how to jointly incorporate knowledge graphs represented in varying contexts — from local (e.g., sentence), document-level, to global knowledge, to enable knowledge-rich and interpretable exchange across contexts. In addition, incorporating varying contexts can especially benefit long document understanding tasks that leverage pre-trained LMs, typically bounded by the input sequence length. In light of these challenges, we propose KALM, a language model that jointly leverages knowledge in local, document-level, and global contexts for long document understanding. KALM firstly encodes long documents and knowledge graphs into the three knowledge-aware context representations. KALM then processes each context with context-specific layers. These context-specific layers are followed by a ContextFusion layer that facilitates knowledge exchange to derive an overarching document representation. Extensive experiments demonstrate that KALM achieves state-of-the-art performance on three long document understanding tasks across 6 datasets/settings. Further analyses reveal that the three knowledge-aware contexts are complementary and they all contribute to model performance, while the importance and information exchange patterns of different contexts vary on different tasks and datasets.

## introduction large language models (lms) have become the dominant paradigm in nlp research, while knowledge graphs (kgs) are the de facto standard of symbolic knowledge representation. recent advances in knowledge-aware nlp focus on combining the two paradigms @xcite @xcite , infusing encyclopedic @xcite , commonsense @xcite , and domain-specific @xcite knowledge with lms. knowledgegrounded models achieved state-of-the-art performance in tasks including question answering @xcite , commonsense reasoning @xcite , and social text analysis @xcite . prior approaches to infusing lms with knowledge typically focused on three hitherto orthogonal directions: incorporating knowledge related to local (e.g., sentence-level), document-level, or global context. local context approaches argue that sentences mention entities, and the external knowledge of entities, such as textual descriptions @xcite and metadata @xcite , help lms realize they are more than tokens. document-level approaches argue that core idea entities are repeatedly mentioned throughout the document, while related concepts might be discussed in different paragraphs. these methods attempt to leverage entities and knowledge across paragraphs with document graphs @xcite @xcite . global context approaches argue that unmentioned yet connecting entities help connect the dots for knowledge-based reasoning, thus knowledge graph subgraphs are encoded with graph neural networks alongside textual content @xcite . however, despite their individual pros and cons, how to integrate the three document contexts in a knowledge-aware way remains an open problem. controlling for varying scopes of knowledge and context representations could benefit numerous language understanding tasks, especially those centered around long documents. bounded by the inherent limitation of input sequence length, existing knowledge-aware lms are mostly designed to handle short texts @xcite . however, processing long documents containing thousands of tokens @xcite requires attending to varying document contexts, disambiguating long-distance co-referring entities and events, and more. in light of these challenges, we propose kalm, a knowledge-aware language model for long document understanding. specifically, kalm first derives three context-and knowledge-aware representations from the long input document and an external knowledge graph: the local context represented as raw text, the document-level context represented as a document graph, and the global context represented as a knowledge graph subgraph. kalm layers then encode each context with context-specific layers, followed by our proposed novel contextfusion layers to enable knowledge-rich information exchange across the three knowledge-aware contexts. a unified document representation is then derived from contextspecific representations that also interact with other contexts. an illustration of the proposed kalm is presented in figure 1 . while kalm is a general method for long document understanding, we evaluate the model on six tasks and datasets that are particularly sensitive to broader contexts and external knowledge: political perspective detection, misinformation detection, and roll call vote prediction. extensive experiments demonstrate that kalm outperforms pretrained lms, task-agnostic knowledge-aware baselines, and strong task-specific baselines on all six datasets. in ablation experiments, we further establish kalm's ability to enable information exchange, better handle long documents, and improve data efficiency. in addition, kalm and the proposed contextfusion layers reveal and help interpret the roles and information exchange patterns of different contexts. ## related work knowledge graphs are playing an increasingly important role in language models and nlp research. commonsense @xcite @xcite @xcite and domain-specific kgs @xcite @xcite serve as external knowledge to augment pretrained lms, which achieves state-of-the-art performance on question answering @xcite @xcite @xcite @xcite @xcite , social text analysis @xcite @xcite , commonsense reasoning @xcite @xcite , and text generation @xcite . these approaches @xcite @xcite @xcite @xcite could be mainly categorized by the three levels of the context where knowledge injection happens. ## conclusion in this paper, we propose kalm, a knowledgeaware long document understanding approach that introduces external knowledge to three levels of document contexts and enables interactive exchange across them. extensive experiments demon-strate that kalm achieves state-of-the-art performance on three tasks across six datasets. our analysis shows that kalm provides insights into the roles and patterns of individual contexts, improves the handling of long documents with greater knowledge intensity, and has better data efficiency than existing works.