Graph Language Models (2024)

Abstract: While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs – which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure – but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM’s architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility.

## introduction knowledge graphs (kgs) are essential for organizing vast data, to facilitate information retrieval, or revealing hidden insights for decisionmaking @xcite . kgs excel in explicitly representing manifold relationships, so with an expanding wealth of information they become crucial tools in the digital age. many kgs consist of knowledge triplets, where nodes are entities and edges represent relationships holding between them. each triplet represents a fact in pseudo-natural language, e.g., (thailand; capital; bangkok) in dbpedia @xcite . despite the (usual) simplicity of each individual 1 https://github.com/heidelberg-nlp/ graphlanguagemodels figure 1 : the glm inherits its architecture from a graph transformer, and its parameters from a lm. this enables it to jointly reason over graphs and language. to use gots effectively, we need meaningful encodings of their components. a natural choice is leveraging lms, as they can capture the semantics of textually encoded entities, relations or entire triplets. but lms are not prepared to capture graphstructured information and cannot model complex interactions in a got. to alleviate this problem, one can leverage graph nns (gnns). but gnns are not well suited to capture meanings associated with text, and hence often lms are used to convert nodes (and possibly edges) to language-based semantic embeddings. but in such settings, semantic encoding leveraged from lms and structural reasoning performed by gnns are separated and are driven by distinct underlying principles. we expect this to limit model performance if both textual and structural information are important for a task. in this work we introduce a graph language model (glm) that resolves this tension through early fusion of textual and structural information. most lms today are transformers. since transformers operate on sets, positional encoding (pe) is used to inform them about the inherent sequential ordering of linguistic inputs. in our glm formulation, we modify pe and self-attention to convert lms (i.e., sequence transformers) to graph transformers that natively operate on graphs, while preserving their lm capabilities. usually, a new architecture requires pretraining from scratch, which is extremely costly. by adopting some non-invasive changes in the lm's self-attention module, we transform the lm to a graph transformer (gt)while maintaining compatibility with its pretrained lm parameters. when encoding a graph, lm-like attention patterns process linearly organized textual information of individual triplets, while gt-like attention patterns aggregate information along the graph structure. hence, the glm inherits text understanding of triplets from the lm, while its gt architecture allows it to directly perform structural reasoning, without additional gnn layers. importantly, for text sequences -which can be seen as a special type of graph -the glm is identical to the original lm. this allows the glm to process interleaved inputs of text and got jointly, handling both modalities in a single framework. our main contributions are: (i) we propose graph language models (glms) and a theoretical framework to construct them. glms are graph transformers, which enables graph reasoning. simultaneously, they inherit and exploit lm weights, enabling them to represent and contextualize triplets in a got. further, by encoding texts and graph components alike, it can naturally take graph and text data as interleaved inputs. (ii) experiments on relation classification in conceptnet subgraphs show that glms outperform lm-and gnn-based methods for encoding gots -even when the inherited lm parameters are not updated during glm training. (iii) kg population experiments on wikidata subgraphs and corresponding wikipedia abstracts show that glms can reason over interleaved inputs of gots and text -again, outperforming strong lm-based methods. ## related work lms one way to augment lms with knowledge from kgs @xcite is to formulate pretraining objectives that operate on a kg. e.g., lms can be trained to generate parts of a kg, encouraging the lm to store kg content in its parameters. typically, single triplets are used for pretraining @xcite @xcite . in such cases, the graph structure is not a target of pretraining. some works generate larger substructures, such as paths or linearized subgraphs @xcite @xcite . in either case, the lm needs to memorize the kg, as it will not be part of the input during inference. another approach is to provide the linearized kg as part of the input during inference. this is common for kg-to-text generation @xcite @xcite , where models learn to take the linearized (typically small-sized) graphs as input. a recent trend is retrieval augmented generation, where relevant parts of a knowledge base, or kg, are retrieved, linearized and provided as part of a prompt @xcite . 2 in both options the graph must be linearized to fit the input or output of a sequence-to-sequence lm. hence, no graph priors can be enforced -instead, the lm has to learn the graph structure implicitly. by contrast, glms model a graph as a true graph and have inductive graph priors instilled in their architecture. this prepares a glm for more proficient graph reasoning, compared to a lm approach. gnns lms excel at representing single triplets, but struggle with structural reasoning. to alleviate this problem, lms can be combined with gnns. many approaches get node and edge features from lms and aggregate this information in the graph with @xcite @xcite . @xcite ; @xcite train models consisting of a lm and a gnn that encode interleaved text and graph inputs jointly. they also use a lm to obtain node features. while some approaches jointly train for textual understanding and graph reasoning, none offer a unified method. by contrast, our glm formulation seamlessly integrates both in a holistic framework for embedding language and kgs. graph transformers gts, a special type of @xcite , gain popularity in nlp and beyond @xcite . e.g., @xcite and @xcite train gts to generate text from kgs and amrs, respectively. most relevant to our work is @xcite , who use gts for kgto-text generation. similar to us, they employ pe matrices, but train their model from scratch, which limits its applicability: while their model trained on webnlg @xcite has a vocabulary size of 2,100, initializing a glm from t5, equips it with t5's full vocabulary of 32,128 tokens. concurrently, and independently from our work, @xcite also convert a lm to a graph transformer. they focus on data-to-text generation, where they unify table, key-value and kg structures in a unified graph format, and apply structureenhanced pre-training to support data-to-text generation with their structure-enhanced transformer model. they apply attention maps similar to ours to better capture the graph-structured input, which the pre-trained model rewrites into natural language. contrary to their work, we do not resort to structureenhanced pre-training -which is restricted in resources -but instead assess the glms' innate capabilities. we showcase the versatility of the inherited lm parameters in conjunction with our graph transformer architecture, by applying them to challenging reasoning tasks, where the model needs to reason over complementary inputs from text and graphs, and where it needs to infer information not present in the input, unlike data-to-text generation. moreover, we demonstrate that our architectural changes are highly compatible with the original lm weights, via linear probing experiments, where the glm outperforms conventional lm and graph transformer models. ## preliminary: graph transformers (gt) this section briefly introduces graph transformers, focusing on architectural choices relevant for our work. we also discuss some general properties of gnns that motivate our design choices in §4. positional encoding the self-attention mechanism of transformer models is permutation invariant, i.e., it doesn't have any notion of the order of its input elements. thus, positional encoding (pe) is used to inform lms of the ordering of tokens in a text @xcite . most approaches employ either absolute pe, where absolute token positions are encoded @xcite or relative pe, which encodes the relative position between pairs of tokens @xcite @xcite . absolute pe is typically combined with the input sequence and hence, the pe does not need to be encoded in self-attention (b @xmath0. for relative pe, b p encodes a bias depending on the relative distances between pairs of tokens -for example, by learning one scalar for each possible distance: similarly, gts use pes to encode the structure of the input, and hence, their pe has to encode a graph structure, as opposed to a sequence. this can again be done with absolute or relative pes. however, defining an "absolute position" of a node or edge in a graph is not straightforward. while many methods exist, they are not directly compatible with the usual (absolute) "counting position" known from sequence encoding in lms. in this work we thus focus on relative pe. given a directed acyclic path in a graph, we can define the (signed) distance between any pair of nodes along a path simply as the number of hops between the nodes. the sign can be set by the direction of the path. thus, by finding a consistent set of such paths in §4, we obtain relative distances and hence the graph's pe. ## experiments we assess the glms' capabilities for embedding gots in two experiments on relation (label) classification, i.e., classifying which relation belongs to a given head and tail entity. one experiment uses conceptnet (cn; @xcite subgraphs that we construct to enable analysis of the impact of structural graph properties. in a second experiment on wikidata @xcite subgraphs and associated wikipedia abstracts we test glms on interleaved inputs of text and graph. ## conclusion we present the graph language model (glm) -a graph transformer initialized with weights from a lm. it excels at graph reasoning, while simultaneously encoding textual triplets in the graph as lms do, thereby bridging the gap between lms and gnns. glms can natively reason over joint inputs from texts and graphs, leveraging and enhancing each modality. experiments show the glm's advantage over lm and gnn based baselines, even in a linear probing setting. in particular, glms greatly outperform graph transformers. this highlights the need for pretrained lm weights, even for graph reasoning. we therefore advocate glms as a valuable tool for advancing research in embedding and leveraging knowledge graphs for nlp tasks.