G ra SAME : Injecting Token-Level Structural Information to Pretrained Language Models via Graph-guided Self-Attention Mechanism (2024)

Abstract: Pretrained Language Models (PLMs) benefit from external knowledge stored in graph structures for various downstream tasks. However, bridging the modality gap between graph structures and text remains a significant challenge. Traditional methods like linearizing graphs for PLMs lose vital graph connectivity, whereas Graph Neural Networks (GNNs) require cumbersome processes for integration into PLMs. In this work, we propose a novel graph-guided self-attention mechanism, GraSAME. GraSAME seamlessly incorporates token-level structural information into PLMs without necessitating additional alignment or concatenation efforts. As an end-to-end, lightweight multimodal module, GraSAME follows a multi-task learning strategy and effectively bridges the gap between graph and textual modalities, facilitating dynamic interactions between GNNs and PLMs. Our experiments on the graph-to-text generation task demonstrate that GraSAME outperforms baseline models and achieves results comparable to state-of-the-art (SOTA) models on WebNLG datasets. Furthermore, compared to SOTA models, GraSAME eliminates the need for extra pre-training tasks to adjust graph inputs and reduces the number of trainable parameters by over 100 million.

## introduction the paradigm of pre-training and fine-tuning has increasingly become the standard approach for leveraging the inherent knowledge of language models in a wide range of natural language processing (nlp) tasks @xcite . pretrained language models (plms) like transformer @xcite , t5 @xcite , and gpt @xcite , which are trained on extensive text corpora, have demonstrated remarkable performance across various nlp challenges. however, these models primarily focus on textual data, presenting a significant limitation in processing structured information, such as knowledge graphs (kgs), molecular graph and social networks. such graph structures are crucial for storing external knowledge and can significantly enhance plm's performance on knowledge-driven tasks @xcite . this limitation becomes particularly evident in tasks that require a deep understanding of both textual and structural data, such as graph-to-text generation @xcite , kg-based fact checking @xcite , and translation between molecules and natural language @xcite . to address the challenge of processing structural input in plms, recent research has explored two main strategies: linearizing graph structures into text sequences @xcite @xcite , and encoding structural information using graph neural networks (gnns) @xcite @xcite . while linearization allows direct fine-tuning of plms, studies have shown that it often fails to preserve the inherent structural information and explicit node connectivity @xcite . conversely, while gnns effectively encode complex structures, the modality difference between text and graphs complicates the integration with plms, requiring additional training for aligning and concatenating embeddings from graph and textual modality @xcite . to merge the strengths of plms and gnns, we introduce grasame, a novel graph-guided self-attention mechanism, enhancing plms' ability to process graph inputs. as depicted in figure 1 , we construct a token-level hierarchical graph structure from the linearized graph sequence to maintain the input graph's structural integrity. grasame is designed to learn the token-level graph information from a gnn and integrate it seamlessly into the text representation for plm. based on a standard transformer architecture, we substitute the figure 1 : an example of kg-to-text generation. we visualize the hierarchical graph structure derived from the linearized graph input, tokenized by t5-large tokenizer. each token from the input text is depicted as a node. various relation types, each indicated by a unique color, are assigned between nodes to establish the hierarchical structure and ensure effective information flow among neighboring nodes. applied to kg-to-text generation task webnlg, we integrate grasame into the encoder-decoder model t5 with multi-task fine-tuning. the kg-totext generation is particularly suitable as it necessitates the processing of both graph and text information, providing a clear intuition to assess the effectiveness of grasame. our experiments demonstrate that grasame is compatible with various gnn architectures such as graphsage @xcite , gat @xcite and rgcn @xcite , yielding performance that not only surpasses baseline models but also comparable to state-of-the-art (sota) models. moreover, grasame effectively integrates structural information purely during fine-tuning and saves over 100 million trainable parameters. in summary, our contributions are: i) introducing a novel graph-guided attention mechanism grasame to incorporate explicit structural information into plms. this innovation enables plms to process both textual and structural inputs smoothly, bridging the modality gap of gnns and plms. with grasame, plms can dynamically interact with gnns, effectively interpreting graph inputs, which is crucial for nlp tasks that require structural information. ii) applying grasame to kg-to-text generation on webnlg datasets, achieving results comparable to sota models while saving over 100 million trainable parameters. ## model in this section, we detail the components of our model. theoretically, grasame is adaptable to any attention-based plms. we choose t5 model @xcite as our foundation due to its encoder-decoder architecture, which is well-suited for kg-to-text generation. ## experiments in this section, we introduce the details of our experiments on kg-to-text generation task. we modify t5 for conditional generation from huggingface @xcite , and implement grasame with the gnn layers provided by pytorch geometric @xcite . ## error analysis we illustrate three examples of generated text in in the first and second examples, the text generated by t5 includes hallucinations not present in the input data. in contrast, grasame strictly adheres to facts conveyed by the input graphs. when handling more complex graph inputs, both t5 and grasame produce fluent and accurate text without introducing extraneous information. however, t5 makes a minor grammatical error with a preposition. this indicates that grasame effectively mitigates the issue of hallucinations, particularly with short and simple inputs. moreover, for longer and more complex inputs, grasame demonstrates a superior understanding of the input structure, resulting in higher-quality text generation. ## conclusion in this work, we introduce grasame, a novel graph-guided self-attention mechanism that enables plms to process token-level structural information. with grasame, plms are capable of 927 model example input translate graph to english: <@xmath0.