Graph-Induced Transformers for Efficient Multi-Hop Question Answering (2022)

Abstract: A graph is a suitable data structure to represent the structural information of text. Recently, multi-hop question answering (MHQA) tasks, which require inter-paragraph/sentence linkages, have come to exploit such properties of a graph. Previous approaches to MHQA relied on leveraging the graph information along with the pre-trained language model (PLM) encoders. However, this trend exhibits the following drawbacks: (i) sample inefficiency while training in a low-resource setting; (ii) lack of reusability due to changes in the model structure or input. Our work proposes the Graph-Induced Transformer (GIT) that applies graph-derived attention patterns directly into a PLM, without the need to employ external graph modules. GIT can leverage the useful inductive bias of graphs while retaining the unperturbed Transformer structure and parameters. Our experiments on HotpotQA successfully demonstrate both the sample efficient characteristic of GIT and its capacity to replace the graph modules while preserving model performance.

## introduction graphs are a widely employed data structure from fake news detection in social media @xcite to molecular graph generation in biomedicine @xcite . more recently, graphs have come to manifest themselves in many subdomains of natural language processing (nlp) for its topological properties that provide useful connectivity inductive bias for model reasoning. particularly, some models in the multi-hop question answering (mhqa) tasks like hotpotqa @xcite and musique @xcite that require textual reasoning through multiple paragraphs (i.e., multi-hop) make use of such graph structures to model the internal structural information within the given document. the external graphs used in these studies have demonstrated the * work was done while working at kaist. effectiveness of harnessing the information from textual units such as paragraphs and sentences to perform multi-hop reasoning in the qa setting. the graphs used in mhqa models like hgn @xcite , sae @xcite and dfgn @xcite are typically placed on top of the pre-trained language models (plm) like bert @xcite and roberta @xcite . these graphs use graph neural network's (gnn) message propagation mechanism to update the node and edge parameters, learning to extract useful inductive, structural bias from the graph. these features are then fused with the contextualized representations from the plm to leverage the information derived from this connectivity (e.g., paragraph-to-sentence) of these graphs. despite the merits of these graphs, such graph modules require numerous samples (i.e., sample inefficiency) to acquire the necessary inductive bias, increasing the overall cost in training. this significantly hinders the model's use of graph connectivity information, especially in a low-resource scenario. to address the sample inefficient training while preserving the structural information of graphs, we propose the graph-induced transformer (git). git is a simple, intuitive approach to leverage graphs in mhqa tasks without changing the plm architecture or using additional graph modules. by injecting attention patterns derived from text-based graphs, git is able to take advantage of the graphs without the extra modules to fuse graph representations with the text (figure 1 ). this enables git to: (i) conduct sample-efficient training under a low-resource setting, and (ii) successfully replace the graph module while preserving performance. we conduct experiments to prove the advantages of using git instead of separate graphs and provide a range of empirical evidence to show the effectiveness of git on the above two aspects. ## related works mhqa models that leverage the useful connectivity inductive bias of graphs have shown substantial performance gains. with dfgn @xcite using entity node graphs, sae @xcite using sentence nodes and hgn @xcite leveraging the hierarchical, paragraph-sentenceentity nodes for its graph, the use of instancespecific graphs has led the development of graphbased mhqa models. nevertheless, this has led to sample inefficient training that prevents the graph modules from sufficiently acquiring the ability to leverage the interconnected node and edge information, especially in low-resource settings. meanwhile, some works have tried to modify the transformer architecture for graph modeling. graph-bert @xcite added nodespecific positional embeddings and modified the encoder to solve the suspended animation problem of graph neural networks (gnn). other studies incorporated heterogeneous graphs by converting them into meta-paths @xcite or using relative temporal positional encodings @xcite in transformers. their modified architectures, however, are specific to graph tasks only, preventing their direct use in the mhqa tasks (and other downstream nlp tasks in general) because they preclude the use of text as input and do not reuse the pre-trained parameters like those of bert. git, in contrast, is designed to handle the mhqa tasks by integrating these graphs into plms while preserving the plm architecture for generalization. ## graph-induced transformer the core idea behind git is to represent the node connections in the form of masked self-attention layers. since the attention mask is an inherent part of the transformer architecture, we can simply alter the attention mask pattern to inject the connective inductive bias of graphs. the major benefit git brings with its graph injected as attention masks is the natural and explicit integration of its inductive bias without the need to learn about the graph structure, thereby ensuring a sample efficient utilization of the graph information during training. the graph-derived attention masks regulate the information flow among the intermediate representations, emulating the inter-node connections within a graph (figure 2 ). in git, the nodes of the graph are represented as a group (or chunks) of tokens that belong to a homogeneous textual unit; if a group of tokens constitute a sentence, then the group as a whole corresponds to a single sentence node. edges of the graph are represented as an attention connection between these two groups of tokens. the dynamically extracted attention mask selectively blocks the connections in a self-attention layer, except for in the above equation, h n @xmath0, where @xmath1, q, k and @xmath2, key and value projection matrix, respectively. o n @xmath3, where the k-th head is assigned an attention mask m g which enforces the connection between one textual unit (e.g., paragraph) to another (e.g., sentence) by zeroing out the other attentions and maintaining the remaining attention coefficients for the connection. since there are multiple edge types that connect one node type to another, we define an edge type as a pair of head and tail node types (e.g., paragraphto-sentence), in addition to the edge types of the original graph. the different edge types are then applied to the transformer encoder as a set of attention masks by assigning a head per edge type; this is based on the previous studies @xcite ) that each head in a transformer learns different linguistic concept. furthermore, considering that nodes in the graph are updated through l different steps given l gnn layers, we allocate l layers inside the plm to emulate the message propagation in gnns. to promptly adjust to different end tasks, we keep the last layer as an original fully connected self-attention layer. ## conclusions this work presented git, the graph-induced transformer that drastically improved mhqa models' sample efficiency and replaces graphs in the models while retaining their performance. our empirical evidences demonstrated that models can enjoy the benefits of connective inductive bias of graphs without additional graph modules in place. the design of git also allowed us to reuse the parameters of plm while incorporating the graph information. future directions of our work may include using git in downstream nlp applications where the graph inductive bias is necessary and dataset is scarce.