Explanation Graph Generation via Generative Pre-training over Synthetic Graphs (2023)

Abstract: The generation of explanation graphs is a significant task that aims to produce explanation graphs in response to user input, revealing the internal reasoning process. This task is challenging due to the significant discrepancy be- tween unstructured user queries and structured explanation graphs. Current research commonly fine-tunes a text-based pre-trained language model on a small downstream dataset that is annotated with labeled graphs. However, due to the limited scale of available datasets, this approach may prove to be insufficient in bridging the gap between natural language text and structured graphs. In this paper, to alleviate the above limitations, we propose a novel pre-trained framework EG3P(for Explanation Graph Generation via Generative Pre-training over synthetic graphs) for the explanation graph generation task. Specifically, we first propose a text-to-graph generative task to pre-train the model with the goal of bridging the text-graph gap. Additionally, we propose an automatic corpus synthesis strategy for synthesizing a large scale of high-quality corpus, reducing the reliance on costly manual annotation methods. Experimental results on ExplaGraphs show the effectiveness of EG3P that our model surpasses all baseline systems with remarkable margins. Besides, further analysis demonstrates that EG3P is able to generate better explanation graphs on actual reasoning tasks such as CommonsenseQA and OpenbookQA.

## introduction generating an explanation to probe why the model obtains answers is a long-term goal in the development of intelligent systems, especially in reasoning-related tasks, such as e-snli @xcite , ecqa @xcite , hot-potqa @xcite and explagraphs @xcite should be able to consent to . argument： do not have the mental capacity to understand the of medical decisions. ## overview and background in this paper, we concentrate on the task of explanation graph generation. an example is depicted in figure 1 . given a piece of natural language text t , the model needs to generate a graph g which encapsulates the internal reasoning path of the input text. the specific content of the input t is contingent upon the specific downstream tasks (belief + augment + stance in stance prediction, question + answer in qa, etc.). for output g, we organize the graph into a sequence of triples in the depth-first search order. in practice, we employ a generative model and treat graph generation as a standard text generation task. a crucial point in this task is addressing the significant discrepancy in semantic expression structure between natural language texts and explanation inference graphs. an ideal way is to let the model learn this expression transfer on a large amount of natural language text and graph alignment data. however, due to the small size of labeled datasets, training on these datasets is difficult to address synthetic question simulated knowledge source language model input output staying healthy used for life dying has subevent causes living (stay healthy, used for, life) (dying, has subevent, living) ... (living, causes, life) ## the text2graph pre-training strategy typical pre-training strategies of various plms are based on the corpus of natural language text (e.g. mlm and nsp(bert), text denoising(bart), and text-to-text generation(t5)). however, in the explanation graph generation task, the explanation graph is different from the natural language text both in the representation form and in the semantic structure, which leads to a huge gap between the two kinds of representations. apart from typical tasks, some pre-training strategies are applied to recover the triples in the knowledge graph for knowledge graph question answering(kgqa) @xcite . however, such a pre-training method is not able to cover the explanation graph generation task due to the separation of pre-training steps between structured corpus and natural language, which is unable to bridge the gap between natural language and structured graphs. since the explanation graph generation task is required to translate natural language text into a graph, we believe the key point is to map the natural language texts to structured graphs in the learning process implicitly. to this end, we set the form of the pre-training task as "text-to-graph" in eg 3 p. as depicted in figure 3 , the format of the pre-training task is analogous to that of a normal generation task. given a query in natural language question and a simulated knowledge source, the model concatenates the two part together as input and generate a sequence of triples representing the reasoning graph from the query to the answer. by learning aligned "text-to-graph" pairs, the model acquires text-to-graph mapping in the process, and its capability for structured text generation is also enhanced. real input samples are presented in appendix b for further reference. the query and the graph of the answer come from the auto-construction method we propose, which will be discussed in the next section. to construct the simulated knowledge source (a collection of triples), we take the triples of the gold answer as a starting point and add random triples that are not relevant to the reasoning process to disrupt the collection. the final size of the simulated knowledge source is approximately 1.5 to 2 times the length of the graph. ## the construction of synthetic corpus pre-training tasks necessitate the support of largescale corpus. however, all the existing datasets with human-labeled graphs are small in scale due to the high cost of manually annotating, which is not enough to support the pre-training process. to address this issue, we propose an automatic method of constructing the pair of the natural language query and the explanation reasoning graph. the conventional way to get a graph from a piece of natural language text is to search in the external knowledge base. however, the complexity of searching would increase exponentially with the number of nodes and the length of edges in graphs. therefore, we invert this process, synthesizing a reasoning graph first and then constructing a query based on the graph next. question concept constraint concept answer staying healthy used for life dying has subevent easy staying healthy is used for [answer] and dying has a subevent of [i_e] then [i_e] will cause [answer] normal what is used for and staying healthy dying has a subevent of then will cause ? hard what is used for with ? staying healthy dying causes living i. concept: life life ii. randomly select triples whose tail is life staying healthy used for life causes living iii. randomly select triples whose tail is living and staying healthy ## conclusion in this paper, we propose a pre-training framework eg 3 p for a structured explanation generation task. distinct from existing pre-training tasks based on natural language text, eg 3 p focuses more on training mapping between natural language and graphs. meanwhile, due to the high cost of manually tagging, we construct queries from the synthetic graph automatically to get a large-scale corpus to support the pre-training process. using explagraph as a main benchmark, experimental results show that eg 3 p could significantly improve the ability of the model to generate explanations. in addition, on the other dataset, the results of the model after pre-training also showed a considerable improvement. our approach offers a new possibility for addressing the challenges of limited labeled data in natural language processing tasks. in the future, the ability of the model to generate explanation graphs will benefit from more datasets released with labels and more and more objective evaluation indicators put forward. additionally, while our current approach processes graphs as strings, utilizing a model architecture that is more suitable for graph generation may further enhance the model's graph generation ability.