Self-supervised Graph Masking Pre-training for Graph-to-Text Generation (2022)

Abstract: Large-scale pre-trained language models (PLMs) have advanced Graph-to-Text (G2T) generation by processing the linearised version of a graph. However, the linearisation is known to ignore the structural information. Additionally, PLMs are typically pre-trained on free text which introduces domain mismatch between pre-training and downstream G2T generation tasks. To address these shortcomings, we propose graph masking pre-training strategies that neither require supervision signals nor adjust the architecture of the underlying pre-trained encoder-decoder model. When used with a pre-trained T5, our approach achieves new state-of-the-art results on WebNLG+2020 and EventNarrative G2T generation datasets. Our method also shows to be very effective in the low-resource setting.

## introduction graph-to-text (g2t) generation @xcite is the task of generating natural language from graph-structured data. while there are several tasks that could leverage a g2t component @xcite @xcite the direct generation of text description from knowledge graphs (kgs) have attracted a lot of attention due to its potential in providing a more accessible presentation of knowledge to non-experts @xcite . in parallel, transformer-based @xcite pre-trained language models (plms) such as bart @xcite , and t5 @xcite have facilitated state-of-the-art (sota) results on several tasks, including earlier sota results for g2t @xcite @xcite . it has been argued that their success, in part, is due to factual memorisation that guides the generation @xcite ). 1 our code is available at https://github.com/ jiuzhouh/graph-masking-pre-training . although plms benefit the g2t generation, the linearisation step required to use these models ignores the structural information of the graph @xcite , while explicitly modelling structured data could also lead to catastrophic forgetting of distributional knowledge @xcite . to address this, @xcite proposed adding extra positional embedding layers to capture the inter-dependency structures of input graphs. @xcite proposed using a structureaware adapter in plms to supplement the input with its graph structure. for table data, @xcite considered the structure of the table input by predicting the surrounding cells for a cell in a table. however, these methods either change the design of the plms (limiting their use for other task settings) or require labelled training data to capture the graph structure information. in this work, we propose self-supervised graph masking pre-training strategies to enhance the structure awareness of plms. to achieve this, we formulate several graph masking strategies to inject local and global awareness of the input structure into the plm. our method has two key advantages: (i) it does not require to introduce extra layers or change of architecture in the underlying plm, and (ii) it pre-trains the plms in a self-supervised setting on graphs, without requiring labelled training data. starting from an existing plm, we further pretrain it with our approach, then the fine-tuning on downstream tasks is done as per usual. we conduct extensive experiments on three g2t generation datasets of diverse graphs. our empirical findings highlight that our self-supervised strategies significantly outperform a strong underlying t5 baseline and achieve two new sota results on two of the datasets webnlg+2020 @xcite and eventnarrative @xcite . additionally, we show our pre-training strategies are very efficient in utilising data and have a great potential for low-resource setting. our desiderata is to infuse structural knowledge into widely used pre-trained encoder-decoder transformer models, without modifying the model architecture or relying on supervision signal. to achieve this, we propose three self-supervised learning tasks to further pre-train a t5-large (raffel et al., 2020) model prior to fine-tuning on g2t generation downstream tasks. in this section we first describe our graph linearisation step which prepares the data in the right format for t5 encoder while injecting some weak structural information into the input ( §2.1), then we introduce our three graph masking pre-training tasks ( §2.2). ## experiments in this section we outline the experimental setups ( §3.1), followed by downstream g2t generation results in full ( §3.2) and low-resource scenarios ( §3.3). we also present a set of generated outputs from our models ( §3.4), and finish by providing an analysis ( §3.5) on the effect of pre-training data size, and an ablation on the role of input augmentation with level markers. ## conclusion and future work we proposed various self-supervised pre-training strategies to improve the structural awareness of plms without refining the architecture or relying on labelled data. our graph masking strategies outperformed the strong plm baseline and achieve new state-of-the-art results on webnlg+2020 and eventnarrative datasets. we demonstrated that our approach is very efficient in utilising even a small pre-training or fine-tuning datasets. for future work, we will explore different graph masking strategies to adapt for different domains of graph.