Stage-wise Fine-tuning for Graph-to-Text Generation (2021)

Abstract: Graph-to-text generation has benefited from pre-trained language models (PLMs) in achieving better performance than structured graph encoders. However, they fail to fully utilize the structure information of the input graph. In this paper, we aim to further improve the performance of the pre-trained language model by proposing a structured graph-to-text model with a two-step fine-tuning mechanism which first fine-tunes model on Wikipedia before adapting to the graph-to-text generation. In addition to using the traditional token and position embeddings to encode the knowledge graph (KG), we propose a novel tree-level embedding method to capture the inter-dependency structures of the input graph. This new approach has significantly improved the performance of all text generation metrics for the English WebNLG 2017 dataset.

## introduction in the graph-to-text generation task @xcite , the model takes in a complex kg (an example is in figure 1 ) and generates a corresponding faithful natural language description ( meanwhile, the plms might also fail to cover all relations in the kg by creating incorrect or missing facts. for example, in we explore the proposed stage-wise fine-tuning and structure-preserving embedding strategies for graph-to-text generation task on webnlg corpus @xcite . our experimental results clearly demonstrate the benefit of each strategy in achieving the state-of-the-art performance on most commonly reported automatic evaluation metrics. ## method token embeddings position embeddings tree-level embeddings triple role embeddings s| pos 1 rol 1 lv 2 karnataka pos 2 rol 1 lv 2 p| pos 3 rol 1 lv 2 northeast pos 4 rol 2 lv 2 [cls] pos 0 rol 0 lv 0 ... ... ... ... 1 tional position embeddings to capture the structure of the kg. to feed the input for the large-scale transformer-based plm, we flatten the graph as a concatenation of linearized triple sequences: ## related work the webnlg task is similar to wikibio generation @xcite , amrto-text generation @xcite and ro-towire @xcite . previous methods usually treat the graphto-text generation as an end-to-end generation task. those models @xcite @xcite usually first lineralize the knowledge graph and then use attention mechanism to generate the description sentences. while the linearization of input graph may sacrifice the inter-dependency inside input graph, some papers @xcite @xcite category output t5-large andrew white (born in 2003) is a musician who is associated with the band kaiser chiefs and marry banilow. he is also associated with the label polydor records and is signed to b-unique records. s| aleksandra kovač p| activeyearsstartyear o| 1990 t5-large walter baade was born in the german empire and graduated from the university of gottingen. he was the doctoral student of halton arp and allan sandage and was the discoverer of 1036 ganymed. s| walter baade p| doctoralstudent o| halton arp; s| walter baade p| doctoralstudent o| allan sandage t5-large +wiki 11264 claudiomaccone was born on the 26th of november, 2005. he has an orbital period of 1513.722 days, a periapsis of 296521000.0 kilometres and an apoapsis of 475426000.0 kilometres. s| 11264 claudiomaccone p| epoch o| 2005-11-26; s| aleksandr prudnikov p| club o| fc amkar perm t5-large +position the chairman of fc spartak moscow is sergey rodionov. aleksandr prudnikov plays for fc spartak moscow and manages fc amkar perm. [ s| fc amkar perm p| manager o| gadzhi gadzhiyev; s| aleksandr prudnikov p| club o| fc amkar perm ] ## conclusions and future work we propose a new two-step structured generation task for the graph-to-text generation task based on a two-step fine-tuning mechanism and novel treelevel position embeddings. in the future, we aim to address the remaining challenges and extend the framework for broader applications.