{\mathcal{P}^2} : A Plan-and-Pretrain Approach for Knowledge Graph-to-Text Generation (2020)

Abstract: Text verbalization of knowledge graphs is an important problem with wide application to natural language generation (NLG) systems. It is challenging because the generated text not only needs to be grammatically correct (fluency), but also has to contain the given structured knowledge input (relevance) and meet some other criteria. We develop a plan-and-pretrain approach, \mathcal{P}^2 , which consists of a relational graph convolutional network (RGCN) planner and the pretrained sequence-tosequence (Seq2Seq) model T5. Specifically, the R-GCN planner first generates an order of the knowledge graph triplets, corresponding to the order that they will be mentioned in text, and then T5 produces the surface realization of the given plan. In the WebNLG+ 2020 Challenge, our submission ranked in 1st place on all automatic and human evaluation criteria of the English RDF-to-text generation task.

## introduction the webnlg 2020 challenge @xcite focuses on automating the conversion between text and knowledge graph domains, both of which are important ways to store and record knowledge. in brief, the challenge is comprised of two tasks and two languages, namely knowledge graph-to-text generation (g2t) and text-toknowledge graph extraction (t2g), with separate tracks for english and russian languages. our team, amazon ai (shanghai), participated in both tasks in english and achieved 1st place in each case. this report will primarily focus on our approach for the g2t task (which received by far the most competing submissions); for the other t2g task, we will briefly summarize our simple strategy in section 5. 1 our code is available at https://github.com/ qipengguo/p2_webnlg2020 . ## webnlg 2020 challenge overview the english version of webnlg 2020 has two tasks: the first task aims to automate knowledge graph-to-text generation (g2t), which is an extension of the g2t task in the previous webnlg challenge 2017 @xcite with broader categories of knowledge graph and text pairs. given a knowledge graph of several (subject, predicate, object) triplets, participant systems need to generate verbalizations of the knowledge graph that maintain the same information but in natural language. the other task, text-to-knowledge graph extraction (t2g), involves extracting entities and relations from text to form a knowledge graph. in terms of data, the english webnlg 2020 track @xcite covers knowledge graphs and text from a variety of domains including airport, artist, astronaut, athlete, building, celestial body, city, comics character, food, mode of transportation, monument, politician, sports team, university, and written work. 2 and the test sets includes three additional domains, film, scientist, and musical work. within these data, each knowledge graph consists of 2 to 7 triplets extracted from dbpedia. to collect the text description corresponding to each knowledge graph, crowd-source workers were asked to verbalize the graphs. the statistics of the dataset are shown in split number of samples train (t-g pairs) 35,426 valid (t-g pairs) 4,464 test (only g, for g2t) 1,779 test (only t, for t2g) 2,155 the overview of our p 2 framework is shown in figure 1 . for a given graph, the r-gcn planner generates a plan, which is a sequence of triplets in the form (subject, predicate, object), and the triplets' order corresponds with the order in which they will be expressed in the text. the plan is then fed into a seq2seq model for the final natural language generation. note that the idea of incorporating a planner and a text decoder has been introduced in @xcite , albeit not with a design that is specifically paired with a seq2seq model. but given that this work has shown the effectiveness of the planner, we focus on how to integrate it with a pretrained seq2seq model in a system. in this regard, our method contains four parts: (1) a r-gcn planner from @xcite , (2) a rule-based interface that converts plans into a seq2seq friendly format, (3) a pretrained t5 seq2seq model following the practice of @xcite , and (4) some canocalization rules to deal with special characters that are not contained with the dictionary of t5. we now describe each of these components in turn. ## experiments we first provide implementation details, followed by a presentation of quantitative comparisons using automatic and human evaluation metrics. ## a simple baseline for t2g extraction we also briefly describe the very simple baseline approach that we submitted for the t2g task at the webnlg challenge 2020. extracting knowledge graph triplets from text contains two steps, the first to identify the entities, and the second to obtain the corresponding relations between them. accordingly, we first do entity linking to match the entities in the text with the dbpedia ontology, and then query the relation between entities from the dbpedia database. ## conclusion we proposed p 2 , a plan-and-pretrain approach that integrates a r-gcn planner and the pretrained t5 model. the resulting performance gain results, at least in part, from unleashing the power of pretrained models, through both the planner and the canocalization of special characters to ensure a consistent vocabulary with t5. the effectiveness of our model is demonstrated through both the automatic and human evaluations of the webnlg 2020 challenge, where p 2 achieved a first-place ranking.