Improving Text-to-Text Pre-trained Models for the Graph-to-Text Task (2020)

Abstract: Converting a knowledge graph or sub-graph to natural text is useful when answering questions based on a knowledge base. High-capacity language models pre-trained on large-scale text corpora have recently been shown to be powerful when fine-tuned for the knowledge-graph-to-text (KG-to-text) task. In this paper, we propose two classes of methods to improve such pre-trained models for this task. First, we improve the structure awareness of the model by organizing the input as well as learning optimal ordering via multitask learning. Second, we bridge the domain gap between text-to-text and KG-to-text tasks via a second-phase KG-to-text pre-training on similar datasets and extra lexicalization supervision to make the input more similar to natural text. We demonstrate the efficacy of our methods on the popular WebNLG dataset. Our best model achieves an almost 3 point BLEU improvement on a strong baseline while lowering the relative slot-error-rate by around 35%. We also validate our results via human evaluation.

## introduction there is an abundance of highly accurate knowledge in various forms of structured data organized in tables and knowledge graphs (kgs). knowledge graphs, such as wikidata or dbpedia, are essential for many applications, where the data is stored in resource description framework (rdf) format with each triple containing a subject, a property, and an object. converting a knowledge sub-graph to natural text is an important step for natural communication in question answering applications such as in voice assistants. for example, when the user asks for information about a politician, the system not only needs to retrieve a sub-graph of rdf triples related with that person, but also needs to fluently describe these triples in natural language responses. as such, kg-to-text is a form of natural language generation (nlg) for generating textual descriptions from a sub-graph of a kg. traditional approaches separate the kg-to-text task into several micro-tasks, including discourse ordering, sentence structuring, lexicalization, and referring expression generation. however, with the advancement of end-to-end neural network models, attempts have been made to build neural nlg systems that treat the problem as a single task and perform both the planning and realization aspects of text generation simultaneously. although end-toend models trained from scratch did not outperform pipeline approaches @xcite @xcite , end-to-end models pre-trained on large-scale text corpora have been shown to achieve state-of-the-art when finetuned on the kg-to-text task @xcite . however, it is computationally prohibitive to develop large pre-trained models for converting any structure (e.g., kg, table) to text. moreover, there is much less available unlabeled structured data than text which is essential for self-supervised pretraining. thus, we look into ways of adapting the pre-trained text-to-text models for the kg-to-text task. this comes with a few challenges: first, the model is not aware of the implicit planning stage that can be essential in the task to form natural descriptions. in this stage, the triples are reorganized but the content within each triple is kept unchanged to smooth the description of the sub-graph. second, such language models are optimized for unstructured text and have generally not seen any kg representations during their pre-training phase. in this work, we study the advantages and limitations of these pre-trained models for the kg-to-text task, and propose improvements from two perspectives. first, we examine improving the model's awareness of structures. this can be achieved via two methods: (1) organizing the input before feeding it to the model; (2) learning the structure in the input by multitask learning of the optimal ordering. second, we bridge the domain gap between textto-text and kg-to-text tasks. this is achieved by (3) a second-phase kg-to-text pre-training, as well as (4) extra lexicalization supervision to make the input more similar to natural text. the rest of this paper is organized as follows: we describe the benchmark dataset and discuss the limitations of text-to-text pre-trained models on kg-to-text task in section 2. our proposed improvements are presented in section 3. the experimental setup and results are discussed in section 4 and 5. section 6 describes related work. we conclude in section 7 and present future directions. ## improvements upon pre-trained language models for kg-to-text inspired by the limitations of the baseline text-totext pre-trained models for the kg-to-text task, we propose methods to improve the triple structure input output (baked alaska, country, france) (hong kong, leader name, carrie lam) (france, leader name, gerard larcher) (france, language, french language) (baked alaska, region, hong kong) baked alaska is a dish from france, where the french language is spoken. carrie lam and gerard larcher are leaders in hong kong. ## experimental setup baseline our baseline model is based on finetuning bart using the flattened triples with the original order. we run our experiments five times with different random seeds to have a better idea of the random performance fluctuations. since small pre-trained models have limited generalization ability after fine-tuning (kale, 2020), we experiment with bart base (6 encoder and decoder layers, 140m parameters) to find the best configuration and deploy the best setting on bart large (12 encoder and decoder layers, 400m parameters). for pre-processing, we converted camel-case entities into plain text and cleaned special tokens. we finetune bart for 50 epochs using lamb optimizer @xcite with learning rate of 1.65e-4 and early stopping with a patience of 10. automatic metrics for measuring similarity between the output and references, we use bleu-4 @xcite with multiple references as the metric. for semantic fidelity, we use slot error rate (ser) measured by the percentage of unique input entities missing in the output. standard ser only considers exact match of entities in the text output, but many entities in the webnlg dataset need to be rephrased in the natural text descriptions (e.g., when the property is nationality, the object united states is often rephrased to american). thus, we modify the ser metric to account for the rephrasing of entities seen in gold references. as such, ser would capture cases where the model is missing some entities (and hence some triples) in the output but would fall short if an entity is misused in other relations. to check whether the improvements on the metrics are statistical significant, we also conduct independent two-sample t-tests with a threshold of 0.05. ## results for each of the improvements proposed in section 3, we consider multiple alternatives and compare their performance. as shown in ## related work in recent years, most studies on kg-to-text and more broadly data-to-text use neural sequence-tosequence (seq2seq) systems to generate text. yet seq2seq systems were found to perform poorly in correctness without proper semantic control @xcite . moreover, seq2seq models for text generation tasks treat all input words in a sequential order, often directly linearizing @xcite . this linearization is not optimal for encoding data with special structures as in kg-to-text. to address this problem, triple-specific or graph-based encoders have been proposed to better process the kg triple input @xcite @xcite @xcite . however, generic pre-trained models cannot be used with a modified encoder and thus the generalizability to unseen domain is limited. there are also works on adding semantic supervisions @xcite , however, it requires extensive efforts to annotate discourse relations in each reference text. due to the implicit microplanning stage in kgto-text, there also has been a debate over whether to use a pipeline model or an end-to-end model. studies found that using separate planning and text generation modules in a pipeline is better than one endto-end model trained from scratch @xcite @xcite . but with the recent advancement of high-capacity models pre-trained on large-scale text data, end-toend models achieved state-of-the-art performance without the need of any extra planning module @xcite . our work follows this approach while further making the end-to-end pre-trained model better fit the kg-to-text task. similar with our property lexicalization technique, @xcite explored having natural text templates for each unique slot in dialog acts. however the dialog acts can be treated as a tree with depth of 1, which is easier to understand by a seq2seq model than a kg structure. our property lexicalization not only considers using templates, but also adds tags to provide structural information. ribeiro et al. (2020) also used pre-trained models on kg-to-text task and experimented with task-adaptive pre-training. however, in this work bart is around 10 bleu worse than t5 on webnlg, but in our experiments bart achieves similar performance, probably indicating significant differences in the fine-tuning settings. moreover, this work found that pre-trained model with prior world knowledge do not need to understand the kg structure for relatively good performance, in contrast, we focus on improving the kg structure awareness of the model. ## conclusions in this work, we address several challenges of using pre-trained models for kg-to-text. we propose techniques to improve text-to-text pre-trained models for kg-to-text task by addressing triple structure awareness and by bridging the domain gap between text-to-text and kg-to-text. our techniques boost bleu and ser over a strong baseline. our best model yields the sota result on the enriched webnlg dataset and also shows significant improvement over baseline by qualitative analysis and human evaluation. to further utilize the benefit of large-scale pre-training, a future direction is to design a pre-training objective that better fits the kg-to-text task.