R e G en: R einforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models (2021)

Abstract: Automatic construction of relevant Knowledge Bases (KBs) from text, and generation of semantically meaningful text from KBs are both long-standing goals in Machine Learning. In this paper, we present ReGen, a bidirectional generation of text and graph leveraging Reinforcement Learning to improve performance. Graph linearization enables us to re-frame both tasks as a sequence to sequence generation problem regardless of the generative direction, which in turn allows the use of Reinforcement Learning for sequence training where the model itself is employed as its own critic leading to Self-Critical Sequence Training (SCST). We present an extensive investigation demonstrating that the use of RL via SCST benefits graph and text generation on WebNLG+ 2020 and TekGen datasets. Our system provides state-of-the-art results on WebNLG+ 2020 by significantly improving upon published results from the WebNLG 2020+ Challenge for both text-to-graph and graph-to-text generation tasks. More details at https://github.com/IBM/regen .

## introduction graph representation of knowledge is a powerful tool to capture real-world information where complex relationships between node entities can be efficiently encoded. automatic generation of knowledge bases (kbs) from free-form text and its counterpart of generating semantically relevant text from kbs are both active and challenging research topics. recently, there has been an increased interest in leveraging pretrained language models (plms) to improve performance for text generation from graph, or graph-to-text (g2t) task @xcite . indeed, large plms like t5 @xcite and bart @xcite ) that have been pretrained on vast amount of diverse and variedly structured data, are particularly good candidates for generating natural looking text from graph data. bart-and t5-related models have been employed by top performers in public challenges such as the webnlg+ 2020 challenge @xcite where both graph-to-text and textto-graph (t2g) tasks are offered, under the names rdf-to-text and text-to-rdf (semantic parsing) respectively; rdf stands for resource description framework, a standard for describing web resources. one can notice that more teams entered the competition for the g2t task than for t2g as the latter is a much harder task. best models generally use plms and fine-tune them for the target modality at hand (either graph or text). this is possible by re-framing the t2g and g2t generations as a sequence to sequence (seq2seq) generation problem, which suits fine-tuning plms well. one can therefore hope to leverage the large pretraining of plms to improve the overall generation quality. the seq2seq formulation requires any input graph to be linearized as a sequence, which is not unique. this creates an opportunity for data augmentation where multiple linearizations are provided to the model at training time so the model learns the content represented by the graph, not the order of its sequential representation. in this work, we are interested in leveraging the power of plms for both g2t and t2g generation tasks, and will demonstrate the strength of our approach by improving upon the best results of the webnlg+ 2020 challenge (rev 3.0) as reported by castro ferreira et al. (2020a) for both t2g (semantic parsing) and g2t (data-to-text) tasks. we will also present results for the tekgen corpus @xcite to show performance on a different, much larger dataset. to illustrate the task of generation, fig. 1 provides examples of g2t and t2g outputs obtained using the proposed generation framework. the first two sentences of the abstract of this paper were used as input for t2g using our best model. the model generates a graph from the input text by simultaneously extracting figure 1 : actual examples of generation for text-to-graph and graph-to-text tasks using our best rl models. the first two sentences of the abstract were processed through our best models. first, a graph was created capturing the facts from the input sentences. then, this graph was used as input to generate text. despite a strong domain mismatch between input data and models, the generated paragraph is capturing most of the original sentences content. both models were trained using rl, specifically self-critical sequence training (scst). since both t2g and g2t generative tasks can be formulated as a seq2seq problem, we propose to use reinforcement learning (rl) as part of the plms fine-tuning on the target domain data. for both g2t and t2g tasks, a differentiable function such as the cross-entropy (ce) loss function is often used, since minimizing it results in maximizing the probability of generating the correct token/word. however, when it comes to evaluating a model's performance, benchmarks often use bleu @xcite , meteor @xcite , and chrf++ @xcite for g2t, or simply f1, precision, and recall scores for t2g, none of which being differentiable. during training, one hopes that by minimizing the ce loss, the model will tend towards better prediction of the target tokens, hence improving on evaluation metrics as a beneficial by-product. thankfully, rl provides a framework where we can update our model parameters so to improve evaluation metrics directly. mixed incremental cross-entropy reinforce from ranzato et al. (2016) introduced using reinforce (williams, 1992) for sequence training. we propose to use one of its variant known as self-critical sequence training (scst) @xcite for both t2g and g2t training. in summary, our main contributions are: • we propose to use rl-based sequence training, specifically scst, for both g2t and t2g tasks. this is the first time that rl based training is proposed to the bi-directional generation of text and graph. to the best of our knowledge, the present work is the first time it is introduced for a t2g task. ## related work in the webnlg+ 2020 challenge, most top performing models relied on fine-tuning of plms. interestingly, all four top teams in this challenge proposed quite different approaches while leveraging plms. 1 st place amazon ai @xcite pipelined a relational graph convolutional network (r-gcn) and a t5 plm with some canonicalization rules. 2 nd place osu neural nlg @xcite , the closest to our approach in spirit, used t5 and mbart plms to fine-tune after special data preprocessing. 3 rd place fbconvai @xcite used bart plm and multiple strategies to model input rdfs. 4 th place bt5 employed a t5 plm trained in a bi-lingual approach on english and russian, even using wmt english/russian parallel corpus. recently, @xcite ; @xcite proposed models trained to generate in both t2g and g2t directions, with consistency cycles created to enable the use of unsupervised datasets. in contrast, our approach of fine-tuning a t5 plm is fully supervised but can produce either the specialized models for t2g and g2t tasks alone, or a hybrid model that can handle both t/g inputs simultaneously to generate the corresponding translated g/t outputs. note that in contrast to many webnlg+ 2020 challenge participants, e.g. @xcite , no preprocessing of the data is performed for text, while for graph triples, we add tokens to mark subject, predicate, and object positions in their linearized sequence representation. moreover, data augmentation is performed by allowing random shuffling of triples order in graph linearization to avoid a model to learn the exact order of triples, especially for the t2g task. while the use of rl training in plm has been explored in many works, the approach of @xcite is closest to ours. however, their work focuses on the improved text generation in the context of natural question generation, while in our algorithm we use it for graph-to-text and text-to-graph generations. ## models models are trained on a dataset d composed of a set of (x t , x g ) i samples, where superscript i denotes the i-th sample in d, x t is made of text (one or more sentences), and x g is a corresponding graph represented as a list of triples x @xmath0, p 1 , o 1 ), . . . , (s k , p k , o k )], where the k-th triple is composed of a subject s k , predicate (relationship) p k , and object o k . for g2t, the model is given x g as input and must generate xt . a cross-entropy loss is computed as an expectation: ## experimental setup in this section, we present the experimental setup used for all the results reported in this paper. models we used t5 plms from @xcite for our experiments for two distinct models, t5large (770m parameters) and t5-base (220m parameters), with a special focus on t5-large as it is the best performing of the two on various nlp tasks. models were fine-tuned to be either specialized on t2g (m t ) or g2t (m g ) task, or to accommodate both directions of generation (m t+g ). data processing graphs are often represented as list of triples. however our model expects a sequence of input words/tokens to work on. the linearization of graph triples is obviously ambiguous as there are many ways to traverse a graph (breadth first search, depth first search, random walk, etc.). in practice, we linearize the triples in the order of the list provided by the dataset, but use this inherent linearization ambiguity as an opportunity to do data-augmentation. indeed, models are first fine-tuned using cross-entropy loss that strongly penalizes generation if it is in any different order than the ground truth order. to avoid the model to overfit to our data and memorize observed triples order, we augment the data by including a few permutations of the graph triples. during graph linearization, we encode the subject, predicate, and object positions by @xmath1,<p>,<@xmath2. in practice, we expand the model vocabulary with these special indivisible tokens that are not split during tokenization. no other preprocessing is done on the data for training. we explored masked and span-masked lm fine-tuning to match t5 pretraining @xcite which did not lead to any noticeable improvements. ## results for all experiments, plms were first exposed to the target datasets (webnlg+, tekgen) by finetuning using l ce loss. they were then switched to rl training by optimizing the l scst loss. although no exact recipe has been established for seq2seq rl-training, starting from a good ce model helps rl training performance in practice @xcite . therefore, we followed the subsequent simple approach: during fine-tuning, the evaluations are conducted on the validation set. from the ce phase, the best performing model iteration is selected based on the meteor and f1 score for the g2t and t2g tasks, respectively, to pursue rl fine-tuning. in case of g2t, potential ties in meteor scores among candidate models, are resolved by using bleu_nltk, followed by the chrf++ metric. note that early stopping selection of ce models led to good performance for t5-base models as well. during the scst phase, the best model iteration on the validation set is selected and its performance numbers on the test set are reported in our tables. webnlg+ 2020 g2t for the webnlg+ 2020 challenge, the results of the top four systems for rdf-to-text task can be found in tab. 1 for all categories (results for seen and unseen categories are given in tab. 5 in the appendix), while descriptions the top teams' systems were given in section 2. we report our g2t results for both t5-large and t5base models as well. for t5-large, regen g2t.ce is the best model from ce fine-tuning. regen g2t.rl is best model performance for scst training while using meteor as reward when starting from g2t.ce model. tab. 1 shows that our ce model is better than models from all top teams, and the scst results further improve significantly in webnlg t2g match f1↑ precision↑ recall↑ team/model amazon ai (shanghai) @xcite another interesting fact is that early stopping of ce model g2t.ce.es (at 5 epochs) leads to the best scst model g2t.rl.es for t5-base, while selecting the best ce model g2t.ce.best (at 11 epochs) still showed some gains from scst model g2t.rl.best. scst needs a good starting point, but a better ce model that has seen a lot more epochs of our dataset maybe harder for scst to stir in a better solution in the parameter space. moreover, the test split contains unseen categories not present in the validation dataset which render choices based on validation sub-optimal for the test dataset. the best models we report in this work are specialized models m g . early in our investigation, hybrid models were the best performing model for g2t reaching 0.547 bleu, 0.543 bleu_nltk and 0.417 meteor, and first to beat the challenge winning team. however, when batch size became larger (20-24 samples), the specialized models took the lead and retain it still. for training, we optimized all our models using adamw @xcite , variant of the adam optimizer with default values of β = [0.9, 0.999] and weight decay of 10 -2 . for learning rate, we used 5.10 -6 for all our experiments as it was better than 10 -5 and 10 -6 as seen in tab. 8 in appendix. all our models were trained with 20-24 minibatch size on webnlg. further details on our experimental setup are provided in the appendix in section a. webnlg+ 2020 t2g results for the text-to-rdf task are reported in tab. 2 for all categories. results for our best model on seen and unseen categories are given in tab. 6 in appendix. amazon ai and bt5 are the top performing teams. again, the proposed regen t2g.ce model shows strong results that are better in term of all metrics, for all matching categories. in themselves, these numbers are a de-facto new state-of-the-art for this dataset, as far as we know. scst model t2g.rl fails to improve on this model though. the exact f1 metric was used as reward, but the model could never pull ahead of the ce model in our experiments. the exact f1 metric may not be a strong enough reward to really capture the dynamics of graph generation properly for webnlg+ as it is very rigid in its measure (one must have an exact match), although the same reward gave good results on our second dataset tekgen. a more sensitive metric could possibly help. we even tried to use n-gram based metrics (like bleu) but to no avail. we further address this issue at the end on this section. tekgen g2t for the tekgen dataset, we present our results on graph-to-text generation in tab. 3. similar to the experiments in webnlg+, we pick the best model during the ce fine-tuning based on the meteor score and proceed with the rl fine-tuning. we observe that the rl fine-tuning step helps boost the test split scores on all metrics. it is worth noting that the scores are slightly under- tekgen t2g results for the text-to-graph for tekgen are reported in tab. 4. once the ce finetuning is done, we continue with the rl fine-tuning using exact f1 as reward. the performance is consistent with what we observe in g2t task for tek-gen, where scst step boosts the performance of the model. since, we reformulate this dataset (refer section 4.1) to offer as t2g and g2t tasks, our approach is the first attempt in understanding the nature of tekgen dataset and our methods provide a baseline for future research. please note that for both t2g and g2t tasks in tekgen, we only start a t5-large plm. summary results on webnlg+ 2020 and tek-gen demonstrated that rl fine-tuning of models leads to significant improvements of results for t2g and g2t, establishing new state-of-the-art results for both tasks. for webnlg+, t2g was a challenging task for rl fine-tuning. in further work, we plan to address this issue by investigating two points: first, look into a more sensible graphdependent sampling for graph structures, rather than the current multinomial sampling of the best tokens at each generation step. second, try a different reward schemes where the reward is more attuned to the challenges of graph generation as well as graph structure, allowing for some curriculum learning, or increasing the harshness of rewards gradually during training. results on tekgen showed that rl fine-tuning is a viable option even on large-scale datasets. to enrich this quantitative study of regen, we provide a few qualitative cherry picked results in tab. 11 and tab. 12 in appendix. ## conclusions in this paper, we proposed to use rl for improving upon current generation for text-to-graph and graph-to-text tasks for the webnlg+ 2020 challenge dataset using pre-trained lms. we not only defined a novel seq2seq training of models in t2g and g2t generation tasks, but we established stateof-the-art results for webnlg+ for both tasks, significantly improving on the previously published results. we provided extensive analyses of our results and of the steps taken to reach these improvements. we then expanded our approach to large scale training by means of tekgen where we demonstrated that rl fine-tuning provides a robust way to improve upon regular model finetuning within a dataset that is orders of magnitude larger than the webnlg+ starting point. we established gains despite a weaker content overlap in text-graph data pairs for tekgen. along the way, we constructed subject, and relation-object boundaries from tekgen sentence-triples pairs that we plan on releasing to benefit the research community. future work will focus on developing a variant of scst that leverages the unique structure of graph by either performing of more sensible graphdependent sampling, or by investigating different reward schemes more attuned to integrating the content and structure of graphs.