Data-to-Text Generation with Iterative Text Editing (2020)

Abstract: We present a novel approach to data-to-text generation based on iterative text editing. Our approach maximizes the completeness and semantic accuracy of the output text while leveraging the abilities of recent pre-trained models for text editing (LaserTagger) and language modeling (GPT-2) to improve the text fluency. To this end, we first transform data items to text using trivial templates, and then we iteratively improve the resulting text by a neural model trained for the sentence fusion task. The output of the model is filtered by a simple heuristic and reranked with an off-the-shelf pre-trained language model. We evaluate our approach on two major data-to-text datasets (WebNLG, Cleaned E2E) and analyze its caveats and benefits. Furthermore, we show that our formulation of data-to-text generation opens up the possibility for zero-shot domain adaptation using a general-domain dataset for sentence fusion.

## introduction data-to-text (d2t) generation is the task of transforming structured data into a natural language text which represents it @xcite . the output text can be generated in several steps following a pipeline, or in an end-to-end (e2e) fashion. neural-based e2e architectures recently gained attention due to their potential to reduce the human input needed for building d2t systems. a disadvantage of e2e architectures is the lack of intermediate steps, which makes it hard to control the semantic fidelity of the output @xcite . we focus on a d2t setup where the input data is a set of rdf triples in the form of (subject, predicate, object) and the output text represents all and only facts in the data. this setup can be used by all d2t applications where the data describe relationships between entities (e.g. @xcite . 1 in order to combine the benefits of pipeline and e2e architectures, we propose to use the neural models with a limited scope. we take advantage of three facts: (1) each triple can be lexicalized using a trivial template, (2) stacking the lexicalizations one after another tends to produce an unnatural sounding but semantically accurate output, and (3) the neural model can be used for combining the lexicalizations to improve the output fluency. in traditional pipeline-based nlg systems @xcite , combining the lexicalizations is a non-trivial multi-stage process. text structuring and sentence aggregation are first used to determine the order of facts and their assignment to sentences, followed by referring expression generation and linguistic realization. we argue that with a neural model, combining the lexicalizations can be simplified as several iterations of sentence fusion-a task of combining sentences into a coherent text @xcite . our contributions are the following: ## background improving the accuracy of neural d2t approaches has attracted a lot of research interest lately. similarly to us, other systems use a generatethen-rerank approach @xcite or a classifier to filter incorrect output @xcite . @xcite split the d2t process into a symbolic text-planning stage and a neural generation stage. other works improve the robustness of the neural model @xcite or employ a natural language understanding model @xcite to improve the faithfulness of the output. recently, chen et al. (2020) finetuned gpt-2 @xcite for a few-shot domain adaptation. several models were recently applied to generic text editing tasks. lasertagger @xcite , which we use in our approach, is a sequence tagging model based on the transformer @xcite architecture with the bert @xcite pre-trained language model as the encoder. other recent text-editing models without a pre-trained backbone include editnts @xcite and levenshtein transformer @xcite . concurrently with our work, kale and rastogi (2020) explored using templates for dialogue response generation. they use the sequence-tosequence t5 model @xcite to generate the output text from scratch instead of iteratively editing the intermediate outputs, which leaves less control over the model. ## our approach we start from single-triple templates and iteratively fuse them into the resulting text while filtering and reranking the results. we first detail the main components of our system and then give an overall description of the decoding algorithm. ## analysis of results we compute the metrics used in the evaluation of the e2e challenge @xcite : bleu @xcite , nist @xcite , meteor @xcite , rouge l @xcite and cider @xcite . the results are shown in accuracy vs. variability our approach ensures zero entity errors, since the entities are filled ver-5 https://github.com/ufal-dsg/tgen batim into the templates and in case an entity is missing in the whole beam, a fallback is used instead. semantic inconsistencies still occur, e.g. if a verb or function words are missing. the fused sentences in the e2e dataset, where all the objects are related to a single subject, often lean towards compact forms, e.g.: aromi is a family friendly chinese coffee shop with a low customer rating in riverside. on the contrary, the sentence structure in webnlg mostly follows the structure from the templates and the model performs minimal changes to fuse the sentences together. see templates on average, there are 12.4 templates per predicate in webnlg and 8.3 in the e2e dataset. in cases where the set of templates is more diverse, e.g. if the template for the predicate country has to be selected from {<@xmath0, <@xmath1, lmscorer helps to select the semantically accurate template for the specific entities. the literal copying of entities can be too rigid in some cases, e.g. atatürk monument ( i̇zmir) is made of "bronze", but these disfluencies can be improved in the fusion step. reordering lasertagger does not allow arbitrary reordering of words in the sentence, which can limit the expressiveness of the sentence fusion model. consider the example in figure 1 : in order to create a sentence english is spoken in dublin, the capital of ireland, the model has to delete and re-insert at least one of the entities, e.g. english, triples (albert jennings fountain, deathplace, new mexico territory); (albert jennings fountain, birthplace, new york city); (albert jennings fountain, birthplace, staten island) step #0 albert jennings fountain died in new mexico territory. step #1 albert jennings fountain, who died in new mexico territory, was born in new york city. step #2 albert jennings fountain, who died in new mexico territory, was born in new york city, staten island. reference albert jennings fountain was born in staten island, new york city and died in the new mexico territory. ## future work although the current version of our approach is not yet able to consistently produce sentences with a high degree of fluency, we believe that the approach provides a valuable starting point for controllable and domain-independent d2t generation. in this section, we outline possible directions for tackling the main drawbacks and improving the results of the model with further research. building a high-quality sentence fusion model, which lies at the core of our approach, remains a challenge @xcite . our simple extractive approach relying on existing d2t datasets may not produce sufficient amount of clean data. on the other hand, the phenomena covered in the discofuse dataset are too narrow for the fully general sentence fusion. we believe that training the sentence fusion model on a larger and more diverse sentence fusion dataset, built e.g. in an unsupervised fashion @xcite , is a way to improve the robustness of our approach. fluency of the output sentences may be also improved by allowing more flexibility for the order of entities, either by including an ordering step in the pipeline @xcite , or by using a text-editing model that is capable of explicit reordering of words in the sentence @xcite . splitting the data into smaller batches (i.e. setting an upper bound for the number of sentences fused together) could also help to improve the con-sistency of results with a higher number of data items. our string matching heuristic is quite crude and may lead to a high number of fallbacks. introducing a more precise heuristic, such as a semantic fidelity classifier @xcite , or a model trained for natural language inference @xcite could help to promote lexical variability of the text. finally, we note that the text-editing paradigm allows to visualize the changes made by the model, introducing the option to accept or reject the changes at each step, and even build a set of custom rules on top of the individual edit operations based on the affected tokens. this flexibility could be useful for tweaking the model manually for a production system. ## conclusions we proposed a simple and intuitive approach for d2t generation, splitting the process into two steps: lexicalization of data and improving the text fluency. a trivial lexicalization helps to promote fidelity and domain independence while delegating the subtle work with language to neural models allows to benefit from the power of general-domain pre-training. while a straightforward application of this approach on the webnlg and e2e datasets does not produce state-of-the-art results in terms of automatic metrics, the results still show considerable improvements above the baseline. we provided insights into the behavior of the model, highlighted its potential benefits, and proposed the directions for further improvements.