Towards Domain-Independent Text Structuring Trainable on Large Discourse Treebanks (2020)

Abstract: Text structuring is a fundamental step in NLG, especially when generating multi-sentential text. With the goal of fostering more general and data-driven approaches to text structuring, we propose the new and domain-independent NLG task of structuring and ordering a (possibly large) set of EDUs. We then present a solution for this task that combines neural dependency tree induction with pointer networks, and can be trained on large discourse treebanks that have only recently become available. Further, we propose a new evaluation metric that is arguably more suitable for our new task compared to existing content ordering metrics. Finally, we empirically show that our approach outperforms competitive alternatives on the proposed measure and is equivalent in performance with respect to previously established measures.

## introduction natural language generation (nlg) plays a fundamental role in data-to-text tasks like automatically producing soccer, weather and financial reports @xcite @xcite , as well as in text-to-text generation tasks like summarization @xcite . generally speaking, nlg involves three key steps @xcite : first there is content determination which selects what information units should be conveyed, secondly there is text structuring, which is responsible for properly structuring and ordering those units; and finally microplanning-realization that aggregates information units into sentences and paragraphs that are then verbalized. the focus of this paper is on the text structuring step, which is critical for the overall performance of an nlg system, as it ensures that the communicative goals of the text are realized in the most structurally coherent and cohesive way possible, making the main ideas expressed by the text easy to follow for the target audience. aiming to develop very general computational methods for text structuring, we keep our study independent from particular ways in which the input information units are represented and from explicitly provided ordering constraints for the target application domain @xcite . more specifically, we propose and attack, in a fully datadriven way, the novel and domain-independent task of simultaneously structuring and ordering a set of elementary discourse units (edus), i.e., clauselike text fragments that the rhetorical structure theory (rst) @xcite assumes to be the building blocks of any discourse structure (see figure 1 (a)(left)). in other words, we assume that the system is given a set of edus (with cardinality @xmath0, as well as the unlabelled rst dependency discourse tree structure for a document consisting of this set of edus, as illustrated in @xcite . our data-driven approach relies on the very recent availability of large treebanks containing hundreds of thousands of (silver-standard) discourse trees that can be automatically generated by distant supervision following the approach presented by @xcite . we formulate the problem as one of the dependency tree induction, repurposing existing solutions @xcite to perform an rst-based text structuring where both edu ordering and tree building are executed simultaneously @xcite . the resulting structures can be highly useful for subsequent nlg pipeline stages such as aggregation, and for downstream tasks like text simplification @xcite . our approach is trainable end-to-end, but since the discourse trees in the training treebank are constituency trees (see figure 1(b) ), we face the additional challenge of turning them into dependency trees (see figure 1(a) ) before the learning process can start @xcite . in a comprehensive evaluation, we compare our solution to three baselines along with a competitive approach based on pointer networks @xcite , which is the established method of choice not only for sentence ordering @xcite , but also for basic domain-specific text structuring in data-to-text applications @xcite . in particular, the comparison involves training and testing the different models on the mega-dt treebank @xcite , containing ≈250,000 discourse trees obtained by distant supervision from a the yelp'13 corpus of customer reviews @xcite . with respect to evaluation metrics, we found the current ways of measuring content ordering (e.g., kendall's τ ) to be inadequate to capture the quality of long sequences of relatively short information units (i.e., sequences of edus of long multi-sentential text). thus, we propose a novel evaluation measure, blocked kendall's τ , that we argue should be used for our new nlg task of ordering and structuring a possibly large set of edus, because it critically measures how well semantically close units are clustered together in the correct order. to summarize the contributions of this paper: (i) we propose the new and domain-independent nlg task involving the structuring and ordering a set of edus, which is intended to enable more general and data-driven approaches to text structuring; (ii) we present a strong benchmark solution for this task, trainable on large discourse treebank, that combines neural dependency tree induction with pointer networks; (iii) we propose a new evaluation metric that is arguably much more suitable for this task than existing ordering metrics; (iv) and on this new metric along with standard tree-quality metrics, we show empirically that our approach outperforms or is comparable to competitive alternatives. the code for our solution and the new metric, as well as the treebank for training, is publicly available. 1 1 http://www.cs.ubc.ca/ cs-research/lci/research-groups/ natural-language-processing/index.html 2 related work (a) text structuring is a key step in nlg, especially when generating long multi-sentential documents. not surprisingly, this is also the case in recent neural approaches. @xcite presented the rotowire corpus, targeting longdocument data-to-text nlg. to generate the document, their model conditions on all records in the data table by weighting their embeddings with attention, in addition to using copying mechanism for out-of-vocabulary data entries. the follow-up work of @xcite , instead of conditioning on all records, arguably performs better text structuring by first selecting and then ordering the entries of a data table using pointer network architecture @xcite . that way, the surface realization module considers previously generated text and only one new data table entry at a time. their model was extended by @xcite , with an additional gru for tracking the entities that the model already referred to in the past. pursuing a rather different approach to improve text structuring, @xcite proposed a hierarchical latent-variable model where the problem is decomposed into dependent sub-tasks, aggregating groups of data table entries into sentences first and then generating the sentences sequentially, conditioned on the plan and already generated sentences. overall, these last three models significantly outperform the initial approach of @xcite both in terms of fluency and coverage, with increasing sophistication of the text structuring module yielding bigger gains, confirming that text structuring is indeed crucial for generating coherent long documents. the task we propose and investigate in this paper can be seen as pushing this line of research even further. we aim for a more ambitious text structuring module inspired by traditional nlg work, viewing the process as the construction of an rst discourse tree for the target document @xcite , which critically includes assigning importance to each constituent. tellingly, our task is also domain-independent and agnostic on the representation of the input information units. the new task we propose in this paper is similar, but more challenging than sentence ordering. instead of ordering sentences, we need to order edus, which are often shorter sentence constituents, and therefore by expressing smaller semantic units they arguably require more finegrained processing. furthermore, our task goes beyond ordering by also requiring the synergistic and simultaneous step of generating the rst discourse structure for the edus. to address these challenges, more powerful techniques for tree induction are needed on top of pointer networks. ## novel task and methods our novel task for text structuring receives as input a set of n edus and returns both an ordering and a discourse structure for that set. we first describe how the edus are encoded, as this is the initial step for all the approaches we consider. then, after discussing a basic method for just ordering the input edus (pointer networks), which will serve as our main baseline, we present our solution for fully solving the task in detail, which combines tree induction with pointer networks. we will refer to our final approach as depstructurer. we conclude the section with two simple baselines for edu ordering and structuring, respectively. ## quantitative and qualitative results results are presented in focusing on the performance of tree induction systems, our depstructurer outperforms the unsupervised model (uti) by a wide margin and has nearly identical performance with ti+lmd model, indicating that a trainable tree induction model is essential to obtain much more accurate trees. lastly, among the unsupervised models, uti outperforms lm across all metrics. this suggests that even without training, forcing a model to generate a tree structure is by itself a useful inductive bias. to highlight the strengths and potential weaknesses of our solution and new metric, we analyze the output of the depstructurer and pointer models for two medium-length illustrative sample documents with 16 and 14 edus respectively (see figures 3 and 4). in each figure, the top row indicates the ordering output of the depstructurer, the middle row is the gold (i.e., correct) ordering, and the bottom is the pointer's output. we color-coded the blocks that each model predicted correctly, with the highlights in the middle gold ordering denoting whether the top or bottom model predicted that block correctly. additionally, for both exam- ## conclusions and future work by proposing the domain-independent task of structuring and ordering a set of edus, we aim to stimulate more general and data-driven approaches for text structuring. the solution we have developed for such task combines neural dependency tree induction with pointer networks, which are both trainable on large discourse treebanks. since existing text ordering metrics are not capturing key aspects of text structuring, we have also proposed a new metric that is arguably much more suitable for the task. in a series of experiments, complemented by qualitative error analysis, we have shown that our solution delivers top performance and represents a promising initial framework for further developments. fruitful directions for future work include: