Neural NLG for Methodius: From RST Meaning Representations to Texts (2020)

Abstract: While classic NLG systems typically made use of hierarchically structured content plans that included discourse relations as central components, more recent neural approaches have mostly mapped simple, flat inputs to texts without representing discourse relations explicitly. In this paper, we investigate whether it is beneficial to include discourse relations in the input to neural data-to-text generators for texts where discourse relations play an important role. To do so, we reimplement the sentence planning and realization components of a classic NLG system, Methodius, using LSTM sequence-to-sequence (seq2seq) models. We find that although seq2seq models can learn to generate fluent and grammatical texts remarkably well with sufficiently representative Methodius training data, they cannot learn to correctly express Methodius’s similarity and contrast comparisons unless the corresponding RST relations are included in the inputs. Additionally, we experiment with using self-training and reverse model reranking to better handle train/test data mismatches, and find that while these methods help reduce content errors, it remains essential to include discourse relations in the input to obtain optimal performance.

## introduction traditional approaches to the task of natural language generation (nlg) have employed a pipeline of modules, moving from an initial abstract meaning representation (mr) to human-readable natural language @xcite . in the last decade, the success of neural methods in other domains of natural language processing (nlp) has led to the development of neural 'end-to-end' (e2e) ## methodius the methodius system @xcite was developed for multilingual text generation, based on the m-piro project @xcite which focused on museum exhibit descriptions. methodius consists of several components. the content module selects content from a database and creates a content plan, which is a tree where the nodes are labeled with rhetorical relations or facts, following the structures proposed in rst. fig. 1 shows a content plan. the content plan is rewritten into a sequence of logical forms, one per sentence, by the sentence planner. the logical forms are then realized as a text by means of a combinatory categorial grammar (ccg) using openccg @xcite . the methodius system is designed to respond to the behaviour of the its intended users. sequences of exhibits, dubbed 'chains', are constructed while the user moves through the museum. the chains control dependencies between exhibit descriptions, limit redundancy, and provide discourse continuity. while rst defines a number of rhetorical relations, methodius incorporates only four of them: elaboration, joint, similarity and con-trast. elaboration connects the main fact about a focal entity with other, peripheral facts about that entity. joint connects two facts of equal status. similarity and contrast each connect two facts of equal status, but they do opposite jobs: similarity is used to express the similarity of two entities in terms of a commonly shared feature, while contrast is used to show that the values of a shared feature of the given entities differ. for instance, unlike the previous coins you saw, which are located in the athens numismatic museum, this tetradrachm is located in the national museum of athens -here unlike signals contrast. in the following example, like signals similarity: like the previous coins you saw, this tetradrachm is located in the national museum of athens. ## experiments and results we ran self-training experiments with two sets of unlabeled data. one of them consists of the content plans generated by methodius. the other one, dubbed 'heuristic,' is developed from the existing labeled data. the heuristic data is produced by the following method: for every content plan produced by methodius, extract the set of subtrees of the content plan which respect some soft constraints on structure. we avoid extracting trees that start with an optional type. the subtrees are randomly selected but their distribution is required to closely follow the distribution of distinct rst types in the training data. since the size of the methodius data set is limited, the heuristic data set provides useful cheap supplementary content for training (compared to the cost of eliciting text corresponding to content plans through e.g. turkers). we are thus interested whether having genuine methodius content plans, which are not straightforward to generate in large amounts, could be completed by a heuristic data set generated from the labeled training data set. the fact models were trained on the fact versions of the data set, which is obtained by simply deleting the rst structure from the rst data set. 4 we refer to the models (for sake of clarity) by the names in we trained the following models: models were trained over several iterations, though for exposition the results reported below concern just the best model iterations. 5 bleu4 is calculated on both the standard and challenge test sets. bleu4, though limited in the conclusions it supports, seems informative enough to allow one to distinguish between rst and fact models; we report it in appendix d. bleu4 is on average 5 or more points higher for rst models than fact models across the test sets. ## related and future work while traditional natural language generation systems, e.g. methodius, often employ knowledge graphs, the use of such structure in neural nlg is underdeveloped. an exception in this respect is webnlg @xcite , which is a multilingual corpus for natural language generation. an f a c t -l b l f a c t -s m -s t -r m r f a c t -l g -s t -r m r r s t -l b l r s t -s m -s t -v a n r s t -l g -s t -r m r 0 1 2 3 mean errors per item repetitions omissions hallucinations figure 3: challenge set entry of webnlg is a set of rdf triples (representing subj, predicate, object) paired with the corresponding text, which is the sequences of sentences which serve as verbalization of those triples. but it is noteworthy that the main focus in webnlg is micro-planning (sentence-level generation). consequently, webnlg only makes use of a single, implicit rhetorical relation, namely elaboration. elaboration is frequent in the methodius corpus. but methodius uses more interesting rhetorical relations, too, including contrast and similarity, thus the content (both in terms of meaning representations and texts) is significantly different from webnlg. for future work, there are number of direction we intend to explore, including the following: total number of errors figure 4: standard set: errors in generating discourse cue words for similarity and/or contrast (unlike and/or like), where towards errors counts if either there is an incorrectly generated discourse cue word, or there has been a cue word generated while the target has none, or no cue word is generated but the reference contains one. the dotted line links two models if there is a significant difference between their performance in terms of fisher's exact test statistics (we take the significance threshold 5%). ## conclusions the overall conclusion is that including rst relations in the input content plans is necessary to achieve optimum performance in correctly and coherently expressing discourse relations in the neural reimplementation of methodius. this is somewhat surprising since the fact-only inputs actually have all the information necessary to infer that a similarity or contrast relation should be expressed, but the models nevertheless struggle to learn the desired same/different generalization. moreover, the errors are often jarring-they produce genuine incoherence in the text. we see the best performance from the rst model with small but clean self-training data (rst-sm), as it comes from methodius and thus follows the same general patterns as the ones in the test set. the large rst model (rst-lg) had similar performance to the small one. fact models, both small and large, show significant self-training improvements when reranking with reverse models. because the rst baseline already performs relatively well, such an improvement is not observable with them. rst-sm with vanilla self-training already showed high performance. in the case of the fact models, we saw that reranking with reverse models lowers repetitions, omissions and hallucinations in total. it was also beneficial for the rst-lg model. despite the highly regular nature of the rulebased texts, even our best models do not get close to zero content errors, highlighting the importance of continued work on eliminating these errors, e.g. using pretrained models @xcite or constrained decoding @xcite .