Text Generation from Discourse Representation Structures (2021)

Abstract: We propose neural models to generate text from formal meaning representations based on Discourse Representation Structures (DRSs). DRSs are document-level representations which encode rich semantic detail pertaining to rhetorical relations, presupposition, and co-reference within and across sentences. We formalize the task of neural DRS-to-text generation and provide modeling solutions for the problems of condition ordering and variable naming which render generation from DRSs non-trivial. Our generator relies on a novel sibling treeLSTM model which is able to accurately represent DRS structures and is more generally suited to trees with wide branches. We achieve competitive performance (59.48 BLEU) on the GMB benchmark against several strong baselines.

## introduction it is not uncommon for text generation systems to produce natural language output from intermediate semantic representations @xcite . the literature presents several examples of generating text from logical forms underlying various grammar formalisms @xcite @xcite , typed lambda calculus @xcite , abstract meaning representations (amr; @xcite @xcite @xcite @xcite @xcite , discourse representation theory (drt; @xcite , and minimal recursion semantics (mrs; @xcite . in this work, we propose neural models to generate high-quality text from semantic representations based on discourse representation structures (drss). drss are the basic meaning-carrying units in discourse representation theory (drt; kamp the man is going to play the piano. tom may stop him. 1981; @xcite , a formal semantic theory designed to handle a variety of linguistic phenomena, including anaphora, presuppositions @xcite , and temporal expressions within and across sentences. drss are scoped meaning representations, they capture the semantics of negation, modals, and quantification. figure 1 displays in box format the meaning representation for a discourse consisting of two sentences. the outermost box is a segmented drs expressing the rhetorical relation contrast between box b 1 representing the first sentence and box b 4 representing the second sentence. boxes b 1 and b 2 are drss, the top layers contain variables (e.g., x 1 , x 2 ) indicating discourse referents and the bottom layers contain conditions (e.g., named(x 3 , "tom")) representing information about discourse referents. variables and conditions have pointers (denoted by b in the figure) pointing to the boxes where they should be interpreted. 1 predicates are disambiguated to their wordnet @xcite senses (e.g., male.n.02 and play.v.03). although there has been considerable activity recently in developing models which analyze text in the style of @xcite @xcite @xcite , attempts to generate text from drss have been few and far between (however see basile 2015 and narayan and gardent 2014 for notable exceptions). this is primarily due to two properties of drs-based semantic representations which render generation from them challenging. firstly, drs conditions are unordered representing a set (rather than a list). 2 a hypothetical generator would have to produce the same output text for any drss which convey the same meaning but appear different due to their conditions having a different order (see figures 1 and 2a which are otherwise identical but the order of conditions in boxes b 1 and b 4 varies). the second challenge concerns variables and their prominent status in drss. variables identify objects in discourse (such as entities and predicates), and are commonly used to model semantic phenomena including coreference, control constructions, and scope. in figure 1 , variables x, e, s, t, p, and b denote entities, events, states, time, propositions and boxes, respectively. variable names themselves are arbitrary and meaningless posing a challenge for learning. our generator must verbalize different variable names to the same surface form. the meaning representations in figures 1 and 2b are identical and both correspond to the same discourse except that the variables have been given different names (b 5 in figure 1 these two problems are further compounded by the way drss are displayed, in a box-like format which is intuitive and easy to read but not convenient for modeling purposes. as a result, drss are often post-processed in a format that can be handled more easily by modern neural network models. for example, drs variables and conditions are converted to clauses @xcite or drss are modified to trees where each box is a subtree and conditions within the box correspond to children of the subtree @xcite . in this paper we propose novel solutions to condition ordering and variable naming. we argue that even though drs conditions appear unordered, they have a latent order due to biases in the way the training data is created. to give a concrete example, the groningen meaning bank (gmb; @xcite provides the largest collection to date of english texts annotated with drss. these annotations were generated with the aid of a ccg parser @xcite ; atomic drs conditions were associated with ccg supertags and then semantically combined following the syntactic ccg derivations. even annotators creating drss manually would be prone to follow a canonical order (e.g., listing named entities first, then verbal predicates and their thematic roles, and finally temporal conditions). we propose a graph-based model which learns to recover the latent order of conditions without explicitly enumerating all possible orders which can be prohibitive. we also handle variable names with a method which rewrites arbitrary indices to relative ones which are in turn determined by the order of conditions. following previous work, we convert drss to a more amenable format. specifically, we consider discourse representation tree structures (drtss; @xcite ) as the semantic representation input to our document generation task, and generate a sequence of words autoregressively. we adopt an encoder-decoder framework with a treelstm @xcite encoder and a standard lstm (hochreiter and schmidhuber, 1997) decoder. problematically, drs trees are wide and the number of children for a given node can be as many as 180. it therefore becomes memory-consuming and sparse to assign a forget gate for each child as in the case of conventional (n -ary) treelstm @xcite . we propose a variant which we call sibling treelstm that replaces n forget gates with a parent gate and a sibling gate. as a result, it reduces memory usage from o(n ) to o(2), and is more suitable for modeling wide and flat trees. our contributions can be summarized as follows: (1) we formalize the task of neural drs-to-text generation; (2) we provide solutions for the problems of condition ordering and variable naming, which render generation from drs-based meaning representations non-trivial; and (3) propose a novel sibling treelstm model that can be also generally used to model wide tree structures. we make our code and datasets publicly available. 3 2 problem formulation let s denote a drs-based meaning representation. the aim of drs-to-text generation is to produce text t that verbalizes input meaning s: our generation model is based on the encoderdecoder framework @xcite and operates over tree structures. moreover, prior to training, variable names are rewritten so that their (arbitrary) indices denote relative order of appearance. we propose a novel sibling treelstm for encoding tree structures. the decoder is a sequential lstm equipped with an attention mechanism generating word sequence @xmath0, t 1 , ..., t m-1 ], where m is the length of the text. at test time, drs conditions are normalized, i.e., they are reordered following a canonical order learned from data, and used as input to our generation model. we first describe our drs-to-tree conversion and variable renaming procedures (sections 2.1 and 2.2). we next present our tree-to-sequence generation model (section 2.3), and explain how drs conditions are ordered (section 2.4). ## experiments our experiments were carried out on the groningen meaning bank (gmb; @xcite ) which provides a large collection of english documents annotated with drss. we used the standard training, development, and test splits that come with the distribution of the corpus. all drss in the gmb were preprocessed into the tree-based format discussed in section 2.1. we also extracted from the training data conditions and their order for training our graph ordering model. dataset statistics are shown in ## related work much previous work has focused on text generation from formal representations of meaning focusing exclusively on isolated sentences or queries. the literature offers a collection of approaches to generating from amrs most of which employ neural models and structured encoders @xcite @xcite @xcite @xcite . other work generates text from structured query language (sql) adopting either sequence-to-sequence @xcite or graph-to-sequence models @xcite . @xcite was the first to attempt generation from drt-based meaning representations. he proposes a pipeline system which operates over graphs and consists of three components: an alignment module learns the correspondence between surface text and drs structure, an ordering module determines the relative position of words and phrases in the surface form and a realizer generates the final text. narayan and gardent ( 2014 temporal information marked in blue, rhetorical relations marked in red, co-reference marked as green, and scope marked in brown. our work is closest to @xcite ; we share the same goal of generating from drss, however, our model is trained end-to-end and can perform long-form generation for documents and sentences alike. we also adopt an ordering component, but we order drs conditions rather than lexical items, and propose a model capable of inferring a global order. there has been long-standing interest in information ordering within nlp @xcite @xcite @xcite @xcite . our innovation lies in conceptualizing ordering as a graph scoring task which can be further realized with graph neural network models @xcite . ## conclusions in this paper, we have focused on document-level generation from formal meaning representations. we have adopted drt as our formalism of choice and highlighted various challenges associated with the generation task. we have introduced a novel sibling treelstm for encoding drss rendered as trees and shown it is particularly suited to trees with wide branches. we have experimentally demonstrated that our encoder coupled with a graph-based condition ordering model outperforms strong comparison systems. in the future, we would like to embed our generator in practical applications such as summarization and question answering.