Evaluating Text Generation from Discourse Representation Structures (2021)

Abstract: We present an end-to-end neural approach to generate English sentences from formal meaning representations, Discourse Representation Structures (DRSs). We use a rather standard bi-LSTM sequence-to-sequence model, work with a linearized DRS input representation, and evaluate character-level and word-level decoders. We obtain very encouraging results in terms of reference-based automatic metrics such as BLEU. But because such metrics only evaluate the surface level of generated output, we develop a new metric, ROSE, that targets specific semantic phenomena. We do this with five DRS generation challenge sets focusing on tense, grammatical number, polarity, named entities and quantities. The aim of these challenge sets is to assess the neural generator’s systematicity and generalization to unseen inputs.

## introduction faithfully generating text from structured representations is an important task in nlp. common tasks include generations from tables @xcite , knowledge graphs @xcite and meaning representations @xcite @xcite . recently, many research efforts have focused on the graph-based semantic formalism abstract meaning representation (amr, @xcite , with approaches based on machine translation @xcite , specialized graph encoders @xcite @xcite @xcite and pre-trained language models @xcite . however, far less attention has been given to generating text from formal meaning representation, such as discourse representation structures (drss). drss are proposed in discourse representation theory @xcite @xcite , a well-studied semantic formalism, covering a wide range of linguistic phenomena. differently from amr, drss explicitly model scope, tense and definiteness. the lack of this information makes amr-to-text challenging @xcite , but their inclusion presents a challenge for the generation methods as well, as they, for example, have to deal with a lot more variables in the representation @xcite . another difference with amr is that drss are in principle language neutral (at least the version of drs that we use in this paper), with gold standard annotations publicly available in four languages @xcite . for these reasons, developing portable and high-quality generation systems for drss is a promising research direction. while there has been some initial work on drsto-text generation @xcite @xcite , most drs-based work has focused on semantic parsing, that is mapping text to drs @xcite @xcite @xcite @xcite . our work has two main contributions. the first is on the modelling side, as we take the first step in drs-to-text generation with neural networks. foot_0 specifically, we use a bi-lstm sequence-to-sequence model that processes linearized drss representations and produces english texts using a character-level decoder (see pipeline in figure 1 ). our second contribution regards the evaluation of the produced text. given the known limitations of reference-based automatic metrics for natural language generation @xcite and in particular for amrto-text @xcite , we design five drs-specific challenge sets @xcite and use them to perform a fine-grained manual evaluation. the general goal of these challenge sets is to assess the robustness of a drs generator with respect to a number of linguistic phenomena. more specifically, we assess (i) generation systematicity with respect to three semantic phenomena (tense change, polarity change, singular↔plural switch), and (ii) generalization to unseen input literals (named entities and quantities). the idea is that by changing the meaning of a drs in a controlled way, robustness of systems can be monitored in detail and assessed accordingly. besides assessing the quality of a generator, these challenge sets also showcase the ease to which drss can be manipulated to express novel meaning combinations. all challenge sets are publicly available. foot_1 ## data and methodology in this section we describe the data and methodology we use for drs generation. first we explain and motivate our representation of drss (input to the nlg system) and the generated text (see figure 1 for a full overview of our source and target representations). then we provide details of our nlg system, which is based on a recurrent neural network, and show how it is trained. ## semantic challenge sets challenge sets are often used in machine translation to assess a model's ability to systematically deal with specific linguistic phenomena that may be infrequent in standard test sets @xcite . following this practice, we created five challenge sets for drss generation that focus on various semantic phenomena (see the resulting semantic challenge sets can be viewed as stress tests: if the generator performs well on these test suites it shows that it can deal with specific semantic phenomena adequately in unforeseen circumstances. we carry out these modifications on subsets of the pmb test data, and we group them into those that assess systematic predictions (tense, polarity, and grammatical number) and those that assess generalisation to unseen input (names and quantities). the specific challenge sets are described in detail below. original tom has three thousand books. ## assessment methods we consider two types of assessment for the generated english sentences. our point of departure are the well-known automatic metrics based on word overlap. we complement these with manual metrics carried out by human experts. ## results and analysis ## conclusion and future work we presented an end-to-end neural approach to generate natural language from discourse representation structures. our model is based on a bi-lstm sequence-to-sequence architecture taking linearized drss as input. comparing character level with word level for producing text, it achieves higher bleu, meteor and rouge scores on the former. for a better understanding of our generator's robustness and its reliability, we designed several challenge sets focusing on specific semantic phe-nomena (tense, polarity, grammatical number) and types of unseen input (quantity and named entities). automatic and manual evaluations on these challenge sets point out to negation as the most challenging phenomenon for drs generation, followed by tense. by contrast, changes in grammatical number and generalizations to unseen quantities or names are well handled by the model. altogether, our results suggest that neural generation from drss is a very promising research direction, but more work is needed to ensure reliability in real-world applications. we hope that our challenge sets will foster future research on this topic and eventually lead to truly robust drs generators. the challenge sets, as we have presented them, can be further refined, and other linguistic phenomena can be added as well. possibilities that spring to mind are challenge sets for pronouns, definite descriptions, comparatives, aspect, and discourse particles. and obviously, we need to generate challenge sets for languages other than english, which might trigger language-specific phenomena as well that could be suitable for challenge sets for drs generation.