Neural Methodius Revisited: Do Discourse Relations Help with Pre-Trained Models Too? (2021)

Abstract: Recent developments in natural language generation (NLG) have bolstered arguments in favor of re-introducing explicit coding of discourse relations in the input to neural models. In the Methodius corpus, a meaning representation (MR) is hierarchically structured and includes discourse relations. Meanwhile pre-trained language models have been shown to implicitly encode rich linguistic knowledge which provides an excellent resource for NLG. By virtue of synthesizing these lines of research, we conduct extensive experiments on the benefits of using pre-trained models and discourse relation information in MRs, focusing on the improvement of discourse coherence and correctness. We redesign the Methodius corpus; we also construct another Methodius corpus in which MRs are not hierarchically structured but flat. We report experiments on different versions of the corpora, which probe when, where, and how pre-trained models benefit from MRs with discourse relation information in them. We conclude that discourse relations significantly improve NLG when data is limited.

## introduction the success of neural methods in numerous subfields of nlp lead to recent development of neural 'end-to-end' (e2e) architectures in natural language generation (nlg) @xcite , where a direct mapping from meaning representations (mrs) to text is learned. while recent neural approaches mostly map flat inputs to texts without representing discourse level information explicitly within mrs, @xcite argues that discourse relations should be reintroduced into neural generation, echoing what has been long argued in more traditional approaches to natural language processing where discourse relations play one of the central roles in natural language text understanding and generation @xcite @xcite . to study whether discourse relations are beneficial for neural nlg, @xcite proposed the methodius corpus, which was developed as an experiment in recreating the classic rulebased nlg system methodius @xcite using a neural generator. in their corpus, the meaning representation (mr) of a text is a tree that encodes the overall discourse structure of the texts plus facts related by discourse relations therein. they were concerned with whether explicit encoding of discourse relations improves the quality of generated texts by lstm recurrent neural networks @xcite . however, they left open the question whether discourse relations are helpful for pre-trained transformer-based @xcite language models @xcite , which have recently shown remarkable performance on nlg tasks. in this work, we address that question using the t5-large implementation of @xcite . a particularly attractive quality of pre-trained models is their ability to generalize from limited data. for example, @xcite proposed to fine-tune a model pre-trained on a large nlg corpus using a small amount of labeled data from a specific domain to adapt the model to generate texts in that domain. in a similar vein, when the labeled data is limited, @xcite suggest to use a large pre-trained model with self-training and knowledge distillation to smaller, faster models. @xcite argue that pre-trained language models make it possible to transform a sequence of semantically correct, but (possibly) ungrammatical template-based texts into a natural sounding, felicitous text of english. they find that template-based textual input is beneficial to use with pre-trained language models when the model needs to generalize from relatively few examples. given these considerations, we cannot answer the question whether it is helpful to include discourse relations in the input to a pre-trained model for nlg without considering the form of the input, the size of the training data, and the extent to which the test data goes beyond what has been seen in training. as such, we conduct experiments using several versions of the methodius corpus, where these versions possess one or more of the following properties: (a) discourse relations included in the mr; (b) discourse relations excluded from the mr; (c) tree-structured mr (a hierarchically structured representation of the meaning); (d) flat, textual mr (i.e., non hierarchically structured). we are furthermore concerned with how the linguistic knowledge encoded in pre-trained language models interacts with the different versions of the corpus. we want to be able to scrutinize the structure of the outputs, i.e., texts, too since our intention is to check the models' capabilities in realizing particular phenomena. for these purposes, we conduct experiments using the following setup: (1) use various portions of the labeled data. (2) train zero-shot models (with respect to certain discourse-related phenomena) together with various few-shot models (with respect to the same phenomena). (3) test various aspects of generated texts, both with respect to discourse structure congruence and correctness (factual information). ## re-lexicalized & flat versions of methodius the methodius system @xcite uses discrete rules to generate texts containing predefined subtexts, such as descriptions of exhibits and historical facts about certain periods. to avoid data sparsity and long sequences, stevens-guille et al. (2020) delexicalize texts as they substitute certain parts of text by tokens, which they dub special terminals. we want to take advantage of pre-trained language models, which are not exposed to these tokens. tokens should be substituted by text wherever possible to ensure the input is consistent with the texts the pre-trained models were trained on. however, using some predefined morpho-syntactic constructions and lexical items makes it more manageable to check whether a model performs well with respect to automatic checks. moreover, if the model experiences problems on such data, it suggests the model would have problems with even less homogeneous data. instead of training the models directly on the methodius corpus or texts harvested through crowdsourcing, we modify the methodius corpus (i.e., mrs paired with texts) by substituting custom homogeneous texts for the methodius corpus's special terminals. we substitute some predetermined names for named entities in the methodius corpus to further homogenize the inputs. this procedure deterministically rewrites the texts in the corpus of stevens-guille et al. (2020) into pure english texts and thus maintains the homogeneity of the methodius corpus. the corresponding mrs are also rewritten into their lexicalized versions. foot_0 moreover, we transform rhetorical structure theory @xcite style hierarchically structured meaning representations of methodius texts into a flat, textual input by translating every fact and every discourse relation into a sequence of sentences. figure 1 shows an mr from the methodius corpus, the corresponding text from the methodius corpus, and the new mr that we have substituted for the methodius corpus mr. ## models: rststruct, factstruct, rstt2t, and factt2t we fine-tune t5-large @xcite on the following types of labeled data: we dub the result factt2t. -lexicalized mr, and its flat, textual version, together with the surface realization (a) delexicalized meaning representation from methodius corpus [__content_plan [__rst_elaboration [__fact_type [__arg1 entity0 ] [__arg2 statue ] ] [__rst_joint [__fact_made_of [__arg1 entity0 ] [__arg2 material_0 ] ] [__fact_exhibit_portrays [__arg1 entity0 ] [__arg2 god_0 ] ] ] ] [__rst_contrast [__fact_creation_period compare_additive [__arg1 entity1 ] [__arg2 historical_period_0 ] ] [__fact_creation_period [__arg1 entity0 ] [__arg2 historical_period_1 ] ] ] [__optional_type [__arg1 entity1 ] [__arg2 vessel ] ] ] (b) lexicalized meaning representation of the foregoing (we treat tokens of the form '[ xyz' and ']' as indivisible tokens in our experiments) [__content_plan [__rst_elaboration [__fact_type [__arg1 entity0 ] [__arg2 statue ] ] [__rst_joint [__fact_made_of [__arg1 entity0 ] [__arg2 bronze ] ] [__fact_exhibit_portrays [__arg1 entity0 ] [__arg2 apollo ] ] ] ] [__rst_contrast [__fact_creation_period compare_additive [__arg1 entity1 ] [__arg2 classical period ] ] [__fact_creation_period [__arg1 entity0 ] [__arg2 hellenistic period ] ] ] [__optional_type [__arg1 entity1 ] [__arg2 vessel ] ] ] ## evaluation methods we adopt the double test set style from stevens-guille et al. (2020). one test set is called standard and the other is called challenge (see ## results below, we report results on the data portions 1%, 3%, 5%, 10%, 20%, and 50% of the few-shot models (results on the corresponding zero-shot data models are provided in appendix a.4). for 100% data usage, we report results of zero-, few-, and ten-shot models. ## discussion and conclusion the development of neural nlg led to an understandable focus on simpler phenomena; the networks in currency at the time seemed to perform best on short, entity-focused texts. while new methods frequently make progress by working on simple domains, we echo the conclusions of stevens-guille et al. (2020) that neural methods can and should address more complex, rhetorically structured text, which they must if they are to produce genuinely coherent discourses @xcite . our results here bolster those conclusions and provide further evidence for the usefulness of explicit discourse coding in the input to neural systems, especially when data is limited in size. in line with the contemporary wisdom concerning pre-trained models, our results suggest that finetuning such models when labeled data for specific domains is limited improves the felicity of generated texts. while increases in available data do always improve the quality of generated texts in terms of grammatically and correctness, we see fast and dramatic improvements when using text inputs, with only more gradual increases in quality in the case of structured input. but we stress that dis-course relations are enormously helpful when the dataset for the domain is limited: at lower levels of data usage, rstt2t consistently significantly outperforms factt2t on every metric we use. given the benefits of explicitly encoding discourse relations in the input to the models reported here, we conclude by recommending the continued development of nlg corpora in which discourse relations are present in the meaning representations. for today, even though various corpora have been designed for natural language generation purposes, corpora with discourse structure information are not available. given our results showing the benefits of having discourse information in the input, we hope that more corpora will be designed where discourse information is provided with the help of discourse relations. 1% 3% 5% 0 5 10 15 data portions 1%, 3%, and 5% mistid errors factstruct rststruct factt2t rstt2t figure 10: few shot models: mistid errors on the challange set 10% 20% 50% 0 10 20 data portions 10%, 20%, and 50% mistid errors factstruct rststruct factt2t rstt2t figure 11: few shot models: mistid errors on the challenge set z-100% f-100% d-100% 0 5 10 15 data portions z-100%, f-100%, and d-100% mistid errors factstruct rststruct factt2t rstt2t figure 12: 100% models: mistid errors on the challenge test 1% 3% 5% 0 50 100 150 200 data portions 1%, 3%, and 5% errors in discourse relations factstruct rststruct factt2t rstt2t