Generating Discourse Connectives with Pre-trained Language Models: Conditioning on Discourse Relations Helps Reconstruct the PDTB (2022)

Abstract: We report results of experiments using BART (Lewis et al., 2019) and the Penn Discourse Tree Bank (Webber et al., 2019) (PDTB) to generate texts with correctly realized discourse relations. We address a question left open by previous research (Yung et al., 2021; Ko and Li, 2020) concerning whether conditioning the model on the intended discourse relation—which corresponds to adding explicit discourse relation information into the input to the model—improves its performance. Our results suggest that including discourse relation information in the input of the model significantly improves the consistency with which it produces a correctly realized discourse relation in the output. We compare our models’ performance to known results concerning the discourse structures found in written text and their possible explanations in terms of discourse interpretation strategies hypothesized in the psycholinguistics literature. Our findings suggest that natural language generation models based on current pre-trained Transformers will benefit from infusion with discourse level information if they aim to construct discourses with the intended relations.

## introduction traditional approaches to discourse have shown the essential importance of discourse (rhetorical) relations in providing coherence to a text @xcite @xcite . while current approaches to natural language generation (nlg) employing pre-trained models have been shown to excel in generating well-formed texts @xcite , their ability to produce coherent texts structured with the help of discourse connectives is understudied @xcite . the impetus for the present study is the growing body of evidence that neural models, whether trained fresh @xcite or pre-trained @xcite , benefit from input which includes specific reference to the discourse structure intended to hold in the output text @xcite . this line of work is novel in the context of current nlg practice, which frequently omits cues to discourse structure in the input. the previous work is purposefully restricted to producing relatively homogeneous texts (museum descriptions and weather predictions). given the findings of this work on generating limited sets of discourse relations and connectives, it is informative to study the performance of neural models in generating texts structured with the help of a richer set of discourse relations realized by a wide variety of discourse connectives. we study whether having discourse relation information in the input helps neural models to realize the intended discourse relation. these conditions more closely approximate the context in which robust nlg systems would be deployed. we expect our results to provide insight into whether and how to include discourse structure cues in fully-fledged nlg systems. we report the results of our experiments using bart @xcite and the penn discourse tree bank @xcite (pdtb) to generate texts with correctly realized discourse relations. we address a question left open by previous research @xcite concerning whether conditioning the model on the intended discourse relation-which corresponds to adding explicit discourse relation information into the input to the model-improves its performance. while we recognize that a positive answer to this question might seem obvious, it has, to date, not been supported with quantitative evidence. we compare our models' performance to baselines in which i) connective choice is determined by the most frequent connective which realizes the intended relation in the corpus, (ii) connec-tive choice is determine by the most frequent connective in the corpus irrespective of the intended relation to be expressed, (iii) connective choice is determined by off-the-shelf bart-large mask substitution, and (iv) connective choice is determined by off-the-shelf bert @xcite mask substitution. we propose two types of models by fine-tuning bart on pdtb: models that have discourse relation information in the input (d+ models) and models that do not (d-models). we find that our fine-tuned d+ models substantially outperform fine-tuned d-models, while both kinds of fine-tuned models dramatically beat the baselines. in addition, fine-tuned d+ models produce systematically fewer errors than corresponding dones when tested against psycholinguistic observations that certain discourse relations tend to be realized implicitly, while others usually are realized by explicit (overt) discourse connectives. it is important to also point out that our fine-tuned models, unlike previous work and some of our baselines, are not given the position into which the connective should be inserted. this more closely approximates the intended usage of end-to-end neural models, where there is no module in which connectives are slotted into predetermined positions in the output string. we find the models' choices for connective positions to be qualitatively good and focus in the sequel on the connective choices themselves. 1 2 background bart, a transformer-based @xcite language model, is trained on purposefully corrupted data so that the model learns to 'denoise' the corrupted input in the process of reconstructing the original data. fine-tuning bart on different versions of input and output lets us probe whether the underlying language model needs or benefits from explicit cues to consistently reconstruct the intended discourse connective. the pdtb is one of the few corpora developed to identify discourse dependencies in texts. pdtb provides a welldeveloped ontology of discourse relations; these discourse relations are used to annotate the wall street journal (wsj) corpus of the penn treebank. we construct versions of the corpus differing in (i) whether the order of the arguments in the output is explicitly encoded in the input, (ii) whether the output is the connective or the connective embedded in the corresponding wsj text, and (iii) whether a discourse relation is included in the input and how specific it is. the third difference is conceptually the most important one since it corresponds to whether the model is conditioned on discourse relation information. to determine how well the models realize discourse relations, in addition to standard metrics (i.e., recall and precision), we employ more recent metrics inspired by psycholinguistic @xcite @xcite and corpus studies @xcite @xcite ) which allow us to find out the degree to which the models' preferences for realizing different discourse relations correspond to reported human preferences for realizing those relations. in particular, it is argued that while some discourse relations are mostly expressed explicitly, by means of a discourse connective (i.e., overt lexical item or items), other discourse relations tend to be expressed implicitly, i.e., without explicit lexical markers. one of the questions we want to answer is whether providing a discourse relation in the input helps models to learn when to realize a discourse relation explicitly and/or implicitly. @xcite argue that the pdtb provides ample evidence for psycholinguistic patterns of behaviour. in lieu of directly running human judgement experiments on our model outputs, we test our models' consistency with psycholinguistic results indirectly: we compare the distributions in model outputs to those distributions in the corpus which have been argued to support psycholinguistic theories. we focus on the following two hypotheses: the continuity hypothesis: 'readers have a bias towards interpreting sentences in a narrative as following one another in a continuous manner . . . additive additive and causal connectives should lead to less processing facilitation than adversative connectives because the former indicate continuity in the discourse whereas the adversatives indicate discontinuity.' -murray @xcite the causality-by-default hypothesis: 'because readers aim at building the most informative representation, they start out assuming the relation between two consecutive sentences is a causal relation . . . subsequently, causally related information will be processed faster, because the reader will only arrive at an additive relation if no causal relation can be established.' @xcite asr and demberg (2012) report that in the pdtb, relations that they consider discontinuous and continuous in the sense of @xcite are more likely to be realized explicitly and implicitly, respectively, which is consistent with the continuity hypothesis. furthermore, they find the proportion of implicit to explicit connectives is highest for causal senses. they conclude this provides support for the hypothesis of causality-by-default. asr and demberg (2013) propose a metric for deriving 'markedness'-how much information about the intended discourse relation is conveyed by a connective or set of connectives-from the pdtb. however, no prior work on the pdtb conditioned models on the discourse relation intended to be communicated. to our knowledge, the following questions, which we report on here, are yet to explored in nlg: (i) whether conditioning on discourse relations improves the prediction of intended discourse connectives; (ii) whether ordering information concerning the arguments to be expressed should be encoded explicitly; and (iii) whether neural models can learn distributions consistent with psycholinguistic results @xcite known to be reflected in the training set @xcite . ## methods our experiments use the bart-base and bartlarge implementations of huggingface fine-tuned on different versions of the reconstructed pdtb corpus, which we describe in the sequel. 2 the corpus is a modified version of the wsj texts derived from reconstructing the texts from the string positions provided by the pdtb. our modifications were intended to make the reconstructions more natural for pre-trained models by using full sentences but without giving away hints to connective location by capitalization or punctuation. an example is provided in figure 1 . 3 the input consists 2 we find little difference in the performance of bartbase and bart-large and therefore focus on bart-base throughout the paper. results on matching for bart-large can be found in appendix g. to determine whether the order of arguments in the pdtb affects the model's choice of connective, we further divided the corpus into versions which included or didn't include explicit encoding of the order of arguments in the output: '12' encodes the case where the first argument precedes the second argument in the reference text, while '21' corresponds to the second argument preceding the fist argument in the reference text. this is in principle useful since the order of arguments in the input need not reflect their order in the output. to control for the influence of generating full texts, we produced versions of the corpus in which the outputs were the discourse connectives without the surrounding wsj text. since whether the discourse connective is left implicit or made explicit is something we would like to test every model on, we include no information about whether the connective should be implicit or explicit in the input. the difference between bart d+ and bart d-, corresponding to whether the input includes the discourse connective's type, to sentences. the string 'none' is inserted into empty contexts. if the pdtb string indices do not correspond to sentences, the context arguments correspond to the sentences in which the pdtb string indices were embedded. the board isn't proposing a slate of its own and the other four current directors don't want to serve beyond the special meeting date, newport said. expansion_conjunction 12 none the board isn't proposing a slate of its own none none the other four current directors don't want to serve beyond the special meeting date ,newport said a wjs text with arguments and discourse connective highlighted the input constructed from the wjs corpus and its pdtb annotation discourse relation order of arguments argument 1 left context argument 1 argument 1 right context argument 2 left context argument 2 argument 2 right context the board isn't proposing a slate of its own and the other four current directors don't want to serve beyond the special meeting date, newport said. rounds out the set of distinctions between corpus versions. details of the corpus split into train/dev/test can be found in appendix a. ## metrics in order to study whether the models can reconstruct the discourse connectives found in the wsj, we report model-reference matching. we further consider matching with respect to implicit and explicit discourse connectives. implicit and explicit matching, both independently and summed, is our first metric. second, we consider the proximity of the mismatches between reference and generated connective in terms of the pdtb sense hierarchy. with respect to figure 1 , matching would be producing and, either in the embedded sentence in which it occurs (full-output) or on its own (conn-only). since the type of and in this context is expan-sion_conjunction, the mismatch could be by substitution of some other connective from expan-sion_conjunction (=full), from one of the subsenses of expansion (=level 1), or from some completely different sense (=level 2). we test the consistency of the models with the continuity and causality-by-default hypotheses by reference to the metrics proposed by @xcite to quantify the support for such hypotheses in the pdtb. the usefulness of the distribution of implicit versus explicit connectives is helpfully summed up by asr and demberg (2012, pg. 2671): "if readers have a default preference to infer a specific relation in the text, this type of relation should tend to appear without explicit markers." this likewise motivates our use of the metric of markedness, to be discussed below, since markedness quantifies how expected a relation is and, in conjunction with the hypothesis of uniform information density (uid) @xcite , how likely it is to be explicitly cued versus left to be inferred @xcite . @xcite propose to define implicitness of pdtb senses in terms of the distribution of implicit discourse relations corresponding to the sense in the corpus (# of implicit tokens of senses divided by # of tokens of senses). following asr and demberg (2012, 2013); jin and de marneffe (2015b), we focus on the two groups of (sub)types in following asr and demberg (2013); jin and de marneffe (2015b), we make use of a metric of markedness, which asr and demberg (2013) argue provides a good picture of how likely a given relation is to appear with a connective and to what degree the relation-connective co-occurrence is unique: the higher the markedness, the more likely the relation is to appear with a set of specific connectives. consequently, it would be more surprising to have that relation cued by a less expected connective or no connective at all-we should then expect both causal and continuous relations to have lower markedness. asr and demberg (2013) report the markedness of level 1 relations, finding the gross cline of @xmath0. they argue these results are consistent with the uid, the continuity-by-default hypothesis, and the causality-by-default hypothesis. we consider the degree to which the markedness cline of our model outputs corresponds to the markedness cline of the corpus to provide evidence of whether the model is learning to produce text consistent with the previously mentioned cognitive biases. markedness is defined in the equation below, where npmi is normalized point-wise mutual information, r belongs to the set of relations r, and c belongs to the set of connectives c minus the null connective. ## results with respect to the types of fine-tuning we experimented with, we find bart d+ models routinely exceed bart d-models. we show here that bart d+ models seem to recover even some of the distributions found by @xcite to support psycholinguistic results concerning discourse structure. 6 in there seems to be a sweet spot in the level of discourse relation type included in the input: there is little improvement between full and level 2 types 6 the chosen connective need not occur directly between the arguments in the input. determining which connective is produced by the full-out model is done by iteratively substituting elements of the input found in the output with the empty string. once this process is complete the remaining strings will include the connective. strings which are not in the complete set of connectives are removed to eliminate noise. if no connective is found after this process then the model evidently chose to leave the relation implicit. 7 note that the mask position for implicits is uniformly between the rightmost position of arg1 and the leftmost position of arg2. we chose this position for uniformity in light of the absence of implicit connective span annotation in the pdtb. but there is greater improvement between level 2 and level 1-for conn-only the bart d-models even sometimes exceed the level one bart d+ models, suggesting the top level type information could hinder connective choice when the connective isn't generated in context. despite the boost to connective matching when producing conn-only, the distinction between models which condition on the order of arguments versus those that do not, controlling for other corpus distinctions, is minimal when present. this, too, is visible in the major difference between models with respect to reconstructing the reference connectives is the difference between the bart d+ and bart d-models. the bart d+ models from the second level to the full level outperform the bart d-models when controlling for the input order, whether the output is full-output or conn-only. in the sequel we report significance results just for the best bart d-model and a corresponding bart d+ model: bart d- (-order,+fulloutput) and bart d+ (depth 3,-order,+fulloutput). while the level 2 bart d+ model ekes out the level 3 model, the difference is uninteresting. the main distinction in matching between bart d+ and bart d-models is due to explicit connectives. both models perform well with respect to reconstructing implicit connectives, though the differences even here are significant, with the bart d-model even improving over the bart d+ model with respect to implicit relations. however, this observation points to a more likely story for bart d-'s performance: the bart dmodel is less accurate. this is corroborated by it generating an excess of 405 implicits for target explicits compared to bart d+ 's 275 implicits for target explicits. the overproduction of implicits is further borne out by the differences in f1 shown model explicit match implicit match bart d+ 69.8% 89.2% bart d54.3% 90.6% in figures 19 and 20 in appendix d; errors concerning connective choice are exemplified and discussed there too. returning to the production of explicit connectives, the improvements of conditioning on discourse structure information are highly significant both with respect to matching per se and with respect to matching explicit connectives. we provide mcnemar's test statistics for explicit matches (@xmath1.00, @xmath2.000), implicit matches (@xmath3.00, @xmath4.025), and their combination (@xmath5.000, @xmath6.00) (the tables can be found in appendix b). with respect to producing matching explicit connectives, the bart d+ model exceeds the bart d-model on every top level type. when bart d+ doesn't produce a matching explicit connective, it is far more likely to produce an explicit connective which expresses the same relation. for each top level type, the severity of the mismatch is less for bart d+ than bart d-. without committing to the position that producing an implicit connective for a relation intended to be expressed explicitly is better or worse than producing an explicit connective for a relation intended to be expressed implicitly, we argue that either mismatch is better than producing a connective which is never used to express the intended relation. on this score, the bart d-model is considerably worse-it is consistently more likely to produce a connective not otherwise used to cue the intended discourse relation. when the metrics are extended to include whether non-matching connectives chosen by the model fit the intended discourse relation, the bart d+ model continues to outperform the best bart d-model. when producing non-matching connectives, we find that the chosen connectives of the bart d+ model correspond to the intended discourse relations more frequently than those produced by the bart d-models. we computed markedness scores for outputs of the bart d+ and bart d-models. by applying ar significance tests on markedness scorebased statistics, we find that the bart d+ output on the test data is significantly closer to the reference data than the output of bart d-. we show in for several discourse relations. we should first note that the bart d+ markedness scores are consistently closer to the reference scores than the bart d-scores. second we note that the continuity hypothesis is partially supported, even just considering this limited set of relations: both con-tingency_cause and expansion_instantiation are less marked than both comparison and tempo-ral_asynchronous. this is consistent with our hypotheses that continuous and causal relations should be less marked than discontinuous relations. however, like jin and de marneffe (2015b); asr and demberg (2013), we found less direct support for the causality-by-default hypothesis, since it is not less marked than expansion_instantiation. this is at best consistent with a weak form of the hypothesis, since we have not here reported contexts which would discriminate between the conjunction of the causality-by-default and continuity hypotheses versus just the continuity hypothesis. our conclusions are further reinforced by one further difference in distribution is predicted by the causality-by-default and continuity hypotheses for those relations that are or are not continuous or causal, exemplified by the relations found in we find that the d+ model shows a lower error proportion with respect to continuous versus discontinuous relations while the d-model shows a higher proportion of such errors, particularly where the relation to be expressed is discontinuous and more marked. we note that the number of d-nonsister errors on temporal_asynchronous, which dwarf the implicit errors on the same relation, is consistent with the continuity hypothesis in particular since these relations, which are not subject to default inferences, are important to mark explicitly yet are difficult to mark correctly in the absence of an explicit cue to the relation. on comparison, d-makes a similar number of nonsister errors, and also makes more than double the number of overall implicit prediction errors. this makes sense if the model recognizes it's important to signal these relations but erroneously treats them as if they were in a default relation where leaving the connective implicit would be more expected. however, we do not have a ready explanation for why temporal_asynchronous does not have more implicit d-errors. the differences in connective choice between models sometimes result in wildly divergent meanings. figure 4 shows the bart d+ full and bart d-outputs for the input in figure 1 . neither model is conditioned on the order of arguments. the bart d-model's output uses because to erroneously communicate that the intentions of the directors cause the intentions of the board, whereas the bart d+ model correctly identifies the intentions of the board and the intentions of the directors without suggesting either intention is dependent on the other (generates and). ## conclusion the main conclusion one can draw from our results is that discourse relation information is essential for consistently generating matching discourse connectives. while large-scale human judgement experiments on our models' predictions are the most obvious next step, the improvement of the bart d+ models over the bart d-models with respect to exact matching is encouraging, especially in light of recent results showing that humans don't uniformly accept substitution of discourse connectives which express the same discourse relation @xcite . with respect to whether mere arguments suffice to generate a discourse connective that correctly realizes the discourse relation holding between them, our results indicate that the purely distributional meaning of texts induced by the models under-determines the relation expressed by explicit discourse connectives. directly conditioning on discourse relations in the input significantly improves the likelihood of the model producing a connective which corresponds to the intended discourse relation. one must note that conditioning on the discourse relation is especially important when the relation is marked, as in these cases the model is apt to predict an incorrect default (causal or continuous) relationship just from the arguments. as for markedness score-based statistics, we can conclude that the presence of discourse relations in the input helped bart d+ to learn the discourse connective distribution patterns of the pdtb. these metrics provide a useful avenue for testing how well generation models recover patterns which hold for a variety of different variables, from discourse relations themselves, to the strength of co-occurrences between discourse relations and the words used to communicate them. to the degree these patterns track cognitive dependencies, they encourage integration of cognitive models of discourse coherence and nlg evaluation.