A practical perspective on connective generation (2021)

Abstract: In data-driven natural language generation, we typically know what relation should be expressed and need to select a connective to lexicalize it. In the current contribution, we analyse whether a sophisticated connective generation module is necessary to select a connective, or whether this can be solved with simple methods (such as random choice between connectives that are known to express a given relation, or usage of a generic language model). Comparing these methods to the distributions of connective choices from a human connective insertion task, we find mixed results: for some relations, it is acceptable to lexicalize them using any of the connectives that mark this relation. However, for other relations (temporals, concessives) either a more detailed relation distinction needs to be introduced, or a more sophisticated connective choice module would be necessary.

## introduction we assume a natural language generation setting in which we know, based on content planning, what coherence relation we wish to express, and the problem consists of choosing a suitable connective to express this relation. of course, this problem is not independent of the discourse relation framework that is used -in the current study, we work with the relation distinctions proposed by pdtb-3 @xcite . from a generation perspective, the task would be easiest if, within relation distinctions, the connectives would be substitutable with each other. for instance, consider a framework that distinguishes causal relations from others, but within this class does not distinguish reason relations from result relations. using such a framework, we would not be able to correctly select between connectives such as "because" and "therefore" as we wouldn't be able to express their difference. in this case, our connective insertion method would have to be able to learn this distinction in order to allow for fitting choices. in this study, we aim to answer the following questions: we conduct two empirical connective insertion studies to test whether humans have a preference for the connective that was originally present in the text when being asked to lexicalize a relation. several findings seem possible: a) all connectives that express a specific coherence relation are fully interchangeable -in this case, we expect to see that human participants do not have any specific preference among the connectives that express the relation. we could conclude that the pdtb distinctions are sufficient, and using a simple connective choice method is sufficient. b) all connectives are correct, but there are other factors, such as information-theoretic aspects, that influence which connective is preferred (e.g., a short vs. a longer / rarer connective). again, we would conclude that pdtb distinctions are sufficient, but that other factors such as information density need to be taken into account. in both cases b) and c), we would expect to find that humans show a peaked distribution, preferring a specific connective or dispreferring a specific connective to express a relation. a more detailed analysis on these cases is conducted in order to check whether the preference is due to a lack of substitutability or rather a softer factor. our first study uses as material a naturalistic distribution of relations and finds that random choice among fitting connectives would achieve good accuracy on this problem -however, the coherence relation distribution in this first study is dominated by a small number of frequent relations. the second study therefore uses a more balanced design to better represent less frequent coherence relations. in this study, we find more nuanced results: while there are indeed some relations for which any of the matching connectives can be inserted, and a language model like gpt-2 @xcite would perform well, there are also some relations for which simple automatic methods would systematically choose unsuitable lexicalizations. we analyse the latter cases in more detail in section 5.2. ## related work there has been continuing interest in the task of connective prediction in recent years, but mostly as an auxiliary task to coherence relation classification. @xcite use a language model for the task of connective prediction and @xcite deploy word pairs as well as a set of linguistically motivated features. more recently, @xcite ; @xcite and @xcite have used connective prediction for implicit relations as a secondary or adversarial task to improve discourse relation classification. the current study tackles a different task than these studies, where the relation label is assumed not to be available. instead, we assume that the generation system knows what relation should be conveyed, and the remaining problem is the lexicalization using a connective. another related study is @xcite , who analyse the discourse abilities of gpt-2 and find that connectives are sometimes incorrect. they propose to use a specific discourse component to address connective generation. again, however, their setting is different from ours, as we assume that connectives need to be inserted into a text which is generated with a known discourse intention. an experimental study closely related to the current contribution was conducted by @xcite . crowd-workers were asked to guess the original connective in a text where the explicit connective had been removed. our analysis of their results shows that the omission of an explicit connective often leads to a change in interpretation of the relation: for ca. 80% of explicitly marked relations, participants did not recover the original connective. therefore, our study is designed differently: we ensure that workers express the intended relation by providing them with a choice among connectives that can mark the original relation. the substitutability of connectives has been studied in previous literature. most notably, @xcite ; @xcite explored this topic using a connective substitution task, and created a hierarchy of connectives based on these results. our study adds quantitative data on connective insertion preferences, as well as a practical perspective by investigating connective choice based on pdtb-3 discourse relation labels (and not just substitutability with respect to other connectives). ## methodology we use crowdsourced human experiments to examine whether a specific dc is preferred for a given discourse relation instance. the approach is straight-forward: from discourse-annotated corpora, we sample a set of explicitly marked discourse relations. the original connectives are then removed from the instances and crowdworkers are asked to select the best-suiting connective. several options of connectives, which are all valid explicit markers of the annotated discourse relation sense, are given to the workers to choose from. if the choices of connectives made by the crowdworkers reproduce the original connective in the data, it indicates that this particular connective should be preferred over the other alternatives for that specific discourse relation instance. on the other hand, if the choices made by the crowdworkers are evenly distributed per instance, it indicates that the discourse relation instance in question could be interchangeably marked by alternative connectives. in addition, we test whether a language model (here, gpt-2, @xcite can select the appropriate connective from the same set of options also given to the human participants. the preferred connective is chosen based on the cross entropy loss of the language model. if the connective with the highest probability based on the language model is identical to the preferred choice by humans, this suggests that a language model is sufficient for generating an appropriate connective for a given discourse relation. if, on the other hand, the language model prefers a connective which is not preferred by humans, this indicates that the language model is missing a relevant aspect of the coherence relation. ## language model connectives were compared with respect to the average negative log likelihood of the sentence including the connective, using openai's pretrained generatively pre-trained transformer (gpt-2) language model @xcite implemented in the transformers library of huggingface @xcite . gpt-2 is a unidirectional transformerbased language model trained on a dataset of 40 gb of web crawling data. the model compared the same set of connective choices as given to the human crowdworkers. the connective choice resulting in the best sequence according to gpt-2 was selected as the predicted connective. 5 results ## discussion we set out to investigate the extent to which the choice of connective to realize a particular relation is constrained. specifically, we saw three possible outcomes: connectives that express a relation are fully interchangeable for the pdtb-3 relations that we work with here, other factors (e.g., informationtheoretic ones) lead to preferences even though interchangeability is given, or the relation sense distinctions are not sufficiently fine-grained, so that an inappropriate connective might be chosen. the main result is that connective choice varies depending on the dataset: in ted-mdb, high accuracy is achieved by choosing the most probable realization according to gpt-2 or choosing the most common connective per relation type. in pdtb, accuracy of the connective choice model is a lot lower. we identified a set of different cases: a very simple connective selection strategy of randomly choosing among fitting connectives, might be good enough for some relations (case 1). for other relations (identified as case 2a above), a languagemodel-based strategy seems to perform well. however, we also identified some more problematic cases (classified as 2b above): for these cases, we would either need to extend the relation inventory in order to capture the more finegrained distinctions, such as temporal aspects (in succession relations), factuality of events (in arg2-as-condition) or whether there is a shared common ground (reason relations). alternatively, a more sophisticated language model would have to be developed, which can learn to use the correct connective in these cases. in future work, we aim to evaluate to what extent this could be learned by a transformer model which has been fine-tuned on a connective insertion task where the arguments and the target relation is given, and the correct connective needs to be selected. we also found some limited evidence for other factors influencing connective choice -for the selection of a rather common and light connective like and or but vs. a more heavy and longer connective like in addition or on the other hand, it seems that while substitutability is given, preferences in terms of focus on the relation and heaviness and distance of the relational arguments may play an important role in connective choice. again, it would be interesting to see whether a more sophisticated language model could pick up on these aspects. the analyses performed here are clearly empirical. we do not claim that our method has allowed us to detect all possible cases where substitutability of connectives might not be given. instead, our analyses provide a practical perspective indicating how a simple generation system for connectives would fare for frequent relations and connectives. in relation to this, the difference in performance of the generation system on ted-mdb and pdtb provides interesting insights. these datasets represent different domains: the ted-mdb dataset consists of spoken data, whereas the pdtb dataset consists of written data. spoken data tends to be characterized by a smaller range of connective types, as was the case in our data. the model's performance on this dataset seems promising, as it generally selects the same connective as the original one. as our subsequent analysis shows, this is mostly because these relations are "easy" -the original connective in ted-mdb is often the most common connective for a given relation type (e.g., and, but). in the pdtb dataset, we manipulated the range and distribution of connectives by specifically choosing instances of a larger number of connectives, to give a more representative picture of the limitations of the simple gpt-2 based model. some of the connectives in this set are infrequent in natural language. we find that the gpt-2 tends to default to frequently occurring connectives within relation senses (e.g., but, and, even though), irrespective of the original connective. finally, we note that the results of our analysis of course also depend on our choice of using pdtb as a framework here -other frameworks can differ in what coherence relations they distinguish, and accordingly, the results regarding which relations need to be distinguished at a more fine-grained level might vary. the methodology used here can also be applied to data from other frameworks to evaluate whether those distinctions are sufficient for computational purposes such as connective generation. ## conclusion the current study showed that, in spoken data, a language generation system can predict the connective that should be used to express a discourse relation with high accuracy. this is partially because the relations tend to be quite simple and be marked by high-frequency connectives: even simply choosing the most prototypical connective for a given relation sense shows high accuracy on this data set. on the other hand, our subsequent analysis of written data, using a more balanced set of relations and connectives, highlighted that there is a clear need for a more sophisticated method for connective choice. the results from human annotators indicated that connectives for many relational classes are not fully interchangeable. in some cases (succession, condition, and reason relations), additional finer-grained relation types are needed to capture more information. in other cases (relating to specific connectives, such as in addition and however), information-theoretic constraints appear to influence connective choice. these insights can be useful for natural language generation researchers as well as research on automatic discourse parsing. relation types original connectives (item count) agree with org. χ 2 test (item count) / connective options maj. lm χ 2 df conjunction and (6), also (6), in addition (7), (19) moreover, or .316 .632 12.01 8 arg2-as-detail indeed (4), in fact (4), specifically (4) (16) in particular (4), and .200 .067 24.81 12 * arg2-as-instance for example (9), for instance (9), (18) in particular, in fact, as .278 .389 20.33 4 *** reason because (6), as (6), since (6), (18) for, when .444 .555 20.78 8 ** result so (6), thus (6), as a result (6), (18) therefore, and .444 .500 36.51 8 *** arg2-as-condition if (9), when (9), until, as long as, (18) in case .444 .278 31.95 4 *** arg1-as-denier although (4), though (4), even if (3), (16) even though (5), while .500 .500 46.82 12 *** arg2-as-denier but (4), however (4), though (4), (20) still (4), yet (4) .150 .250 11.23 16 contrast but (4), while (4), however (3), (15) still (4), on the other hand .530 .400 58.30 12 *** precedence then (5), before (5), until (4), (14) later, ultimately .786 .929 142.60 8 *** succession after (4), when (4), since (3), (17) once (3), previously .611 .500 179.72 16 *** synchronous when (4), as (4), while (3), (18) meanwhile (3), at the same time (3) .611 .389 116.84 16 *** overall (215) .423 .433