Towards Identifying Alternative-Lexicalization Signals of Discourse Relations (2022)

Abstract: The task of shallow discourse parsing in the Penn Discourse Treebank (PDTB) framework has traditionally been restricted to identifying those relations that are signaled by a discourse connective (“explicit”) and those that have no signal at all (“implicit”). The third type, the more flexible group of “AltLex” realizations has been neglected because of its small amount of occurrences in the PDTB2 corpus. Their number has grown significantly in the recent PDTB3, and in this paper, we present the first approaches for recognizing these “alternative lexicalizations”. We compare the performance of a pattern-based approach and a sequence labeling model, add an experiment on the pre-classification of candidate sentences, and provide an initial qualitative analysis of the error cases made by both models.

## introduction the view that discourse relations serve to model central aspects of the coherence of a text is widely accepted, and several approaches with different theoretical commitments have been developed (e.g. @xcite @xcite . our work is situated in the framework of shallow discourse parsing and thus grounded in the penn discourse treebank (pdtb) corpus @xcite . here, some distinctions are commonly being made regarding the surface realization of discourse relations; most importantly: in the pdtb corpus, (1) and ( 3 ) are by far the most frequent cases, and accordingly, they have received much attention in shallow discourse parsing. as for (2), @xcite had developed a first, relatively simple approach; to our knowledge there have not been any follow-up proposals (including all the parsers presented in the conll shared tasks in 2015 and 2016 @xcite and in the recent disrpt tasks @xcite ). with the introduction of the pdtb corpus version 3.0 @xcite , amongst some other changes, the number of annotated altlex instances has grown from 624 (in version 2.0) to 1632. with the corpus now being considerably richer in alt-lex signals, we believe that their role in shallow discourse parsing now needs to be strengthened. besides, also for advancing the theoretical description of the altlex category and its role in coherence marking, it is important to perform empirical studies. essentially, alternatively-lexicalized discourse relations are signalled by an open set of phrases that verbalizes the connection between two discourse arguments. in the pdtb, they are being annotated when no connective is present, and a "connective insertion" test yields an impression of redundancy; in this case annotators are asked to mark the text span that already signals the relation. previous work @xcite @xcite studied the general form of alt-lex expressions and tried to find and formalize patterns that can complement the well-established idea of a fixed list of discourse connectives (e.g. @xcite . example (1) @xcite illustrates the interchangeability of explicit connectives and altlex signals withing the same context, as we could simply substitute the connective therefore by alternative more complex lexicalizations such as this caused and because of this while still preserving the meaning of the relation: contrary to this, in example (2) @xcite the altlex signal is not easily substitutable by a simpler explicit connective as, with a replacement such as because, this sentence would lose information about the reason's importance: in this paper, we aim to overcome the negligence of altlex relations in shallow discourse parsing by proposing two different technical approaches. both tackle the problem without relying on any external lexical resources. specifically, our contributions are: (i) we present the first approach to automatically classifying altlex instances in the pdtb3 corpus; (ii) we compare a simple pattern-based approach to a neural model; (iii) we experiment with pre-classification of altlex-relevant sentences (for dealing with class imbalance in the corpus); (iv) we provide initial observations on types of errors made by the models. section 2 discusses related work, and section 3 briefly describes relevant aspects of the pdtb corpus. section 4 introduces our various methods. we present results in section 5, discuss them in section 6, and finally conclude. ## related work we highlight two different directions that are relevant for our work. the first part shows work related to alternative lexicalized phrases. the second one looks into recent applications for connective identification, whose methods are similar to one of our approaches. alternative lexicalized phrases (altlex). after the introduction of alternative lexicalized phrases, among others, in the pdtb @xcite , a subsequent work by @xcite presented more details and analysed regularities by defining groups of phrases. based on their discoveries, a first attempt was made to extract altlex relations while analysing implicit relations @xcite . their approach is evaluated in combination with all non-explicit relations. thus, the specific alt-lex results of their approach are unfortunately not available for comparison. due to the revision of the pdtb, the definition of discourse signals is made more flexible, and lin et al.'s approach is not applicable to the current problem anymore @xcite . attempts of building a czech discourse treebank have also shown, how challenging the annotation process is. in contrast to the pdtb, the czech discourse treebank distinguishes discourse signals into three groups, namely primary connectives, secondary connectives, and free connective phrases. low inter-annotator agreement was particularly observed for the annotation of free connecting phrases (comparable to alternative lexicalizations) due to the complexity of the task @xcite . based on this tree bank, a template approach is proposed to manually build a lexicon for secondary connectives, analogously to that for primary ones @xcite . one of danlos's single lexicon entry, for example, to recognize the phrases for this/a given reason, would describe the lexical head n of this phrase reason in different possible environments (so-called schemes) with a rule like for [ana-det (adj)/ana-adj] n. furthermore, @xcite concedes that, because "free connective phrases are compositional and include at least two content words", this lexicon-based approach is not applicable. @xcite adapt the pdtb annotation scheme and present another corpus that entirely focuses on causal relations. they do not distinguish between explicit connectives and alternative lexicalizations as done in the pdtb. comparing their annotated signals with the second version of pdtb, they discover an overlap of 8.9 % with the pdtb connective signals. further, they introduce a feature-based system for tagging causal relations between individual events. in contrast to them, we avoid linguistic features by using representations from pretrained language models. in contrast to pdtb conform schemes, rst @xcite contains only information about discourse segments, their relations and nuclearity. the work of @xcite complements the absence of this information by annotating a subpart of the rst-dt corpus with all kinds of signals point toward a discourse relation. their first finding reveals that discourse relations may be signalled by other discourse elements than lexical phrases, inter alia, syntactic structure (e.g. relative clauses, reported speech), semantics (e.g. synonymy, repetition, lexical chain), text genre (e.g. inverted pyramid scheme, newspaper layout). another interesting finding consists in the presence of multiple signals that point to the same relation, e.g. semantic+syntactic, refer-ence+syntactic, and others. further, recent work of @xcite proposed an interesting approach for inferring discourse signals from the given relation senses. however, this approach does not take alternative lexicalizations into account yet. explicit connective disambiguation. some of the approaches that were proposed to this task are relevant for altlex identification as well. @xcite presented a simple feature-based model for the disambiguation (discourse versus sentential usage) of connective candidates extracted by matching entries of a connective lexicon. with their best feature combination, they achieve an f1 score of 94.19 on the test set. recently, knaebel and stede (2020) adapt the original idea and replace hand-crafted features by various types of word embeddings. their state-of-the-art model uses contextualized word embeddings and achieves 97.45 f1 score. in our work, we extend their approach to alternative lexicalized phrases, tackling the problem without external candidate lexicon. furthermore, another line of research started with the focus to avoid lexicon-based solutions. recently, several promising sequence labeling approaches (e.g. @xcite @xcite have been proposed using standard and contextualized word embeddings. among these, @xcite achieves best scores (92.02 f1 score) in extracting connectives without lexicon. they develop a model that combines linguistic information (e.g. part-of-speech tags, dependency relation, sentence length, inter alia) with recent advances in contextualized word representations. in contrast, the work of @xcite achieve slightly worse results (91.15 f1 score) but with a completely feature-free approach. ## penn discourse treebank the recent version (v3) of the penn discourse treebank (pdtb, @xcite is the largest available resource of lexically grounded discourse relations which include both explicitly signalled relations and implicit relations. it describes discourse relations to consist of exactly two arguments with an optional marker to signal the relation. in addition, one or more senses are attached to each relation to describe its meaning. for example, two senses often correlate are temporal.synchronous and comparison.contrast for the explicit connective while. in its previous version, @xcite ) start the annotation process by the identification of connectives (defined by a fixed set of candidates). then, adjacent sentences without explicit relation are examined according to whether there holds an implicit relation, and, in addition, a connective is searched that fits in between the relations' arguments. if the insertion of any connective leads to redundancy, the lexical signal already part of the relation is used instead-the altlex. in their studies on alternative lexicalizations, @xcite use the two properties of syntactical and lexical flexibility to sort these signals into three groups. hereby they demonstrated, that most of the al-tlexes belong to the syntactically and lexically free group with 76.6%. as a consequence of the recent update, the definition of discourse signals has undergone some changes. the set of explicit connectives is expanded (which indirectly changes the set of alt-lexs too) and the position of the connective in relation to its arguments is more relaxed compared to the previous version. in addition, a few changes have been made in the process of identifying alt-lex relations in general. as a result of the introduction of intra-sentential altlex relations (about 900), arbitrary expressions are allowed, also including adjectives and adjective modifiers. also, annotators are allowed to mentally add anaphoric references, (e.g. next [to this], further [to that]), which leads to potential overlap with explicit connectives. signals are not syntactically bound to the second argument of the relation anymore, but possibly combine parts of both arguments. a new sub-class was introduced for lexico-syntactic constructions, the so-called altlexc. in total, almost 1000 altlex relations, including their signals, were added during the revision. in our work, we do ## method input to all models are prepared context sensitive word embeddings. for extracting token-wise context sensitive embeddings, we follow the suggestion of @xcite . given a sentence, tokens are processed by the wordpiece tokenizer @xcite which possibly leads to a higher number of subtokens. these are processed for generating corresponding hidden states on subtoken-level. mlp average candidate context left context right concatenate dense (sigmoid) altlex classifier candidate embedding figure 1: candidate-based classification approach. all token embeddings directly associated with altlex candidate are averaged and concatenated with surrounding context tokens. simple mlp module on top with final classification layer to predict discourse signal. we choose roberta @xcite as it performs best on connective disambiguation compared with other bert variations @xcite . we average multiple subtoken outputs into a single output that corresponds to the full token. following the suggestion of @xcite , the last four hidden layers are concatenated and thus form the final token embeddings that serve as input for the subsequent models. ## experiments in our experiments we study various statistical models on version 3 of the penn discourse treebank @xcite . for all experiments' runs, we randomly split the full dataset into three parts (train, validation, and test), as suggested by @xcite . we set 10 % of the full dataset aside for testing, then the remaining data is split into 90 % and 10 % for training and validation parts, respectively. the reported final results are averaged over three different runs each. for the precision-recall curves illustrated in figure 3 we interpolate the individual curve per run and compute the mean curve surrounded by one standard deviation. we use the average precision scores (ap) of the mean curves for comparison. ## discussion both pattern-based candidate approaches (exact and context-sensitive) achieve better final results compared to the labeling approaches which reflects a similar observation as for studies on explicit connective identification @xcite . however, we have to keep in mind that patterns are extracted on the whole corpus in advance, which makes the lexicon approach for explicit connective identification more similar and, thus, better comparable. the comparison to our sequence labeling approach is somewhat unfair, as we here strictly split the corpus into three parts right at the beginning and these models never have access to all signal variants in the whole corpus. compared to the recognition performance of explicit connectives with about 96%, altlex relations are predicted far less accurately with at most 63% exact match for the pattern-based approach. also, we expect a drop in performance when limiting the extraction to the training corpus only. after examining the errors made by the patternbased mode, we conclude that the length of the errors (unrecognized signals) compared to the correctly recognized patterns is almost the same (2.5 and 2.7 tokens on average each). thus the length of signals is not a crucial factor for this type of model. quite a few poorly recognized examples are related to adverbial phrases (e.g., eventually, further, so far, too). this type of error seems problematic for both model variants. a possible reason is the high imbalance of adverbial signal instances (about 100), compared to extracted candidates ranging from 1000 to 4000 instances. regarding verb gerund forms, the pattern-based approach recognizes most of them. this is interesting as it represents the largest group in pdtb with most variations. on the other side, we also recognize cases where signals are marked as false positives, such as resulting and allowing. here, more more in-depth elaboration is necessary, to check whether the model is truly wrong or just for the sequential model, it is noticeable that prepositions are missing in the predicted signal e.g. only to and opposed to. also, verbs in combination with anaphoric pronouns e.g. that would leave, this creates, are often recognized as discourse signal although there are not annotated as such. the sequential models with additional context information in form of the previous sentence result in unexpectedly low performances. intuitively, because altlexes often connect parts of two consecutive sentences, we would expect a model's performance to increase if it gets access to more information. we assume poor performances are caused by context representation and therefore we wonder whether models especially designed to serve sentence representations would lead to better results in this experiments. ## conclusion our work is a successful first attempt to fully automatically (without hand-crafted rules) extract alternative lexicalized discourse relation signals. for this task, we propose two technically different solutions: first, a pattern-based approach working analogously to lexicon-based connective disambiguation approaches, and second, a sequence labeling approach similar to recent connective labeling ap-proaches without external lexicon. we evaluated these models directly on their corresponding training task and, further, provide more details on the actual recognition task. we wonder, how these two model architectures perform on a different corpus domain such as biomedical data @xcite and whether the pattern-based limitation of the first approach is noticeable. having the new version of the pdtb with about three times as much data as before, it still seems the performance of the sequencelabeling approach is strongly limited by the amount of available data. for future work, it would be interesting to extract different patterns for generating candidates for the first approach. universal partof-speech tags would have the advantage of being a little more flexible (less specific word classes) while at the same time, it could be possible to use similar techniques for other languages when the embeddings model is changed to a different language or to a multi-lingual model. the sequential approach has the advantage to be able to find new patterns without observing them in the training data directly. with more raw data from possibly different domains, it would be interesting to apply this technique and examine new/other variants of alternative lexicalizations that do not occur in such form in the original corpus.