Revisiting Shallow Discourse Parsing in the PDTB -3: Handling Intra-sentential Implicits (2021)

Abstract: In the PDTB-3, several thousand implicit discourse relations were newly annotated within individual sentences, adding to the over 15,000 implicit relations annotated across adjacent sentences in the PDTB-2. Given that the position of the arguments to these intra-sentential implicits is no longer as well-defined as with inter-sentential implicits, a discourse parser must identify both their location and their sense. That is the focus of the current work. The paper provides a comprehensive analysis of our results, showcasing model performance under different scenarios, pointing out limitations and noting future directions.

## introduction discourse parsing is the task of identifying and categorizing discourse relations between discourse segments in a given text. the task is considered to be important for downstream tasks such as question answering @xcite , machine translation @xcite , and text summarization @xcite . there are various approaches to discourse parsing, corresponding to different views of ( 1 ) what constitutes the segments of discourse, (2) what structures can be built from such segments, and (3) what semantic and/or rhetorical relations can hold between such segments @xcite . approaches to discourse structure generally allow discourse relations to hold between segments within a sentence (i.e., intra-sentential discourse relations) or across sentences (i.e. inter-sentential discourse relations) @xcite @xcite . in the penn discourse treebank (pdtb; @xcite , all discourse relations have two arguments, called arg1 and arg2. discourse relations are termed explicit, if the evidence for the relation is an explicit discourse connective (word or phrase). for implicit discourse relations, evidence is in the form of argument adjacency (with or without intervening punctuation), though annotators were asked to record one or more discourse connectives that, if present, would explicitly signal the sense(s) they inferred to hold between the arguments. where annotators felt that the relation was already signalled by an alternative (non-connective) expression, the expression was annotated as evidence for what was called an altlex relation @xcite . the first major release of the pdtb was the pdtb-2 @xcite whose guidelines limited annotation to (a) explicit relations lexicalized by discourse connectives, and (b) implicit and alt-lex relations between paragraph-internal adjacent sentences and between complete clauses within sentences separated by colons or semi-colons. since there were only ∼530 intra-sentential implicit relations among the ∼15,500 implicit relations annotated in the pdtb-2, they were ignored in work on discourse parsing @xcite @xcite , which took implicit relations to hold only between adjacent sentences. the situation changed with the release of the pdtb-3 @xcite . among the ∼5.6k sentence-internal implicit relations annotated in the pdtb-3 are relations between vps or clauses conjoined implicitly by punctuation (ex. 1), between a free adjunct or free to-infinitive and its matrix clause (ex. 2), and between a marked syntactic construction and its matrix clause. there are also implicit relations co-occurring with explicit relations @xcite , as noted in section 3.1. this work takes up some of the challenges of parsing implicit intra-sentential discourse relations. overall, it contributes: (1) a set of bert-based models used as a pipeline for recognizing intrasentential implicit discourse relations as well as classifying their senses; (2) experimental evidence that these bert-based models perform better than comparable lstm-based models on the relation recognition task; (3) evidence that the use of parse tree features can improve model performance, as was earlier found useful in simply recognizing when a sentence contained at least one implicit intra-sentential relation @xcite . ## related work the focus of the current work is parsing implicit intra-sentential discourse relations in the framework of the pdtb-3. as most of the implicit relations in the pdtb-2 were inter-sentential (i.e., ∼95% of its 15.5k implicit relations), its intrasentential implicits were ignored in parser development. nearly all recent work on recognizing inter-sentential implicits in the pdtb-2 used neural architectures. this included multi-level attention in work by @xcite , multiple text representations in work by @xcite , including character, subword, word, sentence, and sentence pair levels, to more fully capture the text. @xcite introduced a paragraph-level neural architecture with a conditional random field (crf, @xcite layer which models inter-dependencies of discourse units and predicts a sequence of discourse relations in a paragraph. @xcite introduced an approach to distill knowledge from word pairs for discourse relation with cnn by joint learning of implicit and explicit relations. @xcite discovered that bert-based models, which were trained on the next sentence prediction task, benefited implicit inter-sentential discourse relation classification. here we assess whether they also benefit classifying intra-sentential implicit relations. looking at implicit relations in the pdtb-3, @xcite consider the difficulty in extending implicit relations to relations that cross paragraph boundaries. @xcite examine whether implicit relation annotation in the pdtb-3 can be used as a basis for learning to classify implicit relations in languages that lack discourse annotation. @xcite explored whether the pdtb-3 could be used to learn finegrained (level-2) sense classification in general, while @xcite looked at whether separating inter-sentential implicits from intra-sentential implicits could improve their sense classification. they also took a first step towards recognizing what sentences contained intra-sentential implicit relations, finding this benefitted from the use of linearized parse tree features. outside the pdtb-3 framework, intra-sentential discourse relations are handled by (1) identifying discourse units (dus), (2) attaching them to one another, and (3) associating the attachment with a coherence relation @xcite . one can therefore ask why we did not simply adopt this framework in the pdtb-3 and exploit the relatively good performance by systems in the disrpt shared task on sentence-level discourse unit segmentation @xcite . there are two main reasons: first, disrpt (and the approaches to discourse structure it covers) assumes that discourse segments cover a sentence with a non-overlapping partition. this is not the case with the pdtb, where the presence of overlapping segments (both within and across sentences) has been well documented @xcite . second, discourse segments in these approaches are taken to correspond to syntactic units, which leads to both over-segmentation and under-segmentation in the pdtb-3. of course, there are "work-arounds" for over-segmentation, such as rst's use of a same-segment relation @xcite , and under-segmentation can be addressed through additional segmentation. however, we decided that starting from scratch would allow us to clearly identify the problems of parsing intra-sentential implicits, at which point, we could consider what we could adopt from work done on the disrpt shared task on sentence-level discourse unit segmentation @xcite . ## methodology given an input sentence s represented as a sequence of tokens s 1 • • • s n , our aim is to identify the span of arg1 and arg2 if there exist an implicit discourse relation in that sentence and then to predict its corresponding sense relation. we treat the identification of argument spans as a sequence tagging problem, and the prediction of senses as a classification task. thus, given s, our aim is to output both a tag sequence y of length n and a sense label c for the identified relation. the generated tag sequence y 1 • • • y n contains token-level labels where y @xmath0, b-arg2, i-arg1, i-arg2, o} indicating whether the token belongs to arg1, arg2, or other. we adopt the bio format (ramshaw and marcus, 1999) since arguments (1) can span multiple tokens, (2) can occur in either order, (3) need not be adjacent, and (4) do not overlap. (future work will address two additional properties of intra-sentential implicits: (1) as shown in sec 3.1, a sentence can contain more than one such relation and ( 2 ) even though most arguments are continuous spans, 264 intra-sentential implicit relations (4.2%) have discontinuous spans.) this section describes the two parts of our approach. section 3.1 describes the creation of two datasets based on the pdtb-3: ## experiments for our experiments with lstm-based models, we set the hidden dimensions to 256, the word embeddings to 100, and the vocabulary size to 50k. the word embeddings are initialized either using pre-trained glove vectors (6b tokens, uncased) or a pre-trained base-uncased bert. for experiments with bert-based models, we use the same configuration with base-uncased bert from @xcite . all of our training used the adam optimizer @xcite . lstm-based models used a learning rate of 1e-3 and bert-based models used a learning rate of 5e-5. we also use gradient clipping with a maximum gradient norm of 1 and we do not use any form of regularization. for sense classification, the gradient clipping is set with a maximum gradient norm of 0.5. we carry out assessment on the datasets mentioned in section 3.1 in two different ways: (1) using a random split of each dataset into training (60%), development (20%) and test (20%) subsets, and (2) accepting the argument in @xcite that there is too much variation across the corpus for a single random split to produce representative results and that n-fold cross-validation will deliver more reliable and predictive results. in this work, we perform 10-fold cross-validation. we use loss on the development set to perform early stopping. all of our models were trained on a single tesla p100 gpu with a batch size of 32. our model implementation uses pytorch @xcite . we used the bert implementation from the transformers library @xcite , and the crf implementation from allennlp library @xcite for constrained decoding for the bio scheme. ## conclusions and future work to the best of our knowledge, this is the first work to attempt to identify the arguments of intrasentential implicit discourse relations in the framework of the pdtb-3, as well as their order and at least one of their sense relations. we used a model architecture similar to models of a sequence tag-ging task, and concluded that bert-based models have better performance than lstm-based models in both exactly matching gold annotations of arguments and correctly predicting the order of arg1 and arg2. we confirmed that using parse trees features as input to the model assists with these tasks. we also provide evidence that our sense classifier, together with the argument recognizer, can be used as a pipeline for handling intra-sentential implicit relations. we also find that the sense classifier can be used to aid the selection of the right argument from the set of predicted argument candidates.