Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation (2023)

Abstract: Implicit discourse relation classification is a challenging task due to the absence of discourse connectives. To overcome this issue, we design an end-to-end neural model to explicitly generate discourse connectives for the task, inspired by the annotation process of PDTB. Specifically, our model jointly learns to generate discourse connectives between arguments and predict discourse relations based on the arguments and the generated connectives. To prevent our relation classifier from being misled by poor connectives generated at the early stage of training while alleviating the discrepancy between training and inference, we adopt Scheduled Sampling to the joint learning. We evaluate our method on three benchmarks, PDTB 2.0, PDTB 3.0, and PCC. Results show that our joint model significantly outperforms various baselines on three datasets, demonstrating its superiority for the task.

## introduction discourse relations, such as cause and contrast, describe the logical relation between two text spans @xcite . recognizing discourse relations is beneficial for various nlp tasks, including coherence modeling @xcite , reading comprehension @xcite , argumentation mining @xcite , and machine translation @xcite . discourse connectives (e.g., but, as a result) are words or phrases that signal the presence of a discourse relation @xcite . they can be explicit, as in @xcite , or implicit, as in ( 2 ): when discourse connectives are explicitly present between arguments, classifying the sense of a discourse relation is straightforward. for example, @xcite proved that using only connectives in a text as features, the accuracy of 4-way explicit discourse relation classification on pdtb 2.0 can reach 85.8%. however, for implicit cases, there are no connectives to explicitly mark discourse relations, which makes implicit discourse relation classification challenging @xcite . existing work attempts to perform implicit discourse relation classification directly from arguments. they range from designing linguistically informed features from arguments @xcite to modeling interaction between arguments using neural networks @xcite . despite their impressive performance, the absence of explicit discourse connectives makes the prediction extremely hard and hinders further improvement @xcite . the huge performance gap between explicit and implicit classification (85.8% vs. 57.6%) @xcite motivates recent studies to utilize implicit connectives for the training process of implicit relation classifiers. for instance, @xcite developed an adversarial model to transfer knowledge from the model supplied with implicit connectives to the model without such information, while @xcite proposed a multi-task learning framework to incorporate implicit connectives prediction as another training objective. however, we argue that these methods are suboptimal since connectives are still not explicitly present in input texts. this is demonstrated by @xcite , concluding that adding implicit connective prediction as a training objective provides only negligible gain for implicit relation classification on pdtb 2.0 (we empirically found the conclusion also held on the adversarial model). in this paper, we design a novel end-to-end model to leverage discourse connectives for the task of implicit discourse relation classification. the key inspiration is derived from the annotation process of implicit discourse relations in pdtb, which consists of inserting a connective that best conveys the inferred relation, and annotating the relation label based on both the inserted implicit connectives and contextual semantics @xcite . we imitate this process by explicitly generating discourse connectives for the implicit relation classifier. specifically, our model jointly learns to generate discourse connectives between arguments and predict discourse relations based on the arguments and the generated connectives. a potential drawback of this joint model is that the poorly generated connectives at the early stage of joint training may mislead the relation classifier. one possible solution is always feeding true connectives to the implicit relation classifier for training. but it leads to severe discrepancies between training and inference @xcite , since manually-annotated connectives are unavailable during evaluation @xcite . to address this issue, we adopt scheduled sampling @xcite into our method. to be more specific, our relation classifier is first trained with hand-annotated implicit connectives and then gradually shifts to use generated connectives. we evaluate our model 1 on two english corpora, pdtb 2.0 @xcite , pdtb 3.0 @xcite , and a german corpus, pcc (bourgonje and stede, 2020), and compare it with other connective-enhanced approaches and existing stateof-the-art works. results show that our method significantly outperforms those connective-enhanced baselines on three datasets while offering comparable performance to existing sota models. in addition, we perform the first systematic analysis of different connective-enhanced models to investigate why our method works better. our studies show that: (1) models learn to use connectives more effectively when putting connectives in the input rather than using them as training objectives; ## related work implicit discourse relation classification, as a challenging part of shallow discourse parsing, has drawn much attention since the release of pdtb 2.0 @xcite . most of the work focused on predicting implicit relations directly from input arguments. for example, early statistical methods have put much effort into designing linguistically informed features from arguments @xcite @xcite . more recently, neural networks @xcite @xcite @xcite have been applied to learning useful semantic and syntactic information from arguments due to their strength in representation learning. despite achieving impressive results, the absence of connectives makes their performance still lag far behind explicit discourse parsing. the question of how to leverage discourse connectives for implicit discourse relation classification has received continued research attention. @xcite proposed a pipeline method to investigate the benefits of connectives recovered from an n-gram language model for implicit relation recognition. their results show that using recovered connectives as features can achieve comparable performance to a strong baseline. this pipeline-based method is further improved by following efforts, including integrating pre-trained models @xcite and using prompt strategies @xcite . however, some works @xcite pointed out that pipeline methods suffer cascading errors. recent studies have shifted to using end-to-end neural networks. @xcite proposed a feature imitation framework in which an implicit relation network is driven to learn from another neural network with access to connectives. @xcite designed an encoder-decoder model that generates implicit connectives from texts and learns a relation classifier using the representation of the encoder. @xcite investigated a multi-task learning approach to predict connectives and discourse relations simultaneously. our method is in line with those recent approaches exploiting connectives with an end-to-end neural network. the main difference is that those models focus on using implicit connectives in a non-input manner (i.e. they do not input implicit connectives as features but utilize them as another training signal), whereas our method explicitly generates connectives and inputs both arguments and the generated connectives into the relation classifier. our method can be viewed as a joint learning framework. such a framework has been used to learn information exchange and reduce error propagation between related tasks @xcite . @xcite designed a unified neural model to perform tagging, chunking, and ner jointly. @xcite refined this unified framework by putting low-level tasks supervised at lower layers. @xcite presented an lstm-based model to extract entities and the relations between them. @xcite proposed a joint model for semantic role labeling (srl), in which dependency parsing results were used to guide the attention module in the srl task. compared with these works, our joint learning framework is different in both motivation and design. for example, instead of simply sharing an encoder between tasks, we input the results of connective generation into the relation classifier. ## method inspired by the annotation process of pdtb, we explicitly generate discourse connectives for implicit relation classification. following previous work @xcite , we use the gold standard arguments and focus on relation prediction. figure 1 shows the overall architecture of our proposed model. it consists of two components: (1) generat-ing a discourse connective between arguments; (2) predicting discourse relation based on arguments and the generated connective. in this section, we describe these two components in detail and show the challenges during training and our solutions. formally, let x 1 = {x 1 , ..., x n } and x 2 = {x n+1 , ..., x n+m } be the two input arguments (arg1 and arg2) of implicit relation classification, where x i denotes the i-th word in arg1 and x n+j denotes the j-th word in arg2. we denote the relation between those two arguments as y. similar to the setup in existing connective enhanced methods, each training sample (x 1 , x 2 , c, y) also includes an annotated implicit connective c that best expresses the relation. during the evaluation, only arguments (x 1 , x 2 ) are available to the model. ## experiments we carry out a set of experiments to investigate the effectiveness of our method across different cor-pora and dataset splittings. in addition, we perform analyses showing that our model learns a better balance between using connectives and arguments than baselines. ## conclusion in this paper, we propose a novel connectiveenhanced method for implicit relation classification, inspired by the annotation of pdtb. we introduce several key techniques to efficiently train our model in an end-to-end manner. experiments on three benchmarks demonstrate that our method consistently outperforms various baseline models. analyses of the models' behavior show that our approach can learn a good balance between using arguments and connectives for implicit discourse relation prediction.