Multi-Label Classification for Implicit Discourse Relation Recognition (2024)

Abstract: Discourse relations play a pivotal role in establishing coherence within textual content, uniting sentences and clauses into a cohesive narrative. The Penn Discourse Treebank (PDTB) stands as one of the most extensively utilized datasets in this domain. In PDTB-3, the annotators can assign multiple labels to an example, when they believe the simultaneous presence of multiple relations. Prior research in discourse relation recognition has treated these instances as separate examples during training, with a gold-standard prediction matching one of the labels considered correct at test time. However, this approach is inadequate, as it fails to account for the interdependence of labels in real-world contexts and to distinguish between cases where only one sense relation holds and cases where multiple relations hold simultaneously. In our work, we address this challenge by exploring various multi-label classification frameworks to handle implicit discourse relation recognition. We show that the methods for multi-label prediction don’t depress performance for single-label prediction. Additionally, we give comprehensive analysis of results and data. Our work contributes to advancing the understanding and application of discourse relations and provide a foundation for the future study.

## introduction discourse relations serve as a fundamental framework for creating coherence among sentences and clauses within a text. the automated identification of the discourse relations, which connect sentences and clauses, holds significant importance for a wide range of downstream natural language processing (nlp) tasks including text summarization @xcite , question answering @xcite , and event relation extraction @xcite . penn discourse treebank (pdtb) @xcite is one of the most widely used datasets for this task. it includes over 2000 documents from the wall street journal published in 1989, which have been manually annotated with discourse relations. with regard to pdtb annotation, annotators can assign multiple sense labels to an example when they believe that all hold simultaneously. consider the following example from pdtb-3. ## dataset and evaluations we employed pdtb-3 @xcite for our evaluation. pdtb-3 represents an advancement over pdtb-2 @xcite , offering a more extensive collection of annotated multi-label examples. in our study, we concentrated exclusively on implicit discourse relations, disregarding those with explicit connectives. and we consider both intra-sentential and inter-sentential implicits. about 5% of pdtb-3 implicit discourse relations receive multiple labels, which corresponds to instances with two annotated labels. we treated such instances as single examples with multi-labels during training, and during testing, predictions were considered correct only if they match the specific label. while previous studies @xcite @xcite typically allocate sections 2-20 of pdtb for training, sections 0-1 for validation, and sections 21-22 for testing, the limited size of the test set poses challenges, particularly for rare label and label pairs within the dataset. acknowledging the concerns raised by shi and demberg regarding label sparsity, we addressed this issue by employing cross-validation for level-2 classification. in line with the methodology proposed by @xcite , we adopted a crossvalidation approach at the section level. we divided pdtb-3 into 12 folds, with each fold partitioned into 21 sections for training, two for development, and two for testing. by splitting the data at the section level, we can preserve the inherent paragraph and document structures, ensuring that data from the same sections are grouped together in the same pool. following the work in multi-label classification for other tasks @xcite , we adopt f1 scores @xcite as our main evaluation metric. precision, recall and hamming loss @xcite are reported in the appendix. we present the macroaveraged results of f1 scores, precision and recall. ## conclusions we conducted a comprehensive exploration of implicit discourse relation recognition (idrr) as a multi-label classification problem, addressing realworld complexity and label interdependence. we found that multi-label approaches can have better performance than traditional single-label metrics when evaluated on the criteria for the single-label prediction. our exploration of multi-label frameworks, loss function optimization, and cross-validation strategies sets the stage for future research advancements. additionally, our detailed analysis can provide valuable insights for further investigations in this field. considered to see whether it either agree with or contradict multi-sense pdtb-3 sense annotation, although discogem is not inherently a multi-label dataset but rather a collection with diverse annotations stemming from varying perspectives.