What Causes the Failure of Explicit to Implicit Discourse Relation Recognition? (2024)

Abstract: We consider an unanswered question in the discourse processing community: why do relation classifiers trained on explicit examples (with connectives removed) perform poorly in real implicit scenarios? Prior work claimed this is due to linguistic dissimilarity between explicit and implicit examples but provided no empirical evidence. In this study, we show that one cause for such failure is a label shift after connectives are eliminated. Specifically, we find that the discourse relations expressed by some explicit instances will change when connectives disappear. Unlike previous work manually analyzing a few examples, we present empirical evidence at the corpus level to prove the existence of such shift. Then, we analyze why label shift occurs by considering factors such as the syntactic role played by connectives, ambiguity of connectives, and more. Finally, we investigate two strategies to mitigate the label shift: filtering out noisy data and joint learning with connectives. Experiments on PDTB 2.0, PDTB 3.0, and the GUM dataset demonstrate that classifiers trained with our strategies outperform strong baselines.

## introduction discourse relations, such as contrast and cause, describe the logical relationship between two text spans (i.e., arguments). they can either be signaled explicitly with connectives, as in (1), or expressed implicitly, as in (2): corpora of explicit discourse relations are relatively easy to create manually and automatically @xcite since connectives are strong cues for identifying relations @xcite . in contrast, the annotation of implicit relations is hard and expensive because one must infer the sense from input texts. this has prompted many early studies @xcite @xcite to use explicit examples to classify implicit relations (dubbed explicit to implicit relation recognition). the main idea is to construct an implicit-like corpus by removing connectives from explicit instances, and use it to train a classifier for implicit relation recognition. while this method attains good results on test sets constructed in the same manner, it is reported by @xcite to perform poorly in real implicit scenarios. they claim this is caused by the linguistic dissimilarities between explicit and implicit examples, but provide no corpus-level empirical evidence. more recent works @xcite focus on enhancing the transfer performance from explicit to implicit discourse relations. however, little attention has been paid to the underlying causes of the poor results. in this paper, we show that one cause for the poor transfer performance is the occurrence of label shift during the construction of the implicit-like corpus. removing connectives from explicit examples affects the discourse relations they originally expressed. referring to example (3), which contains the connective then and is annotated as a temporal.asynchronous relation: when the connective then is removed, the example, however, tends to express a contingency.cause relation because the first argument describes a re-sult of "stock plummet" and the second argument gives the reason, a suspension of dividend pay. to verify the existence of the label shift, we first manually analyze a small number of explicit examples with connectives removed, and summarize different cases of instances suffering such shift. then, we provide empirical evidence to demonstrate that label shift is present not only in a few examples but at the corpus level. consequently, classifiers trained on the implicit-like corpus learn a chaotic pattern for relation classification (i.e., being taught to predict an example with cause relation as asynchronous relation as in the above example), resulting in poor performance in real implicit scenarios. we further analyze why label shift happens in the implicit-like corpus by considering factors such as the syntactic role played by connectives, ambiguity of connectives, and more. our results reveal that the syntactic role played by connectives contributes the most to the occurrence of the label shift. based on this observation, we investigate two strategies from both the data and training aspects to alleviate the influence of the label shift. we devise a label shift metric to quantify the degree of label shift that occurs in each explicit example and employ it for sample-level filtering. additionally, we study a joint-learning strategy from the training side to further alleviate the impact of the shift in cases where the filtering results are imperfect. specifically, our classifier jointly learns to recover a connective from arguments and identify a relation based on the recovered connective and arguments. we evaluate the effectiveness of our approach on two datasets: the penn discourse treebank 2.0 (pdtb 2.0, @xcite and 3.0 (pdtb 3.0, @xcite . experiments show that our model improves the performance of explicit to implicit discourse relation recognition, achieving encouraging results on both datasets. furthermore, to test the generalizability of the proposed method, we conduct experiments on the gum dataset @xcite , which is annotated with relations from rhetorical structure theory (rst, @xcite . the results suggest that our filtering mechanism and joint training strategy also help with the explicit to implicit relation classification on the gum dataset. ## related work learning to use lexically-marked examples to classify implicit relations has received continued re-search attention. @xcite train the first classifier for implicit intra-sentential discourse relations using explicitly-marked examples from a raw english corpus, blipp @xcite , and the rst treebank @xcite . @xcite present a similar approach using blipp but focus on sentenceinternal temporal relations. blair-goldensohn et al. @xcite extend this work by refining the training process using parameter optimization, topic segmentation, and syntactic parsing on the gigaword @xcite and pdtb @xcite . these three works are evaluated on test sets constructed in the same manner as the training set and show good performance. @xcite and @xcite investigate the applicability of this approach to real implicit scenarios and find its performance is poor. they claim, based on manual analysis of a few instances, that the linguistic dissimilarities between explicit and implicit examples may be the cause. however, a corpuslevel empirical analysis is not provided to establish how widespread the problem is. more recent work has focused on improving the performance from explicit to implicit discourse relation recognition. @xcite propose to use typical examples with linguistic structure shared between explicit and implicit relations for training. @xcite adopt techniques such as resampling from transfer learning to handle the mismatched label distribution between explicit and implicit corpora. @xcite follow a similar domain adaptation idea but focus on minimizing the distance between representations of explicit and implicit examples with an adversarial training framework. @xcite tackle this task from a distant-supervision perspective. in contrast to the above work, we aim to present a new understanding of the question "why classifiers trained on explicit examples perform poorly in real implicit scenarios" and to provide corpus-level empirical evidence to support our findings. connective information has been widely studied in discourse relation recognition. @xcite train a classifier with connectives in the text as the only features and find it could achieve over 90% accuracy on explicit relation recognition. similarly, many attempts have been made using connectives to improve the recognition performance on implicit relations, including pipeline methods @xcite 2021), multi-task training @xcite , adversarial training @xcite , joint training @xcite , and prompt learning @xcite . our work differs from them in both motivation and application scenarios. for example, we use connective information not as a feature to the classifier but as a filtering mechanism to select good training instances. ## experimental setup we introduce the task of explicit to implicit relation recognition and describe the experimental setup 1 used for analyses in section 4 and improvements in section 6. task. the task of explicit to implicit relation recognition builds an implicit classifier relying on explicit examples. the traditional approach to achieving this is to construct an implicit-like corpus by excluding connectives from explicit examples, and then train a classifier on this corpus with the original explicit relations as ground-truth labels. datasets. the datasets we use for analyses are pdtb 2.0 @xcite and pdtb 3.0 @xcite . pdtbs are corpora annotated with a lexical-based framework where instances are divided into different groups, including the discourse relation categories we focus on: explicit and implicit. this clear grouping makes them very suitable for explicit to implicit relation recognition @xcite since we do not need to distinguish explicit or implicit examples by ourselves. more importantly, the two corpora offer manually annotated connectives for implicit examples (see appendix a.1), facilitating our comparative analysis of explicit and implicit relations. we follow previous work @xcite ) to use pdtb sections 2-20, 0-1, and 21-22 as training, development, and test set, respectively. we conduct experiments on both the top-and second-1 https://github.com/liuwei1206/exp2imp level relations of the two corpora. models. the relation classifier employed in this paper, including models for analysis in section 4 and baselines in section 6, consists of a pre-trained encoder and a linear layer. we follow previous work @xcite to use roberta base as the encoder. we show in appendix b.2 that our findings are consistent across different pre-trained models and sizes. see appendix a for more detailed settings. ## strategies to alleviate the label shift in this section, we introduce two strategies to alleviate the impact of label shift in the task of explicit to implicit relation recognition. ## experiments we conduct experiments to show that our method not only improve the performance of explicit to implicit relation recognition on pdtb 2.0 and 3.0, but also works well on a corpus annotated with rst relations. ## conclusion we find that one cause for the poor transfer performance from explicit to implicit relations is the occurrence of label shift when deleting connectives from explicit examples. we present both manual and empirical evidence to demonstrate the existence of such shift in the explicit corpus. we de-sign a cosine similarity-based metric to measure label shift in the corpus, filter out noisy data, and investigate a joint learning framework to mitigate label shift. experiments on pdtb 2.0 and pdtb 3.0 demonstrate that training classifiers on the filtered corpus with our joint learning strategy can significantly enhance the performance of explicit to implicit relation recognition. furthermore, we show that our approach also works well on the gum dataset, suggesting its generalizability.