Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition (2025)

Abstract: Implicit discourse relation recognition (IDRR) – the task of identifying the implicit coherence relation between two text spans – requires deep semantic understanding. Recent studies have shown that zero-/few-shot approaches significantly lag behind supervised models. However, LLMs may be useful for synthetic data augmentation, where LLMs generate a second argument following a specified coherence relation. We applied this approach in a cross-domain setting, generating discourse continuations using unlabelled target-domain data to adapt a base model which was trained on source-domain labelled data. Evaluations conducted on a large-scale test set revealed that different variations of the approach did not result in any significant improvements. We conclude that LLMs often fail to generate useful samples for IDRR, and emphasize the importance of considering both statistical significance and comparability when evaluating IDRR models.

## introduction idrr is the task of identifying the covert discourse relation (drs) between two given text spans (the arguments: arg1 and arg2) in the absence of a specific discourse connective (dc) such as because or moreover. explicit labelling of discourse structure is beneficial to inform llms in summarization tasks @xcite @xcite @xcite . however, idrr is challenging both for humans @xcite and models (sota, 56.50% f1 and 64.87% accuracy in @xcite , particularly in a cross-domain setting @xcite @xcite . prompting of large pre-trained language models (llms), despite the human-or even superhumanlevel performance in various reasoning tasks (e.g., @xcite @xcite , was found to be not successful in idrr. few-shot prompting using gpt-4 only reaches 30.90% f1 and 29.40% accuracy on the penn discourse treebank (pdtb 3.0, @xcite , and 28.87% f1 and 32.67% accuracy on the multi-domain discogem corpus @xcite , even with task-specific prompt engineering @xcite @xcite . on the other hand, a previous study demonstrated that llms can instead be used to generate synthetic data to augment the pdtb 3.0, improving the performance of classes that the baseline struggles to predict @xcite . this work explores the application of synthetic data augmentation for cross-domain idrr. using raw texts from different target domains, we prompt llms to generate discourse continuations that express specific drs. the generated data is then used to adapt the base model trained on humanannotated data of the source domain, which is the pdtb 3.0. we experimented with outputs of different llms, prompt templates, screening strategies and various advanced methods of domain adaptation, and evaluated on the entire discogem corpus, which is 4 times the size of the test set used in previous work @xcite . we only found marginal differences between models adapted to target-domain synthetic data compared with the base model. the benefit of generating synthetic data is unclear compared with direct application of the cross-domain base model, or using it to pseudolabel target-domain data. we have derived the following insights from the experimental results: ## discussion and conclusion contrary to the improvements reported in previous studies, our results did not confirm the benefits of synthetic data augmentation for idrr in a cross-domain setting. this is in line with recent reports that synthetic samples generated by llms do not improve abstract and ambiguous tasks, such as irony detection @xcite . we also found high variance in the model performance. in particular, the f1 scores of the rare classes are unlikely to be significant due to the skewed data distribution. the evaluation methods for instances with multiple gold labels also vary across studies, leading to inconsistencies. for example, while some works, including the current study, discard unmatched alternative gold labels; other works, such as @xcite , count them as true positives in the f1 calculation. significant differences from the baseline model, based on variations across runs, are marked with * . the synthetic data sizes of the domain-specific models are averaged across the models of the three domains. in fact, interpretation of implicit drs involves deep cognitive pro-cessing that is universally difficult for humans @xcite @xcite . the major reason is that multiple interpretations are often possible based on the perspectives of the readers @xcite . we manually annotated a random subset of 100 synthetic samples and found a 65% agreement with the intended drs in the prompts. 28 of the 35 disagreement were valid alternative interpretations. this shows that the generated drs are valid but less ambiguous than natural ones, which often lack clear cues and allow multiple plausible readings. figure 1 provides several examples: while alternative discourse relations can be inferred from the genuine dr samples, the synthetic examples tend to be clearer and more straightforward. the limited effectiveness of synthetic data for idrr may therefore be explained by the perspectivist nature of dr inference. since prompting llms to generate specific relations tends to bias outputs of prototypical cases, a potential improvement could involve prompting examples that reflect multiple plausible senses, combined with cooccurrence-aware, multi-label screening. future