Multitask Semi-Supervised Learning for Class-Imbalanced Discourse Classification (2021)

Abstract: As labeling schemas evolve over time, small differences can render datasets following older schemas unusable. This prevents researchers from building on top of previous annotation work and results in the existence, in discourse learning in particular, of many small class-imbalanced datasets. In this work, we show that a multitask learning approach can combine discourse datasets from similar and diverse domains to improve discourse classification. We show an improvement of 4.9% Micro F1-score over current state-of-the-art benchmarks on the NewsDiscourse dataset, one of the largest discourse datasets recently published, due in part to label correlations across tasks, which improve performance for underrepresented classes. We also offer an extensive review of additional techniques proposed to address resource-poor problems in NLP, and show that none of these approaches can improve classification accuracy in our setting.

## introduction learning the discourse structure of a text has been shown to be helpful for diverse tasks such as event extraction @xcite , sentiment analysis @xcite , natural language generation @xcite , summarization @xcite , storyline discovery @xcite , and even misinformation detection @xcite . however, even as recent advances in nlp allow us to achieve impressive results across a variety of tasks, discourse learning, a supervised learning task, faces the following challenges: (1) discourse datasets tend to be very class-imbalanced. 2 (2) discourse learning is a complex task: human annotators require training and conferencing to achieve moderate agreement @xcite . (3) discourse learning tends to be resource-poor, as annotation complexities make large-scale data collection challenging ( we propose a multitask neural architecture (section 2) to address this hypothesis. we construct tasks from 6 discourse datasets, an events dataset, and an unlabeled news dataset (section 3), including a novel discourse dataset we introduce in this work. although different datasets are developed under divergent schemas and have different goals, our framework learns correlations between schemas, and does not "waste" labeling work done by generations of nlp researchers. our experiments show that a multitask approach can help us improve discourse classification on a primary task, newsdiscourse @xcite , from a baseline performance of 62.8% micro f1 to 67.7%, an increase of 4.9 points (section 4), with the biggest improvements seen in underrepresented classes. on the contrary, two data augmentation approaches, training data augmentation (tda) and unsupervised data augmentation (uda), fail to improve performance. we give insight into why this occurs (section 5). in the multitask approach, the primary task's underpresented labels are correlated with labels in other datasets. however, if we only provide more data without any correlated labels (tda and uda), we overpredict the overrepresented labels. we test many other approaches proposed to address classimbalance and observe similar negative results (appendix f). taken together, this analysis indicates that the signal from labeled datasets is essential for boosting performance in class-imbalanced settings. in summary, our core contributions are: ## methodology we formulate a multitask approach to discourse learning with the newsdiscourse dataset as our primary task (section 3). our multitask architecture uses shared encoder layers and task-specific classification heads foot_3 . our objective is to minimize the weighted sum of losses across tasks: ## datasets we use 8 datasets in our multitask setup, shown in a substantial volume of news discourse is not factual assertion, but analysis, explanation, and prediction @xcite . we thus include the argumentation dataset (arg) @xcite , a dataset consisting of 5 labels applied to 300 news editorials. 12 each of these four datasets assigns a single label to each sentence. we treat them as multiclass datasets, as shown in some of van dijk's discourse elements differ 11 for more information on the dataset we introduce in this paper, see appendix b. 1. 12 this dataset contains articles from 3 news outlets: aljazeera.com, foxnews.com and theguardian.com 13 for more details, see appendix b.2. 14 specifically, if pm(n) and pa(n) are the likelihood of a document d with n sentences in the main and auxiliary datasets respectively, we sample with weight w @xmath0. pm(n) and pa(n) were determined empirically by nn/n total (nn: # of docs with sentencelength n in a or m, n total : # of docs in a or m). when processed as described above, each of these datasets assign multiple labels to each sentence. we treat them as multilabel datasets. knowledge base population (kbp) 2014/2015 some of van dijk's discourse elements differ based on the presence or absence of an event. for example, the elements previous event (c2) and current context (c1) both describe the context before a main event, but the former describes events while the latter describes general circumstances. we hypothesize that a dataset identifying event occurrence can help our model differentiate these elements. we collect an additional non-discourse dataset, the kbp 2014/2015 event nugget dataset, which annotates the trigger words for events by event-type. we treat this as a multilabel dataset. all-the-news (u) for semi-supervised dataablation experiments, described in section 4.3, we sample 6,000 documents from an unlabeled news dataset. 15 we downsample in the manner described above for pdtb and rst. ## experiments and results in this section, we briefly discuss experiments using vd2 as a single classification task. then, we discuss the experiments using vd2 in a multitask setting. finally, we discuss our experiments with data augmentation as ablations. we leave a more detailed analysis of single-task experiments for appendix f, focusing here on multi-task experiments. ## discussion as shown in figure 3 , a multitask approach significantly increases performance for underrepresented classes while not hurting performance for others. this is in contrast to pure data augmentation approaches, like uda or tda. improving performance in low-support classes improves overall macro f1, as expected, and micro f1 ( multitask learning can help learn part of the data manifold where an underrepresented class exists by learning signal from a class which is correlated. ta- ## conclusion we have shown a state-of-the-art improvement of 4.9 micro f1-score above baseline, from 62.8% f1-score to 67.7% f1-score, for discourse tagging on the newsdiscourse dataset, the largest dataset currently available for van dijk discourse tagging. this dataset has a number of challenges: distinctions between discourse labels are complex and multifaceted and this dataset is class-imbalanced, with the overrepresented classes being 3 times more likely than the underrepresented classes. we showed that a multitask approach is especially helpful in this circumstance, improving performance for underrepresented labels. one reason might be the high correlations observed between label predictions between tasks, indicating that auxiliary tasks are giving signal to our primary task's underrepresented labels. this includes a novel dataset that we introduce based on the same schema with some minor alterations. we show an additional benefit that our approach can reconcile datasets with slightly different schema, allowing nlp researchers not to "waste" valuable annotations. finally, we perform a comparative analysis of other strategies proposed in the literature for dealing with small datasets or class-imbalanced problems. we show in exhaustive experiments, in appendix f, that these approaches do not help us improve above baseline. these negative experiments include extensive analyses and provide a justification for the necessity of our multitask approach.