Out-of-Domain Discourse Dependency Parsing via Bootstrapping: An Empirical Analysis on Its Effectiveness and Limitation (2022)

Abstract: Discourse parsing has been studied for decades. However, it still remains challenging to utilize discourse parsing for real-world applications because the parsing accuracy degrades significantly on out-of-domain text. In this paper, we report and discuss the effectiveness and limitations of bootstrapping methods for adapting modern BERT-based discourse dependency parsers to out-of-domain text without relying on additional human supervision. Specifically, we investigate self-training, co-training, tri-training, and asymmetric tri-training of graph-based and transition-based discourse dependency parsing models, as well as confidence measures and sample selection criteria in two adaptation scenarios: monologue adaptation between scientific disciplines and dialogue genre adaptation. We also release COVID-19 Discourse Dependency Treebank (COVID19-DTB), a new manually annotated resource for discourse dependency parsing of biomedical paper abstracts. The experimental results show that bootstrapping is significantly and consistently effective for unsupervised domain adaptation of discourse dependency parsing, but the low coverage of accurately predicted pseudo labels is a bottleneck for further improvement. We show that active learning can mitigate this limitation.

## introduction discourse parsing aims to uncover structural organization of text, which is useful in natural language processing (nlp) applications such as document summarization @xcite @xcite @xcite ), text categorization @xcite , question answering @xcite , and information extraction @xcite . in particular, dependency-style representation of discourse structure has been studied intensively in recent years @xcite @xcite @xcite . figure 1 shows an example of discourse dependency structure, which is recorded in covid-19 discourse dependency treebank (covid19-dtb), a new manually annotated resource for discourse dependency parsing of biomedical abstracts. state-of-the-art discourse dependency parsers are generally trained on a manually annotated treebank, which is available in a limited number of domains, such as rst-dt @xcite for news articles, scidtb @xcite for nlp abstracts, and stac @xcite and molweni @xcite for multi-party dialogues. however, when the parser is applied directly to out-of-domain documents, the parsing accuracy degrades significantly due to the domain shift problem. in fact, we normally face this issue in the real world because human supervision is generally scarce and expensive to obtain in the domain of interest. unsupervised domain adaptation (uda) aims to adapt a model trained on a source domain, where a limited amount of labeled data is available, to a target domain, where only unlabeled data is available. bootstrapping (or pseudo labeling) has been shown to be effective for the uda problem of syntactic parsing @xcite @xcite . in bootstrapping for syntactic parsing, we first train a model on the labeled source sentences, the model is used to give pseudo labels (i.e., parse trees) to unlabeled target sentences, and then the model is retrained on the manually and automatically labeled sentences. on the contrary, despite the significant progress achieved in discourse parsing so far @xcite @xcite @xcite an example of discourse dependency structure for a covid-19 related biomedical paper abstract @xcite , which we manually annotated for our new dataset. @xcite , bootstrapping for the uda problem of discourse parsing is still not well understood. @xcite and @xcite explored how to enrich the labeled dataset using bootstrapping methods; however, their studies are limited to the in-domain setup, where the labeled and unlabeled datasets are derived from the same domain. in contrast to these studies, we focus on the more realistic and challenging scenario, namely, out-of-domain discourse parsing, where the quality and diversity of the pseudo-labeled dataset become more crucial for performance enhancement. in this paper, we perform a series of analyses of various bootstrapping methods in uda of modern bert-based discourse dependency parsers and report the effectiveness and limitations of these approaches. figure 2 shows an overview of our bootstrapping system. specifically, we investigate self-training @xcite , co-training @xcite , tri-training @xcite , and asymmetric tri-training @xcite of graph-based and transition-based discourse dependency parsing models, as well as confidence measures and sample selection criteria in two adaptation scenarios: monologue adaptation between scientific disciplines and dialogue genre adaptation. we show that bootstrapping improves out-of-domain discourse dependency parsing significantly and consistently across different adaptation setups. our analyses also reveal that bootstrapping has a difficulty in creating pseudo-labeled data that is both diverse and accurate, which is a current limiting factor in further improving accuracy, and furthermore it is difficult to boost the coverage by simply increasing the number of unlabeled documents. we show that an active learning approach can be an effective solution to the limitation. 1 the rest of this paper is organized as follows: section 2 provides an overview of related studies. section 3 clarifies the problem and describes the methodology: bootstrapping algorithms, discourse dependency parsing models, confidence measures, and sample selection criteria. section 4 describes the details of covid19-dtb. section 5 describes the experimental setup. in section 6 we report and discuss the experimental results and provide practical recommendations for out-of-domain discourse dependency parsing. finally, section 7 concludes the paper. ## related work various discourse parsing models have been proposed in the past decades. for constituencystyle discourse structure like rst @xcite , the parsing models can be categorized into the chart-based approach @xcite @xcite , which finds the globally optimal tree using an efficient algorithm like dynamic programming, or the transition-based (or sequential) approach @xcite @xcite @xcite @xcite @xcite , which builds a tree incrementally by performing a series of decisions. for dependency-style discourse structure like the rst variants @xcite @xcite or segmented discourse representation theory @xcite , the models can also be categorized into the graphbased approach @xcite @xcite or the transition-based (sequential) approach @xcite @xcite . recently, pre-trained transformer encoders such as bert @xcite and spanbert @xcite have been shown to greatly improve discourse parsing accuracy @xcite . in this paper, we are not aiming at developing novel parsing models. instead, we aim to investigate the effectiveness and limitations of bootstrapping methods for adapting the modern bert-based discourse parsers. manually annotated discourse treebanks are significantly scarce, and their domains are limited. for example, the most popular discourse treebank, rst-dt @xcite , contains only 385 labeled documents in total. to address the lack of large-scale labeled data, a number of semi-supervised, weakly supervised, and unsupervised techniques have been proposed in the discourse parsing literature. @xcite proposed a semi-supervised method that utilizes unlabeled documents to expand feature vectors in svm classifiers in order to achieve better generalization for infrequent discourse relations. @xcite and @xcite proposed to exploit document-level class labels (e.g., sentiment) as distant supervision to induce discourse dependency structures from neural attention weights. @xcite investigated a data programming paradigm @xcite , which uses rule-based labeling functions to automatically annotate unlabeled documents and trains a generative model on the weakly supervised data. @xcite and @xcite proposed fully unsupervised discourse constituency parsers, which can produce only tree skeletons and rely strongly on pre-trained word embeddings or human prior knowledge on document structure. technically most similar to our work, @xcite and @xcite proposed to enlarge the training dataset using a combination of multiple parsing models. @xcite used co-training for enlarging the rst-dt training set with 2,000 wall street journal articles, with a focus on improving classification accuracy on infrequent discourse relations. @xcite proposed to exploit discourse subtrees that are agreed by two different models for enlarging the rst-dt training set. interestingly, their proposed methods improved the classification accuracy especially for infrequent discourse relations. these studies mainly assume the in-domain scenario and focus on enlarging the labeled set (e.g., rst-dt training set) using in-domain unlabeled documents, and the system evaluation is generally performed on the same domain with the original labeled set (e.g., rst-dt test set). in this paper, instead, we particularly focus on the uda scenario, where the goal is to parse the target-domain documents accurately without relying on human supervision in the target domain. we believe this research direction is important for developing usable discourse parsers, because a target domain to which one would like to apply a discourse parser is normally different from the domains/genres of existing corpora, and manually annotated resources are rarely available in most domains/genres. ## covid19-dtb we release a new discourse dependency treebank for scholarly paper abstracts on covid-19 and related coronaviruses like sars and mers in order to test unsupervised domain adaptation of discourse dependency parsing. we name our new treebank covid-19 discourse dependency treebank (covid19-dtb). ## experimental setup datasets we evaluated the bootstrapping methods on two uda scenarios: the first setup was a monologue adaptation between scientific disciplines: nlp and biomedicine (especially on covid-19), which is actually an important scenario because there is still no text-level discourse treebank on biomedical documents. we used the training split of scidtb @xcite as the labeled source dataset, which contains 742 manual discourse dependency structures on the abstracts in acl anthology. we also used the 2020 september snapshot of cord-19 @xcite as the unlabeled target dataset, which contains about 76,000 biomedical abstracts. we used the development and test splits of covid19-dtb for validation and testing, respectively. the discourse relation labels in the scidtb training set were mapped to the corresponding classes of covid19-dtb. we mapped textual-organization relations in covid19-dtb to elaboration, because there is no corresponding class in scidtb. we also mapped temporal relations in the two datasets to condition to reduce the significant class imbalance. the second setup was an adaptation across dialogue genres, that is, dialogues in a multi-party game and dialogues in ubuntu forum. we used the training split of stac @xcite as the labeled source dataset, which contains 887 manually labeled discourse dependency structures on multi-party dialogues in the game, the settlers of catan. we also used the ubuntu dialogue corpus @xcite as the unlabeled target dataset, which contains dialogues extracted from the ubuntu chat logs. we retained dialogues with 7-16 utterances and 2-9 speakers. we also removed dialogues with long utterances (more than 20 words). finally, we obtained approximately 70,000 dialogues. we used the development and test splits of molweni @xcite for validation and testing. each split contains 500 manually labeled discourse dependency structures on multi-party dialogues derived from the ubuntu dialogue corpus. the unlabeled target documents in both setups were segmented into edus using a publicly available edu segmentation tool @xcite . evaluation we employed the traditional evaluation metrics in dependency parsing literature, namely, labeled attachment score (las) and unlabeled attachment score (uas). we also used root accuracy (ra), which indicates how well a system can identify the most representative edu in the document (i.e., the dependent of the special root node). implementation details as the pre-trained transformer encoders, we used scibert @xcite and spanbert @xcite in the first and second adaptation setups, respectively. the dimensionality of the mlps in the arc-factored model and the shift-reduce models are 100 and 128, respectively. we used adamw and adam optimizers for optimizing the transformer's parameters (θ bert ) and the task-specific parameters (θ task ), respectively, following @xcite . we first trained the base models on the labeled source dataset using the following hyper-parameters: batch @xmath0, learning rate (lr) for θ @xmath1, lr for θ @xmath2, warmup @xmath3.4k. then, we ran the bootstrapping methods using the models with: batch @xmath4, lr for θ @xmath5, lr for θ @xmath6, warmup @xmath7. we trained all approaches for a maximum of 40 epochs. we applied early stopping when the validation las does not increase for 10 epochs. we verified the effectiveness of bootstrapping methods on the two uda scenarios. we evaluated the source-only models, which were trained only on the labeled source dataset, as the baseline. these results indicate that bootstrapping is significantly and consistently effective for uda of discourse dependency parsing in various adaptation scenarios, and that employing multiple models is particularly effective in reducing the unintended tendency of st to amplify its own errors. next, we further analyzed in what kind of documents the bootstrapping system is particularly effective. we divided the covid19-dtb test set into bins by the number of edus in each document (@xmath10, 15 < @xmath11, @xmath12, and for each bin we examined the percentage of examples improved by the bootstrapping systems over the source-only system. figure 4 shows the results. when the document length was 10 or shorter, there was no improvement in most examples; however, when the length was longer than 30, the percentage was jumped to around 80% with ct, tt, and at. these results indicate that the longer the documents (or maybe the higher the document complexity) in the target domain, the greater the benefit of bootstrapping. we also investigated the importance of employing different types of parsing models in ct. the theoretical importance of employing models with different views (or inductive biases) in ct has been discussed @xcite @xcite . we trained base models with the same neural architecture but with different initial parameters on the labeled source dataset, and then retrained them using ct. we can see from the results in ## conclusion in this paper, we investigated the effectiveness and limitation of bootstrapping methods in unsupervised domain adaptation of bert-based discourse dependency parsers. the results demonstrate that bootstrapping is effective significantly and consistently in various adaptation scenarios. however, regardless of the tuned confidence measures and sample selection criteria, the bootstrapping methods have a difficulty in generating both diverse and accurate pseudo labels, which is the current limiting factor in further improvement. this low-coverage problem cannot be mitigated by just increasing the unlabeled corpus size. we confirmed that the active learning can be the effective solution to this problem. we have a limitation in this study: our experiments use only english documents. although bootstrapping and discourse parsing models are language-independent at the algorithmic level, experiments of domain adaptation require labeled datasets on both source and target domains for training and evaluation. in order to investigate the universality and language-dependence features of the bootstrapping methods in various languages, it is necessary to develop discourse treebanks in a variety of languages. in the future, we will expand the covid19-dtb dataset with additional biomedical abstracts to facilitate the exploration and application of discourse parsing technologies to biomedical knowledge acquisition.