The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error (2022)

Abstract: Discourse analysis allows us to attain inferences of a text document that extend beyond the sentence-level. The current performance of discourse models is very low on texts outside of the training distribution’s coverage, diminishing the practical utility of existing models. There is need for a measure that can inform us to what extent our model generalizes from the training to the test sample when these samples may be drawn from distinct distributions. While this can be estimated via distribution shift, we argue that this does not directly correlate with change in the observed error of a classifier (i.e. error-gap). Thus, we propose to use a statistic from the theoretical domain adaptation literature which can be directly tied to error-gap. We study the bias of this statistic as an estimator of error-gap both theoretically and through a large-scale empirical study of over 2400 experiments on 6 discourse datasets from domains including, but not limited to: news, biomedical texts, TED talks, Reddit posts, and fiction. Our results not only motivate our proposal and help us to understand its limitations, but also provide insight on the properties of discourse models and datasets which improve performance in domain adaptation. For instance, we find that non-news datasets are slightly easier to transfer to than news datasets when the training and test sets are very different. Our code and an associated Python package are available to allow practitioners to make more informed model and dataset choices.

## introduction coherence analysis of text is a key area of natural language processing. discourse parsing models are trained on a dataset annotated according to a discourse framework, wherein the discourse structure and how the discourse units are connected ⋆ k. atwell and a. sicilia contributed equally. to estimate the extent of a model's generalizability on a particular train/test pair, common proposals suggest using two-sample statistics which capture distributional shift in the feature space @xcite . however, the working hypothesis of this paper is that changes in feature-distribution do not necessarily equate to changes in a classifier's error; i.e., from train to test sample. figure 1 captures this idea by illustrating some examples in simple 2d-space where domain shift may occur without high error, and vice versa, in the context of discourse parsing. motivated by this hypothesis, we look to existing theoretical domain adaptation literature. we propose to use a statistic which has not only been designed to incorporate information about the classifier we would like to transfer, but has also been shown (theoretically) to directly relate to model performance on the test set. namely, we consider generalization of the source-guided discrepancy @xcite which we call the hdiscrepancy defined for any classifier h (we introduce and define this metric in section 4). we provide novel theoretical analysis of the errors of this statistic in estimating adaptation performance and, based on this, hypothesize this statistic will correlate more substantially with the classifiers' generalization ability than the two-sample statistics previously mentioned. we support this hypothesis by illustrating these correlations across several different widely-used discourse datasets (described in section 3). we also provide a detailed empirical analysis of the estimation error of this statistic in predicting adaptation performance using a regression model. in doing so, we provide insights on the effect of various properties of different discourse models and datasets on performance in domain adaptation, which we enumerate in section 6. we expand on these contributions next. first, we contribute a new theoretical analysis to characterize the bias of the h-discrepancy as an estimator of performance in domain adaptation. although this discrepancy is typically biased, we provide upper and lower bounds on this bias and interpret them to provide insight on the use of this statistic in practice. in particular, we show that a small h-discrepancy often means the practitioner can be confident in transferring the model from the train-to the test-set. our theoretical analysis motivates our hypothesis that the h-discrepancy should outperform common two-sample statistics. next, we empirically study the aforementioned hypothesis. we compare correlation of the hdiscrepancy with performance in domain adaptation against correlation of various two-sample statistics across multiple discourse datasets. as we are aware, this large-scale comparison has never been done for discourse relation classification. as mentioned above, the results of this analysis provide support for our hypothesis that the h-discrepancy is the best estimator of performance changes under domain shift. as such, we argue that computational discourse practitioners should use this statistic to determine the model/dataset likely to maximize performance under domain shift. we also perform a regression analysis of the estimation errors of the h-discrepancy as an estimator for domain adaptation performance. this analysis allows us to understand the properties and pitfalls of our estimator. further, it allows us to gain useful insights into how different types of datasets, genres, feature representations, and models influence the generalizability of discourse parsers. we enumerate these insights and discuss their implications for discourse researchers in section 5. in the sections below, we further discuss and motivate the need for domain-adaptation bounds tied directly to the error gap for more informed insights into performance gaps under domain shift. we hope that discourse researchers use our results, and our code, as a starting point for model and dataset selection in their own studies. ## quantifying meaningful domain shift identifying and quantifying domain shift is a classical problem. perhaps, the most widely used mechanism for this task is the two-sample test; i.e., a test designed to indicate difference of distribution between two samples. we begin this section by discussing a few of the statistics used in these tests. we observe a common problem in using these statistics to predict adaptation performance, and following this, discuss the aforementioned h-discrepancy. ## conclusion this work provides a statistic for model and dataset selection, that we also use to conduct large-scale analysis of model transfer in discourse parsing. our analysis provides useful insights for the practitioner. for one, the correlations indicate that, for datasets with the rst-dt annotation framework, the statistics that quantify distributional shift without being directly tied to error-gap (where error-gap refers to the performance gap between train and test splits) are very weakly correlated with error-gap. this also holds for non-news targets, and indicates that the h-discrepancy is especially useful for predicting the effects of domain shift in these cases. additionally, we find that: (1) increased variability in the target domain appears to make domain adaptation more difficult, even if the training set contains a similar level of variability; (2) s-bert is better than a-bert when domains are similar, but a-bert outperforms s-bert when the domains further diverge; (3) non-news texts (such as those in the biodrb) are easier to adapt to than news texts (such as those in the pdtb). this is the first computational and empirical study that looks at distribution shifts across different discourse datasets and evaluates the performance of various models under these shifts. this is also the first work that examines the efficacy of different two-sample tests for predicting the errorgap when compared to a metric that is theoretically tied to error gap. future work can extend these results by using the h-discrepancy metric to predict the error-gap for other nlp tasks or for other components needed for discourse parsing, such as constructing the rst-dt dataset.