The Role of Context and Uncertainty in Shallow Discourse Parsing (2022)

Abstract: Discourse parsing has proven to be useful for a number of NLP tasks that require complex reasoning. However, over a decade since the advent of the Penn Discourse Treebank, predicting implicit discourse relations in text remains challenging. There are several possible reasons for this, and we hypothesize that models should be exposed to more context as it plays an important role in accurate human annotation; meanwhile adding uncertainty measures can improve model accuracy and calibration. To thoroughly investigate this phenomenon, we perform a series of experiments to determine 1) the effects of context on human judgments, and 2) the effect of quantifying uncertainty with annotator confidence ratings on model accuracy and calibration (which we measure using the Brier score (Brier et al, 1950)). We find that including annotator accuracy and confidence improves model accuracy, and incorporating confidence in the model’s temperature function can lead to models with significantly better-calibrated confidence measures. We also find some insightful qualitative results regarding human and model behavior on these datasets.

## introduction the context of an utterance influences the interpretation. linguistics, philosophy, cognitive science and neuroscience @xcite @xcite studies have shown the effect of context in text interpretation. in this paper, we hypothesize that context can influence the prediction of discourse relation labels. motivating the need to discover the effects of context on annotation accuracy is the following example, where argument 1 is bolded and argument 2 is italicized: this added context makes it more apparent that a causal relation holds between the two arguments, and thus our annotator chose the correct relation, contingency.cause, after being presented with this context. while the annotation process of discourse corpora such as the penn discoruse treebank (pdtb) exposes annotators to the full context of the document, we do not yet understand how human annotation behavior would change if context were limited. additionally, context is limited for most of the models built for automatic discourse relation classification. we start with a basic question: is the argument pair (where arguments are defined as the minimum span a relation could be interpreted in pdtb @xcite ) enough to determine the discourse relation? in addition, context may affect human annotation to varying degrees and impact annotators' confidence in their judgements, how can we make sure that this information is factored into discourse parsers and model confidence? for the first time, we propose to study and measure the human annotator's confidence and incorporate it into the architecture of the deep-learning-based discourse parsers. we utilize our human-annotated confidence scores to predict human confidence, and test whether this method improves model accuracy and calibration (how well the predicted probabilities produced by a model reflect the true likelihood of the corresponding events to occur in the studied population). model calibration is an important issue in modern neural networks @xcite , and to our knowledge we are the first to study it for discourse relation classification. properly calibrating a model (i.e. properly quantifying uncertainty) is especially important for this task, because determining the correct discourse sense involves a large degree of uncertainty, and more correctly quantifying uncertainty allows a model to be more explainable. our two main research questions, as described above, are illustrated in figure 1 , and our contributions can be summarized as the following: our code can be found here foot_0 . the effects of context on implicit sense classification as mentioned above, implicit sense classification is a very challenging task. further, most implicit sense classifiers @xcite do not include context outside of the two arguments contained in a discourse relation, foot_1 despite the annotators having access to context during the pdtb annotation task, wherein the annotator inserts a connective between the two arguments and then determines the discourse relation. an example of this connective insertion from the pdtb is as follows, with arg1 in bold and arg2 in italics: however, no paper has yet studied the effect of context on the discourse annotation task, nor has a work attempted to use insights from annotators' proficiency and confidence on the model. the closest work to ours in this space is scholman and demberg (2017), who use a connective insertion task to crowdsource discourse sense annotations and examine the effect of context on this task. they find that under certain conditions (such as when argument 1 refers to an entity/event in the surrounding context or the sentence after argument 2 expands on argument 2), the presence of context can improve annotator agreement. however, this annotation task is simplified and only covers 6 level-2 relations. thus, their result is not fully representative of the pdtb annotation task. we wish to examine this question more in-depth by presenting annotations using the traditional pdtb annotation task to a trained linguist in settings with and without additional context, and comparing these annotations against ground truth data. in doing so, we believe we can gain more insight into factors that affect human understanding in more similar conditions to the ones present in the original pdtb annotation task. calibration of neural networks for discourse parsing calibration in machine learning refers to the distribution of error and the model's level of self-assessment, or confidence @xcite . it was found that neural networks tend to be badly calibrated @xcite , which can result in poor explainability and uncertainty quantification. however, pretrained models, even very complex ones, were found to generally be more wellcalibrated, and to benefit from temperature scaling @xcite . therefore, we use annotator confidence scores to scale the temperature according to a the example's simplicity (as perceived by the annotator). in our work, we choose to use the brier score @xcite to measure calibration because it is a proper scoring function. as far as we are aware, we are the first paper to study the calibration of implicit sense classification models and the impact of using annotator confidence measures to improve model calibration. ## modeling insights above, we illustrate some insights attained from the annotated data with respect to changes in domain and access to context. in this section, we use both the annotator correctness and confidence metrics to inform our model decisions, in order to determine whether annotator performance in any way correlates with model performance. in addition to model accuracy, we evaluate model calibration scores when the model is and is not given access to annotator confidence. to influence the model's decisions when given access to annotator confidence, we adjust the training and validation mechanisms accordingly. we hypothesize that access to annotation metrics will improve model accuracy, and that changing the temperature function with respect to annotator confidence will improve model calibration. we describe the setup for each of our experiments below, and report our results in section 4.2. ## conclusion in the previous sections, we first show the effects of presenting additional context when compared to providing only the two arguments. we find that adding context has an overall positive impact on annotator accuracy and confidence. more specifically, we find that context helps for certain discourse relations, but not all relations. we hypothesize that in cases where adding context worsens annotation, it is because context may add confusing information about argument relationship (such as whether one is a quote) or the completeness of a list of items in a particular set (blurring the line between senses such as instantiation and level-of-detail). we believe this is worth exploring further. secondly, we find that utilizing the human performance metrics we collected in the first half of the paper yielded better model performance when compared to the baseline, suggesting that these metrics give the model some useful information about its own predictions. in particular, we find that using confidence scores to adjust the training weights improves model accuracy, while using them to adjust the softmax temperature improves model calibration, the latter of which is important for explainability and for tasks with a high degree of uncertainty (discourse relation classification being one such task). to our knowledge, ours is the first work to study calibration with respect to discourse models. we hope that future work will continue to study the role of context in discourse relation classification, as well as model calibration for this task. we will release our annotations and model code upon the publication of this paper.