Uncertainty-Aware Natural Language Inference with Stochastic Weight Averaging (2023)

Abstract: This paper introduces Bayesian uncertainty modeling using Stochastic Weight Averaging-Gaussian (SWAG) in Natural Language Understanding (NLU) tasks. We apply the approach to standard tasks in natural language inference (NLI) and demonstrate the effectiveness of the method in terms of prediction accuracy and correlation with human annotation disagreements. We argue that the uncertainty representations in SWAG better reflect subjective interpretation and the natural variation that is also present in human language understanding. The results reveal the importance of uncertainty modeling, an often neglected aspect of neural language modeling, in NLU tasks.

## introduction arguably, human language understanding is not objective nor deterministic. the same utterance or text can be interpreted in different ways by different people depending on their language standards, background knowledge and world views, the linguistic context, as well as the situation in which the utterance or text appears. this uncertainty about potential readings is typically not modeled in natural language understanding (nlu) research and is often ignored in nlu benchmarks and datasets. instead, they usually assign a single interpretation as a gold standard to be predicted by an artificial system ignoring the inherent ambiguity of language and potential disagreements that humans arrive at. some datasets like snli @xcite ) and mnli @xcite do, however, contain information about different readings in the form of annotation disagreement. these datasets include the labels from five different rounds of annotation which show in some cases clear disagreement about the correct label for the sentence pair. those labeling discrepancies can certainly be a result of annotation mistakes but more commonly they arise from differences in understanding the task, the given information and how it relates to world knowledge and personal experience. moving towards uncertainty-aware neural language models, we present our initial results using stochastic weight averaging (swa) @xcite and swa-gaussian (swag) @xcite on the task of natural language inference. swag provides a scalable approach to calibrate neural networks and to model uncertainty presentations and is straightforward to apply with standard neural architectures. our study addresses the two main questions: in this paper, we first test the performance of swa and swag in snli and mnli tasks. we then study if adding weight averaging improves the generalization power of nli models as tested through cross-dataset experiments. finally, we analyse the probability distributions from swa and swag to test how well the model uncertainty corresponds to annotator disagreements. ## experiments we test the performance of swa and swag on the natural language inference task using three nli datasets, including cross-dataset experiments, and study the effect on both hard and soft labeling. code for replicating the experiments is available on github: https://github.com/helsi nki-nlp/uncertainty-aware-nli ## conclusions our results show that weight averaging provides consistent and clear improvement for both snli and mnli datasets. the cross-dataset results are slightly mixed but also show the trend of improved cross-domain generalization. finally, we demonstrate a clear increase in the correlation with human annotation variance when comparing swag with non-bayesian approaches. for future work we consider making use of multiple annotations also during training and extensions of swag such as multiswag @xcite . we also plan to test the methods on different nlu datasets, especially those with a high number of annotations (e.g. @xcite , and compare the annotation variation and system predictions in more detail. finally, in our future work we will explore other uncertainty modeling techniques, like mc dropout @xcite , in nlu and see how they compare with stochastic weight averaging techniques.