Distributed NLI : Learning to Predict Human Opinion Distributions for Language Reasoning (2022)

Abstract: We introduce distributed NLI, a new NLU task with a goal to predict the distribution of human judgements for natural language inference. We show that by applying additional distribution estimation methods, namely, Monte Carlo (MC) Dropout, Deep Ensemble, Re-Calibration, and Distribution Distillation, models can capture human judgement distribution more effectively than the softmax baseline. We show that MC Dropout is able to achieve decent performance without any distribution annotations while Re-Calibration can give further improvements with extra distribution annotations, suggesting the value of multiple annotations for one example in modeling the distribution of human judgements. Despite these improvements, the best results are still far below the estimated human upper-bound, indicating that predicting the distribution of human judgements is still an open, challenging problem with a large room for improvements. We showcase the common errors for MC Dropout and Re-Calibration. Finally, we give guidelines on the usage of these methods with different levels of data availability and encourage future work on modeling the human opinion distribution for language reasoning.

## introduction natural language understanding (nlu) and reasoning play a fundamental role in natural language processing (nlp) research. it has almost become the de facto rule that newly proposed generic language models will be tested on nlu tasks and progress obtained on general nlu often bring potential improvement on other aspects of nlp research @xcite . the well-known nlu tasks include sentiment analysis @xcite , natural language inference (nli) @xcite , commonsense reasoning @xcite , etc., covering a representative set of problems for nlp. one common practice shared by most of the language understanding and reasoning tasks is that they are formalized as a classification problem, where the model is required to predict a single most preferable label from a predefined candidate set, and the goal is to reverse-engineer how a reasonable human chooses the best one. this simplification not only helps standardize the evaluation, i.e., accuracy could become the canonical measure, but also help make the annotation task more straightforward during crowdsourcing data collection. however, recent findings suggest that inherent disagreements exist in both the natural language inference (nli) and commonsense reasoning datasets @xcite @xcite and advocate that nlu evaluation should explicitly incentivize models to predict distributions of human judgments. similarly, @xcite suggest that nli should account for annotator random effects. this is intuitive since there might be different subjective views of the world and people might think differently given the same reasoning task especially those involving pragmatic reasoning @xcite . modeling the distribution of human opinions provides a higher level "meta-view" of the collective human intelligence which would be valuable for all aspects of nlp applications. in this work, as a case study for learning the distribution of human judgements on nlu, we extend the nli task to distributed nli -a new task in which models are required to predict the distribution of human judgements for natural language inference. we introduce the new task based on the data from prior works @xcite with new experimental guidelines designed for the distribution annotations. standard nlp models are trained towards predicting single labels, while in theory models trained on single labels should still be able to capture the whole label distribution (see appendix e.3 for a more detailed discussion), their predicted distribution may not be reliable @xcite . to achieve better distribution estimation and to maintain the merits of sota models, we consider four distribution estimation methods that do not need major architecture changes, namely, mc dropout @xcite , deep ensemble @xcite , re-calibration @xcite , and distribution distillation for distributed nli. these methods have achieved empirical success in estimating the aleatoric uncertainty @xcite , calibrating the neural network prediction confidence @xcite , and neural network knowledge distillation @xcite , respectively. we show that all four methods can substantially outperform the baseline and that re-calibration and distribution distillation can provide further improvement by making use of additional distribution annotations. specifically, our primary contributions are: ## dataset and experiment design in this section, we describe a typical design of dataset and experiment of the distributed nli task, and is used in later experiments in this work. for a typical nli task, the dataset is split into train, development, and test set where each example is associated with one ground truth label. the model will be trained using examples in the training set. accuracy on the development set is used for model selection, and accuracy on the test set will be reported as the final metric. for distributed nli, in order to develop models that can predict the label distribution, we assume that each example in the test set will also have a sufficient amount of human labels to approximate the real human label distribution to evaluate the model's prediction. let us define the d train , d dev , d dev s , and d test s to be the different splits of the dataset. the subscript s in d dev s and d test s indicates that the examples in these two splits have soft labels representing the human label distribution, while there is no such label in d train and d dev . d dev s is a very small set of examples with soft-labels besides the test set. this gives a good simulation for real production because in practice, d dev s will be extremely scarce. the goal of the distributed nli is to develop models that can predict human label distributions and minimize the average divergence between predicted label distributions and approximated human label distributions on the test set using examples in d train , d dev , and d dev s . 5 even though obtaining these soft label distributions is expensive, our design can generalize to the situation where we can also have enough training data with soft-label by simply making a new split d train s on which the model can be trained. ## distribution estimation the output of a typical nli classifier is a vector @xmath0. in modern nli models, the classifier is usually a deep neural network and the final output is: ŷ = softmax(z), ĉ = arg max(z) where ŷ is the normalized label distribution whose element ŷi = e z i / e z i , and ĉ is the predicted label. prior works @xcite revealed that the distribution ŷ produced by the softmax layer gives a poor estimation on the real human label distribution. in this work, we experiment on using distribution estimation methods for predicting human opinion distribution on nli, and we show that they can achieve better performance than the softmax output. these methods have been used in uncertainty estimation and confidence calibration with some empirical success. although the problem of uncertainty estimation is different from opinion distribution estimation, the essence of the two are the same -the estimation of a distribution. foot_2 ## results the performances of different distribution estimation methods are shown in in what follows, we explain the main findings. mc dropout is the most preferable method without additional soft-labeled data. the first thing we can observe from the first group (row 2-5) in further improvement can be obtained by using soft-labeled data, but still below estimated human upper-bound. 11 from the second group (row 6-8) of in-domain improvements hold on the out-ofdomain set. correlation exists between opinion distribution and graded individual judgement. as explained in sec. 2.3 and 5.1, the distribution of human opinions on nli examples is different from the individual graded plausibility judgement. in ## related work inherent disagreement and ambiguity in nlp annotations has a long history @xcite involving tasks like coreference resolution @xcite @xcite , pos-tagging @xcite @xcite , semantic frame disambiguation @xcite , humorousness prediction @xcite , etc. most previous works design methods to predict one single gold label by aggregating the noisy information @xcite @xcite @xcite . on the contrary, following the recent definition in nli works @xcite @xcite , we directly try to predict distribution labels that accurately reflect the opinion of a large population. @xcite is most similar to us in label definition, and studied the advantage of using distribution labels in image classification. uncertainty and calibration have also been studied in various nlp models, from traditional structured prediction models @xcite , to seq-to-seq models @xcite @xcite and transformers @xcite . @xcite suggests a constructive view of nli modeling in which the prediction is explicitly grounded on annotator identifiers, incorporating the annotator random effects. @xcite propose an ensemble-based framework that can identify examples with high label disagreement. xiao and wang (2019) shows that explicitly modeling the uncertainty can improve performance, and @xcite propose a label smoothing method that improves calibration for nmt. instead of aiming for a better uncertainty, our work uses multiple uncertainty estimation methods for more accurate distribution prediction. concurrently, meissner et al. (2021) explores training models directly on the multiple labels from each annotator in snli and mnli, @xcite also leverage distribution labels in the model development process and explore training methods combining the supervision signal of one-hot and distributional labels. in comparison, our work studies additional bayesian estimation methods and provides a detailed discussion on why and how modeling distribution labels is beneficial for nlu, including the motivation, nuances, and evaluation standardization. ## conclusion we introduce distributed nli -an extension of nli with a new goal of predicting human opinion distribution. we show that several distribution estimation methods can capture such distributions more effectively than softmax, but the best results are still far below the estimated upper-bound. we analyze the properties and weaknesses of the methods, highlight the importance of the task, and encourage future work on developing better models for estimating the human opinion distribution.