  We introduce distributed NLI, a new NLU task with a goal to predict the
distribution of human judgements for natural language inference. We show that
models can capture human judgement distribution by applying additional
distribution estimation methods, namely, Monte Carlo (MC) Dropout, Deep
Ensemble, Re-Calibration, and Distribution Distillation. All four of these
methods substantially outperform the softmax baseline. We show that MC Dropout
is able to achieve decent performance without any distribution annotations
while Re-Calibration can further give substantial improvements when extra
distribution annotations are provided, suggesting the value of multiple
annotations for the example in modeling the distribution of human judgements.
Moreover, MC Dropout and Re-Calibration can achieve decent transfer performance
on out-of-domain data. Despite these improvements, the best results are still
far below estimated human upper-bound, indicating that the task of predicting
the distribution of human judgements is still an open, challenging problem with
large room for future improvements. We showcase the common errors for MC
Dropout and Re-Calibration. Finally, we give guidelines on the usage of these
methods with different levels of data availability and encourage future work on
modeling the human opinion distribution for language reasoning.
