  We study estimating inherent human disagreement (annotation label
distribution) in natural language inference task. Post-hoc smoothing of the
predicted label distribution to match the expected label entropy is very
effective. Such simple manipulation can reduce KL divergence by almost half,
yet will not improve majority label prediction accuracy or learn label
distributions. To this end, we introduce a small amount of examples with
multiple references into training. We depart from the standard practice of
collecting a single reference per each training example, and find that
collecting multiple references can achieve better accuracy under the fixed
annotation budget. Lastly, we provide rich analyses comparing these two methods
for improving label distribution estimation.
