Natural Language Inference with Mixed Effects (2020)

Abstract: There is growing evidence that the prevalence of disagreement in the raw annotations used to construct natural language inference datasets makes the common practice of aggregating those annotations to a single label problematic. We propose a generic method that allows one to skip the aggregation step and train on the raw annotations directly without subjecting the model to unwanted noise that can arise from annotator response biases. We demonstrate that this method, which generalizes the notion of a mixed effects model by incorporating annotator random effects into any existing neural model, improves performance over models that do not incorporate such effects.

## introduction a common method for constructing natural language inference (nli) datasets is (i) to generate text-hypothesis pairs using some methodcommonly, crowd-sourced hypothesis elicitation given a text from some existing resource @xcite or automated text-hypothesis generation @xcite ; (ii) to collect crowd-sourced judgments about inference from the text to the hypothesis; and (iii) to aggregate the possibly multiple annotations provided for a single text-hypothesis pair into a single label. this final step follows common practice across annotation tasks in nlp; but for nli in particular, there is growing evidence that it is problematic due to disagreement among annotators that is not captured by the probabilistic outputs of standard nli models @xcite . one way to capture this disagreement would be to directly model the variability in the raw annotations. but this approach presents a challenge: it can *equal contribution. this work is licensed under a creative commons attribution 4.0 international license. license details: http://creativecommons.org/licenses/ by/4.0/ . be difficult to assess how much disagreement arises from disagreement about the interpretation of a text-hypothesis pair and how much is due to biases that annotators bring to the task. such biases can be extreme. for instance, figure 1 plots the distribution by annotator of @xcite ratings-with -50 clear contradiction and 50 clear entailment-for the same 20 nli pairs in pavlick and kwiatkowski's dataset. despite describing responses to the same items, the distributions are quite variable, suggesting variability in how annotators approach the task. this difference in approach may be relatively shallow-e.g. given some true label (or distribution thereon), annotators merely differ in their mapping of that value to the response scale-or they may be quite deep-e.g. annotators differ in how they interpret the relationship between texts and hypotheses. we investigate both of these possibilities within a mixed effects modeling framework @xcite . the core idea is to incorporate annotator-specific parameters into standard nli models that either (i) merely modify the output of a standard classification/regression head or (ii) modify the parameters of the head itself. these two options correspond to the mixed effects modeling concepts of random intercepts and random slopes, respectively. for the same reason that such random effects can be incorporated into effectively any generalized linear model in a modular way, our components can be be similarly incorporated into any nli model. we describe how this can be done for a simple roberta-based nli model. we find (i) that models containing only random intercepts outperform both standard models and models containing random slopes when annotators are known; and (ii) that when annotators are not known, performance drops precipitously for both random effects models. together, these findings suggest that those building nli datasets should provide annotator information and that those developing nli systems should incorporate random effects into their models. ## extended task definition in the standard supervised setting, nli datasets are (graphs of) functions from text-hypothesis pairs t i , h @xmath0, neutral, en-tailed} or {not-entailed, entailed}, but may also be a finer-grained (e.g. five-point) ordinal scale @xcite or bounded continuous scale @xcite . the nli task is to produce a single label from y given a text-hypothesis pair. we extend this setting by assuming that nli datasets are (graphs of) functions from texthypothesis pairs and annotator identifiers a @xmath1. a particular model need not make use of the annotator information during training and may similarly ignore it at evaluation time. though many existing datasets do not provide annotator information, it is trivial for a dataset creator to add (even post hoc), and so this extension could feasibly be applied to any existing dataset. ## models we consider two types of model: one containing only annotator random intercepts and another additionally containing annotator random slopes. the first assumes that differences among annotators are relatively shallow-e.g. given some true label for a pair (or distribution thereon), annotators have their own specific way of mapping that value to a response-and the second assumes that the differences among annotators are deeper-e.g. annotators differ in how they interpret the relation between texts and hypotheses. this distinction is independent of the labels y: regardless of whether the labels are discrete or continuous, random effects can be incorporated. in the language of generalized linear mixed models, the link functions are the only thing that changes. we consider two label types: three-way ordinal and bounded continuous. annotator random intercepts amount to annotator specific bias terms ρ a i on the raw predictions of a classification/regression head. unlike standard fixed bias terms, however, what makes these terms random intercepts is that they are assumed to be distributed according to some prior distribution with unknown parameters. this assumption models the idea that annotators are sampled from some population, and it yields 'adaptive regularization' (mcelreath, 2020), wherein the biases for annotators who provide few labels will be drawn more toward the central tendency of the prior. random intercepts for categorical outputs can take two forms, depending on whether the model enforces ordinality constraints-as linked logit models do (agresti, 2014)-or not. since most common categorical nli models do not enforce ordinality constraints, we do not enforce them here, assuming that the model has some independently tunable function h θ : r m × r @xmath2: random intercepts for continuous outputs are effectively shifting terms on the single value predicted by some independently tunable function h : r m × r @xmath3. if the continuous output is furthermore bounded, a squashing function g is necessary. in the bounded case, we assume that the variable-scaled to (0, 1)-is distributed beta (following @xcite ) with mean µ i and precision ν @xmath4. megaveridicality someone knew that something happened. that thing happened. someone thought that something happened. that thing happened. ## experiments we compare models both with and without random effects when fit to nli datasets conforming to the extended setting described in §2. the model without random intercepts (the fixed model) simply ignores annotator information-effectively locking ρ a i to 0 for all annotators a i . encoder all models use pretrained roberta @xcite as their encoder. we use the basic lm pretrained versions (no nli fine-tuning). data to our knowledge, the only nli datasets that both publicly provide annotator identifiers and are large enough to train an nli system are megaveridicality (mv; @xcite , which contains three-way categorical annotations aimed at assessing whether different predicates give rise to veridicality inferences in different syntactic structures, and meganegraising (mn; an and white, 2020), which contains bounded continuous [0, 1] annotations aimed at assessing whether different predicates give rise to neg(ation)-raising inferences in different syntactic structures. evaluation we evaluate all of our models using 5-fold cross-validation. we consider four partitioning methods: (i) random: completely random partitioning; (ii) predicate: partitioning by the main clause predicate found in the text (a particular main clause predicate occurs in one and only one partition); (iii) structure: partitioning by the syntactic structure found in the text (a particular structure occurs in one and only one partition); and (iv) annotator: a particular annotator occurs in one and only one partition. for the first three methods, we ensure that each annotator occurs in every partition, so that random intercepts and random slopes for that annotator can be estimated. for the annotator method, where we do not have an estimate for the random effects of annotators in the held-out data, we use the mean of the prior. 1 we report mean accuracy on held-out folds for the categorical data (mv); and following chen et al. (2020), we report mean rank correlation on heldout folds for the bounded continuous data (mn). to make these metrics comparable, we report them relative to the performance of both a baseline model and the best possible fixed model. for the categorical data, the baseline model predicts the majority class across all pairs, and the best possible fixed model predicts the majority class across annotators for each pair. similarly, for the bounded continuous data, the baseline model predicts the mean response across all pairs, and the best possible fixed model predicts the mean response across annotators for each pair. 2 these relative scores are 0 when the model does not outperform the baseline and 1 when the model performs as well as the best possible fixed model. it is possible for a random effects model to obtain a score of greater than 1 by leveraging annotator information or less than 0 if it overfits the data. ## results consistent with pavlick and kwiatkowski's findings, these results suggest that variability in annotators' responding behavior is substantial; otherwise, it would not be possible for the random effects models to outperform the best possible fixed model, and we would not expect the observed drops in performance when annotator information is removed. but this variability is likely relatively shallow: if these differences were due to deeper differences in annotators' interpretation of the pair, we would expect this to manifest in better performance by the random slopes models, as the latter subsumes the random intercepts model and can leverage the additional power of annotator-specific classification or regression heads. of course, it remains a live possibility that the encoder we used does not extract features that are linearly related to the relevant interpretive variability, and so further investigation of random slopes models with different encoders may be warranted (see @xcite . contrasting the results on ordinal and bounded continuous data, the fixed model tends to perform better on ordinal data than on bounded continuous data. a similar trend is not seen for the random effects models. indeed, the random intercepts model performs substantially better on the bounded continuous data under all settings except for an-notator. these results could be due to the link function we used for the bounded continuous data: the fixed model consistently learned small values for the precision parameter ν 0 , resulting in sparse (bimodal) beta distributions. but the fact that the random intercepts model reliably outperforms the best possible fixed model implies that any tweaks to the link function would not bring the fixed model up to the level of the random intercepts model. ## analysis to understand how annotator biases tend to pattern with ordinal and bounded continuous scales, we investigate the mean ρ a for each annotator a in the random intercepts models across folds under the random partition method. figure 2 plots the distribution of biases across categorical annotators when the fixed effect potentials-h θ (z i ) in the equations in §3-are set to 0: softmax(ρ a ). this distribution can be thought of as an indicator of how an annotator would respond in the absence of any correct answer. we see the most variability in terms of annotators' biases for or against neutral: the interquartile range for neutral biases is [0.23, 0.42] compared to [0.24, 0.41] for contradiction and [0.25, 0.40] for entailment. interestingly, these biases do not reflect the fact that the scale is ordinal: if they did, we would expect more positive correlations between adjacent values; but neutral biases are more strongly rank anticorrelated with contradiction (@xmath7.57) and entailment (@xmath8.48) than contradiction is with entailment (@xmath9.35). this finding suggests that three-value "ordinal" nli scales are better thought of as nominal. figure 3 plots the analogous distribution for the bounded continuous annotators, with the y-axis showing ρ a1 and the x-axis showing logit -1 (ρ a2 ). the lines behind the points show, for particular values of h θ (z i ), the ρ a1 at which the distribution for a particular annotator becomes sparsei.e. where α, β < 1-heavily favoring responses very near zero or one, rather than the mean. we see a weak rank correlation (@xmath10.24, @xmath11.05) between precision and annotators' biases to give responses nearer to one, suggesting that one-biased annotators tend to give less sparse responses. this correlation might, in part, explain the poor performance of the bounded continuous models in the annotator cross-validation setting. ## related work the models developed here are closely related to models from item response theory (irt). irt has been used to assess annotator quality @xcite @xcite @xcite and various properties of an item (passonneau and carpenter, 2014; @xcite , including difficulty @xcite @xcite . other non-irt-based work attempts to measure the relationship between annotator disagreement and item difficulty @xcite . other recent work focuses on incorporating annotator information in modeling annotatorgenerated text. @xcite find that concatenating annotator ids as input features to a bertbased text generation model yields improved performance on several datasets. although we reach similar conclusions about the importance of annotator information in this work, our approach differs in at least one critical respect: by explicitly distinguishing linguistic input from annotator information, our model cleanly separates the linguistic representations from representations of the annotators interpreting or producing those representations. this clean separation is of potential benefit not only to those interested in using nli models (or deep learning architectures more generally) in an experimental (psycho)linguistics setting, where distinguishing the two sorts of representations can be crucial, but also to those interested in possibly quite substantial reductions in model size. ## conclusion we find (i) that models containing only random intercepts outperform standard models when annotators are known, and (ii) that models that further contain random slopes do not yield any additional benefit. these results indicate that, though differences among nli annotators' response behavior are important to model, these differences may not be particularly deep, limited to the ways in which annotators use the response scale, but not relating to deeper interpretive differences.