Competency Problems: On Finding and Removing Artifacts in Language Data (2021)

Abstract: Much recent work in NLP has documented dataset artifacts, bias, and spurious correlations between input features and output labels. However, how to tell which features have “spurious” instead of legitimate correlations is typically left unspecified. In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems. For example, the word “amazing” on its own should not give information about a sentiment label independent of the context in which it appears, which could include negation, metaphor, sarcasm, etc. We theoretically analyze the difficulty of creating data for competency problems when human bias is taken into account, showing that realistic datasets will increasingly deviate from competency problems as dataset size increases. This analysis gives us a simple statistical test for dataset artifacts, which we use to show more subtle biases than were described in prior work, including demonstrating that models are inappropriately affected by these less extreme biases. Our theoretical treatment of this problem also allows us to analyze proposed solutions, such as making local edits to dataset instances, and to give recommendations for future data collection and model design efforts that target competency problems.

## introduction attempts by the natural language processing community to get machines to understand language or read text are often stymied in part by issues in our datasets @xcite . many recent papers have shown that popular datasets are prone to shortcuts, dataset artifacts, bias, and spurious correlations @xcite @xcite . while these empirical demonstrations of artifact statistics in @xmath0.01/28k neutral contradict entailment figure 1 : a statistical test for deviation from a competency problem, where no individual feature (here words) should give information about the class label, plotting the number of occurrences of each word against the conditional probability of the label given the presence of the word. the label associated with each point is marked by color and superscript. all features above the blue line have detectable correlation with class labels, using a very conservative @xcite deficiencies in the data are useful, they often leave unanswered fundamental questions of what exactly makes a correlation "spurious", instead of a feature that is legitimately predictive of some target label. in this work we attempt to address this question theoretically. we begin with the assumption that in a language understanding problem, no single feature on its own should contain information about the class label. that is, all simple correlations between input features and output labels are spurious: p(y|x i ), for any feature x i , should be uniform over the class label. we call the class of problems that meet this assumption competency problems ( §2). 1 this assumption places a very strong restriction on the problems being studied, but we argue that it is a reasonable description of complex language understanding problems. consider, for example, the problem of sentiment analysis on movie reviews. a single feature might be the presence of the word "amazing", which could be legitimately correlated with positive sentiment in some randomly-sampled collection of actual movie reviews. however, that correlation tells us more about word frequency in movie reviews than it tells us about a machine's ability to understand the complexities of natural language. a competent speaker of a natural language would know that "amazing" can appear in many contexts that do not have positive sentiment and would not base their prediction on the presence of this feature alone. that is, the information about the sentiment of a review, and indeed the meaning of natural language, is contained in complex feature interactions, not in isolated features. to evaluate a machine's understanding of language, we must remove all simple feature correlations that would allow the machine to predict the correct label without considering how those features interact. collecting data that accurately reflects the assumptions of a competency problem is very challenging, especially when humans are involved in creating it. humans suffer from many different kinds of bias and priming effects, which we collectively model in this work with rejection sampling during data collection. we theoretically analyze data collection under this biased sampling process, showing that any amount of bias will result in increasing probability of statistically-significant spurious feature correlations as dataset size increases ( §3). this theoretical treatment of bias in data collection gives us a new, simple measure of data artifacts ( §3.2), which we use to explore artifacts in several existing datasets ( §4). figure 1 revisits prior analyses on the snli dataset @xcite with our statistical test. an analysis based on pointwise mutual information (e.g., @xcite would correspond to a horizontal line in that figure, missing many features that have less extreme but still significant correlations with class labels. these less extreme correlations still lead models to overweight simple features. the problem of bias in data collection is pervasive and not easily addressed with current learning techniques. our framework also allows us to examine the theoretical impact of proposed techniques to mitigate bias, including performing local edits after data collection ( §5) and filtering collected data ( §6). we derive properties of any local edit procedure that must hold for the procedure to effectively remove data artifacts. these proofs give dataset builders tools to monitor the data collection process to be sure that resultant datasets are as artifact-free as possible. our analysis of local edits additionally suggests a strong relationship to sensitivity in boolean functions (o'donnell, 2014), and we identify gaps in the theory of sensitivity that need to be filled to properly account for bias in sampled datasets. we believe our theoretical analysis of these problems provides a good starting point for future analyses of methods to improve nlp data collection, as well as insights for inductive biases that could be introduced to better model competency problems. ## competency problems we define a competency problem to be one where the marginal distribution over labels given any single feature is uniform. for our analysis, we restrict ourselves to boolean functions: we assume an input vector x and an output value y, where @xmath1, 1} d and @xmath2, 1}. foot_1 in this setting, competency means p(y|x i ) = 0.5 for all i. in other words, the information mapping x to y is found in complex feature interactions, not in individual features. our core claim is that language understanding requires composing together many pieces of meaning, each of which on its own is largely uninformative about the meaning of the whole. we do not believe this claim is controversial or new, but its implications for posing language understanding as a machine learning problem are underappreciated and somewhat counterintuitive. if a model picks up on individual feature correlations in a dataset, it has learned something extra-linguistic, such as information about human biases, not about how words come together to form meaning, which is the heart of natural language understanding. to push machines towards linguistic competence, we must control for all sources of extra-linguistic information, ensuring that no simple features contain information about class labels. for some language understanding problems, such as natural language inference, this intuition is already widely held. we find it surprising and problematic when the presence of the word "cat", "sleeping" or even "not" in either the premise or the hypothesis gives a strong signal about an entailment decision @xcite . competency problems are broader than this, however. consider the case of sentiment analysis. it is true that a movie review containing the word "amazing" is more likely than not to express positive sentiment about the movie. this is because of distributional effects in how humans choose to use phrases in movie reviews. these distributional effects cause the lexical semantics of "amazing" to carry over into the whole context, essentially conflating lexical and contextual cues. if our goal is to build a system that can accurately classify the sentiment of movie reviews, exploiting this conflation is useful. but if our goal is instead to build a machine that understands how sentiment is expressed in language, this feature is a red herring that must be controlled for to truly test linguistic competence. ## biased sampling to get machines to perform well on competency problems, we need data that accurately reflects the competency assumption, both to evaluate systems and (presumably) to train them. however, humans suffer from blind spots, social bias, priming, and other psychological effects that make collecting data for competency problems challenging. examples of these effects include instructions in a crowdsourcing task that prime workers to use particular language, foot_2 or distributional effects in source material, such as the "amazing" examples above, or racial bias in face recognition @xcite and abusive language detection datasets @xcite . in order to formally analyze the impact of human bias on collecting data for competency problems, we need a plausible model of this bias. we represent bias as rejection sampling from the tar-get competency distribution based on single feature values. specifically, we assume the following dataset collection procedure. first, a person samples an instance from an unbiased distribution p u (x, y) where the competency assumption holds. the person examines this instance, and if feature x @xmath3, the person rejects the instance and samples a new one, with probability r i . if @xmath4, a high value for r i would lead to "amazing" appearing more often with positive sentiment, as is observed in typical sentiment analysis datasets. we do not that claim rejection sampling is a plausible psychological model of dataset construction. however, we do think it is a reasonable first-order approximation of the outcome of human bias on data creation, for a broad class of biases that have empirically been found in existing datasets, and it is relatively easy to analyze. ## empirical analysis with a hypothesis test in hand, we can examine existing datasets for evidence of statisticallysignificant feature bias, and then explore the extent to which this bias impacts models supervised with this data. prior work has used pointwise mutual information (pmi) to find features that have high correlation with labels (e.g., @xcite . this measure is useful for understanding why certain features might get used as deterministic decision rules by models @xcite . however, studies involving pmi have also intuitively understood that pmi by itself does not tell the whole story, as a strict ranking by pmi would return features that only appear once in the dataset. to account for this problem, they used arbitrary cutoffs and included information about feature occurrence in addition to their pmi ranking. a benefit of our approach to defining and detecting artifacts is that we have a single statistical test that takes into account both the number of times a feature appears and how correlated it is with a single label. we use this test to find features with the strongest statistical evidence for artifacts ( §4.1) and then show empirically that models use these features inappropriately when making predictions ( §4.2). this analysis goes beyond deterministic prediction rules, showing that the impact of sampling bias on model behavior is subtle and pervasive. ## mitigating artifacts with local edits many works have tried to remove data artifacts by making minimal changes to existing data @xcite @xcite . in this section we show that this kind of data augmentation can be effective with an appropri-ately sensitive edit model, where sensitivity refers to how often a change to inputs results in the label changing. however, because humans are involved in making these changes, achieving appropriate sensitivity is challenging, and bias in this process can lead to the introduction of new artifacts. this suggests that care must be taken when performing edit-based data augmentation, as large edited training datasets are not likely to be artifact-free (cf. @xcite . imagine a new dataset d e consisting of samples x , y generated by making local edits according to the following repeated procedure: proposition 1 (proof in §b). if r @xmath5, then, an analogous statement holds @xmath6. thus, local editing will dilute or cancel out artifacts from rejection sampling if the edit sensitivity is high enough, but if it is too high, it can introduce artifacts in the opposite direction. in practice, if it is possible to pick a dimension to flip more or less uniformly, then e i ≈ 0 in a high-dimensional space, so engineering s ≈ 0.5 should produce artifact-free additional samples with p e (y |x i ) ≈ 0.5. furthermore, edit sensitivity and edit dimension are empirically measurable when constructing a dataset. this empirical measurement can give theoretical guarantees for the degree to which the local editing will alleviate artifacts (i.e., this gives us a principled way to decide between edit models). ## other mitigation techniques in this section we briefly discuss the implications of our theoretical analysis for other artifact mitigation techniques that have been proposed in the literature. our analysis in this section is not rigorous and is meant only to give high-level intuition or potential starting points for future work. more annotators one suggested mitigation technique for dataset artifacts is to increase the number of annotators @xcite . especially when people generate the text that is used in a dataset, there can be substantial person-specific correlations between features and labels. having more annotators washes out those correlations in aggregate, making the data less biased overall. we briefly analyze this procedure using our rejection sampling framework. for simplicity, we have so far only considered a single possible rejection probability, where an instance is rejected with probability r i if x @xmath7. if we introduce additional rejection probabilities for the other three possible combinations of values for x i and y, there will be the possibility that some rejections balance out other rejections. we can model multiple anno-tators by splitting a dataset into k different slices that have their own bias vectors r. if the r vectors are uncorrelated, it seems likely that as k increases, the probability that p(y|x i ) deviates from p u (y|x i ) tends towards zero. even in our simplistic model, if we assume a sparse r, averaging more and more of them will make the deviation tend toward zero, if the non-zero dimensions are uncorrelated. however, if the r vectors are correlated, increasing the number of annotators will not produce data reflecting the competency assumption. when might the r vectors be correlated? this could happen due to societal biases, word usage frequencies, or priming effects from data collection instructions given to all annotators. surely across any pool of annotators there will be some dimensions along which r values are correlated, and other dimensions along with they are not. increasing the number of annotators thus helps mitigate the problem, but does not solve it completely. data filtering a recent trend is to remove data from a training set that is biased in some way in order to get a model that generalizes better @xcite @xcite . while this method can be effective for very biased datasets, it is somewhat unsatisfying to remove entire instances because of bias in a single feature. in the extreme case where r i ≈ 1, such as with "nobody" in snli (fig. 1 ), this process could effectively remove x i from the observed feature space. to understand the effect of these automated methods on dataset artifacts, we repeat the analysis from §4.1 on data that was classified as "ambiguous" according to dataset cartography @xcite . this data was shown to provide better generalization when used as training data compared to the original training set. the ambiguous instances did not have a balanced label distribution, so we downsampled the data to balance it, then downsampled the whole training data to get the same number of instances as the balanced ambiguous set. the resulting artifact plots are shown in figure 4 . as can be seen, the "ambiguous" instances have many fewer deviations from the competency assumption, across the entire range of our hypothesis test. it is not just high pmi values that are getting corrected by finding ambiguous instances; all statistical deviations are impacted. this effect is striking, and it further corroborates our arguments about the competent models an interesting question is whether we can inject a "competency inductive bias" into models, i.e., discourage relying on individual features. the closest works we are aware of are methods that ensemble weak models together with strong models during training @xcite , or ensembles of models with unaligned gradients @xcite . 12 comparing the lower part of figure 4 to figure 1 also corroborates our derived result ( §3.1) that larger datasets are more likely to have artifacts. with 24% of the data there are many fewer artifacts. other works use ensembles with models targeted at known sources of data artifacts, but these are less close to a competency assumption @xcite . ## conclusion the more nlp models advance, the better they are at learning statistical patterns in datasets. this is problematic for language understanding research if some statistical patterns allow a model to bypass linguistic competence. we have formalized this intuition with a class of problems called competency problems, arguing that, for any language understanding task, all correlations between simple features and labels are spurious. collecting data meeting this assumption is challenging, but we have provided theoretical analysis that can inform future data collection efforts for such tasks. we conclude with some final thoughts on general best practices for data collection, informed by the analysis in this paper. if annotators are generating text for some data collection task, find ways to decrease priming effects. this could involve using images as prompts instead of text @xcite , or randomly sampling words to include in the generated text. if existing text is being collected and annotated, make local edits to the text while monitoring the sensitivity of those edits according to the guidelines in §5, perhaps using different processes between train and test, to minimize correlations between train features and test labels.