Label Noise in Context (2020)

Abstract: Label noise—incorrectly or ambiguously labeled training examples—can negatively impact model performance. Although noise detection techniques have been around for decades, practitioners rarely apply them, as manual noise remediation is a tedious process. Examples incorrectly flagged as noise waste reviewers’ time, and correcting label noise without guidance can be difficult. We propose LNIC, a noise-detection method that uses an example’s neighborhood within the training set to (a) reduce false positives and (b) provide an explanation as to why the ex- ample was flagged as noise. We demonstrate on several short-text classification datasets that LNIC outperforms the state of the art on measures of precision and F0.5-score. We also show how LNIC’s training set context helps a reviewer to understand and correct label noise in a dataset. The LNIC tool lowers the barriers to label noise remediation, increasing its utility for NLP practitioners.

## introduction label noise-examples with incorrect or ambiguous labels in a training set-degrades the performance of the learned model, resulting in inaccurate predictions @xcite . automated data collection risks generating noisy datasets, and human annotators may introduce noise through a lack of attention or expertise. automatic noise-detection algorithms analyze a training set and flag "suspicious" examples that are likely mislabeled @xcite . suspicious examples can be deleted, automatically corrected by an algorithm, or reviewed by a human. human review is the most effective of these mitigation options but is comparatively expensive. two problems contribute to making human review time consuming: false positives and a lack of explanation. false positives are examples that are incorrectly flagged as noise; reviewing such examples wastes the annotator's time. showing a reviewer a suspicious example without an explanation is effective in the simplest cases, but is likely to cause difficulty and frustration in the more common case of non-obvious noise that requires a deeper comprehension of the data. to date, few noise-detection algorithms have been designed with human review in mind. @xcite is the only work we are aware of that recognized that a noise-detection algorithm for use in a human review process should emphasize precision (i.e., reduce the proportion of false positives). however, we are unaware of any existing work that addresses the explainability of detected label noise. we propose the label noise in context system, or lnic, which uses the neighborhood surrounding a suspicious example in the training set to improve both precision and explainability. by calculating a similarity matrix for the dataset, we are able to identify a suspicious example's neighborhood and use a method similar to a nearest-neighbors classifier to filter out false positives. applying a set of simple heuristics to the same similarity matrix allows us to construct a training set context, like that in the main contributions of this work are • we describe lnic's nearest-neighbors-based algorithm to improve precision and explainability of automatically detected label noise (sec. 3). a demo video is available at https://www. youtube.com/watch?@xmath0, and a live web demo is at http://lnic.mybluemix.net/ 2 related work noise detection. @xcite conducted a comprehensive survey of the various approaches to detecting and remediating label noise. many works advocate removing label noise to improve model performance @xcite @xcite . @xcite advocates automatic relabeling, while others present the case for human-inthe-loop @xcite @xcite and hybrid techniques @xcite . in work contemporaneous with ours, @xcite remove examples where a classifier's confidence is low. the most directly related work is @xcite , describing a noise detection method using predictions from an ensemble of classifiers, and @xcite , proposing the high agreement random forest (harf) system; both systems are described in detail in section 3.1. @xcite dropped suspicious examples but propose correction instead as future work. @xcite note that precision of noise-detection is important when a human will review all suspicious examples. garcia et al. (2016)'s experiments show that harf also achieved stateof-the-art f 1 scores on a variety of datasets. active learning similar to label noise remediation, active learning @xcite seeks to minimize the effort a human needs to expend on data labeling activities in order to improve model performance. however, active learning aims to select the most informative unlabeled data to label next, while label noise detection identifies alreadylabeled data that may require additional labeling effort. we consider active learning and label noise detection as complimentary technologies, that might be woven together within a robust model improvement flow. at a technical level, some active learning and label noise detection techniques are based on similar foundations. query by committee (qbc) @xcite active learning uses an ensemble of classifiers, selecting examples on which the ensemble disagrees for labeling. similarly ensemblebased noise detection algorithms select examples where the ensemble agrees (but disagrees with the given label). model uncertainty, which underpins many effective active learning strategies such as least confident, margin, and entropy, is also the basis of label noise detection methods such as cleanlab @xcite . explainability. with the rise of increasingly complex classification models, explaining classifier predictions has received a great deal of attention. perhaps the most well-known system is lime @xcite . the lime authors noted that explaining classifier predictions increases human trust and provides insights that can be used to improve the model. to explain a classifier's prediction on a particular example, the algorithm collects nearby examples and the model's predictions for them. it trains a linear model on a simpler representation of this data, allowing it to indicate which words or super-pixels are important in the classifier's decision. numerous recent works in nlp and machine learning emphasize explainability. @xcite explained classifier predictions with positive features that push an example towards its assigned class and negative features whose absence prevent an example from being placed in a different class. @xcite jointly trained a generator and an encoder in order to generate rationales for sentiment prediction and a similar-questionretrieval task. @xcite used a convolutional neural network to predict codes describing the diagnosis and treatment of patients given the clinical notes on the patent encounter. their attention mechanism not only improved the system's precision and f 1 , but also highlighted the text that was most relevant to each code. chiyah garcia et al. (2018)'s system used an expert-generated decision tree and a set of templates to generate natural language explanations of what an autonomous underwater vehicle was doing and why. despite the interest in explainable models, no work that we are aware of has attempted to make detected label noise explainable. ## experiments we hypothesize that adding a neighborhood-based filter after noise detection reduces the rates of false positives while retaining true noisy examples. we test this by injecting noise into datasets, running algorithms over them, and measuring the correctly and incorrectly flagged suspicious examples. ## results figure 1 shows average precision and f 0.5 scores across the five datasets, and for both random and next-best noise, agreed correction with activation neighborhood filtering achieves the best average precision. for random noise, harf with activation-neighborhood filtering gives the best f 0.5 across noise rates. however, for next-best noise, harf suffered a dramatic loss in recall when error rates exceeded about 12% (figure 1d ), leading it to have low overall f 0.5 . this may be due to the random forest's use of bagging: if a subset of trees trains on samples with a great deal of non-random noise, those trees could learn to misclassify systematically. agreed correction with feature neighborhood filtering gave the highest average f 0.5 for next-best noise. the upward trend in precision as error rates increase suggests that the same core of false positives are consistently detected. as the number of true positives increases with higher error rates, the core of false positives makes up a smaller fraction of the total number of examples flagged as suspicious. ## the lnic tool the lnic tool implements the algorithms described above and provides a web interface to review label noise in context. the interface visually summarizes the overall label noise within a dataset and links to groups of suspicious examples in context. lnic's representation of the noise summary column. 4 clicking on a cell brings the user to a (figure 2 ) is similar to a confusion matrix. in the list of examples flagged as noise. each of these label noise matrix each cell indicates the number examples can be expanded to show the context, as of noisy examples discovered where the context includes the classes specified by the row and the illustrated in figures 3 and 4 . data from stack exchange illustrates how context helps a reviewer understand problems in a dataset. sometimes, context shows that an example is mislabeled. without context, it is easy for an annotator to be uncertain of whether a question about the existence of a myth belongs in the history class; it is a question about a historical civilization, after all. however, from the context in figure 4 , it is clear that even questions about the history of myths are categorized as mythology, and so the example's label should be changed to maintain consistency. other times, context can reveal more complex issues with the class structure of the data. figure 5 shows a suspicious example from the health class that the noise detection algorithm suggests may belong in the fitness class. the context shows that in fact, both classes include questions about the timing of meals with regard to exercise. a human reviewer should make a decision about where the boundary between these two classes should lie and assign these utterances consistently to one class. ## conclusion although nlp practitioners know that label noise harms performance, and noise detection algorithms have long been available, this technology is not being applied in practice, perhaps because human review of detected errors is difficult and time consuming. lnic makes human review of possible label noise easier and more efficient. it reduces the number of false positive examples that the reviewer must look at, providing state-of-the-art precision and f 0.5 across several short text datasets. and by providing an explanation of why the model flagged an example as suspicious, it makes the output of label noise detectors understandable and actionable.