N oisywiki H ow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processing (2023)

Abstract: Large-scale datasets in the real world inevitably involve label noise. Deep models can gradually overfit noisy labels and thus degrade model generalization. To mitigate the effects of label noise, learning with noisy labels (LNL) methods are designed to achieve better generalization performance. Due to the lack of suitable datasets, previous studies have frequently employed synthetic label noise to mimic real-world label noise. However, synthetic noise is not instance-dependent, making this approximation not always effective in practice. Recent research has proposed benchmarks for learning with real-world noisy labels. However, the noise sources within may be single or fuzzy, making benchmarks different from data with heterogeneous label noises in the real world. To tackle these issues, we contribute NoisywikiHow, the largest NLP benchmark built with minimal supervision. Specifically, inspired by human cognition, we explicitly construct multiple sources of label noise to imitate human errors throughout the annotation, replicating real-world noise, whose corruption is affected by both ground-truth labels and instances. Moreover, we provide a variety of noise levels to support controlled experiments on noisy data, enabling us to evaluate LNL methods systematically and comprehensively. After that, we conduct extensive multi-dimensional experiments on a broad range of LNL methods, obtaining new and intriguing findings.

## introduction large-scale labeled data has become indispensable in the notable success of deep neural networks (dnns) in various domains and tasks @xcite . due to imperfect sources like crowd-sourcing and web crawling @xcite @xcite output due to the lack of appropriate benchmarks, previous research often studied synthetic label noise to simulate real-world label noise @xcite . as a general and realistic noise, real-world noise may have several noise sources (i.e., be heterogeneous) @xcite and be instance-dependent (i.e., p (ỹ|y, x), where the probability of an instance being assigned to the incorrect label ỹ depends on the original ground-truth label y and data x) @xcite . however, synthetic noise is generated from an artificial distribution and is thus instance-independent (i.e., p (ỹ|y)), which may not always work well in practice. recently, various benchmarks for learning with real-world noisy labels have been proposed across fields like computer vision (cv) @xcite , audio signal processing (asp) @xcite , and natural language processing (nlp) @xcite . to fully evaluate robust learning methods with real-world label noise, benchmarks should be as close to real-world scenarios as possible. meanwhile, controlled ex-periments are encouraged to verify whether lnl methods can remain effective over a wide range of noise levels @xcite . nevertheless, the noise levels in most datasets are fixed and unknown, resulting in uncontrolled label noise @xcite . moreover, the noise therein often comes from the same or ambiguous sources @xcite , which conflicts with the heterogeneous characteristics of real-world noise. these problems prevent a better understanding of lnl methods. to bridge this gap, we present noisywikihow, a new nlp benchmark for evaluating lnl methods focusing on the intention identification task. intention identification promotes numerous downstream natural language understanding tasks, from commonsense reasoning @xcite to dialogue systems @xcite . additionally, the complexity of the task (total of 158 categories) facilitates a deeper investigation of the efficacy of lnl approaches. the task form is shown in to make the benchmark more representative of real-world scenarios, we propose a practical assumption: real-world label noise in a dataset is mainly induced by human errors, regardless of whether the dataset's construction is automated or crowd-sourced. existing psychological and cognitive evidence further supports our hypothesis. it shows that different annotators have different preferences and biases @xcite , which means human labeling errors typically result from multiple noise sources. furthermore, humans may make random labeling errors due to random attention slips. but they are more likely to produce label noise when labeling hard cases @xcite (i.e., noise is instance-dependent), such as instance (c) in motivated by this human cognition, we first collect data from the wikihow website, foot_1 which contains a collection of professionally edited how-to guideline articles, providing a vast quantity of clean scripts and corresponding categories for free to help achieve controlled experiments and ensure benchmark quality. after that, we explicitly inject a variety of noise sources into clean data to replicate human annotation errors, thus introducing real-world label noise into the benchmark. notably, training samples in our benchmark exhibit a long-tailed class distribution, which is in line with the facts, i.e., data in real-world applications is heavily imbalanced @xcite . besides, we achieve minimal human supervision by using a series of automated labeling procedures, saving lots of time and human effort. to evaluate noisywikihow, we carry out extensive experimentation across various model architectures and noise sources, execute plentiful lnl methods on our benchmark, compare the more realistic real-world noise with the extensively studied synthetic noise, and investigate a case study and long-tailed distribution characteristics. ## experiments we first present the general settings for experiments (section 4.1). further, we systematically evalu- ## conclusion in this paper, we study the problem of learning with noisy labels and establish an nlp benchmark called noisywikihow with minimal human supervision, which contains more than 89k procedural events with heterogeneous and controlled realworld label noise. experimental results reveal several new findings. (1) some widely accepted lnl methods are not always impactful, especially with real-world label noise. (2) different noise sources may have varying difficulties resisting label noise, although they are all from real-world noise. (3) few lnl methods can effectively combat real-world noise and synthetic noise at the same time. (4) the model trained under the real-world label noise has better generalization performance.