Label Errors in BANKING 77 (2022)

Abstract: We investigate potential label errors present in the popular BANKING77 dataset and the associated negative impacts on intent classification methods. Motivated by our own negative results when constructing an intent classifier, we applied two automated approaches to identify potential label errors in the dataset. We found that over 1,400 (14%) of the 10,003 training utterances may have been incorrectly labelled. In a simple experiment, we found that by removing the utterances with potential errors, our intent classifier saw an increase of 4.5% and 8% for the F1-Score and Adjusted Rand Index, respectively, in supervised and unsupervised classification. This paper serves as a warning of the potential of noisy labels in popular NLP datasets. Further study is needed to fully identify the breadth and depth of label errors in BANKING77 and other datasets.

## introduction nlp researchers and practitioners use standard benchmark datasets in the selection, development, and comparison of advanced nlp methods. the use of standard benchmarks enables an apples-toapples comparison of competing methods, as well as an evaluation of a method under different business scenarios. recently, researchers have proposed three promising intent classification benchmark datasets that are large (>10,000 instances) and include more than 50 unique intents: banking77 (cas), hwu64 @xcite , and clinc150 (lar). the aforementioned datasets have been used to evaluate pretrained transformers @xcite , density-based models (gon), few-shot learning (luo), open intent detection @xcite , and intent discovery (cha). these benchmark datasets are hand-labelled by humans and their categorization can be subjective in nature. in addition, humans may make mistakes in the labelling process. as such, it is im-portant to assess the accuracy of the human-given labels @xcite . our recent experience with banking77 suggested that several labeling errors were present in the dataset. using confident learning @xcite and our own cosine similarity methodology (section 3.2), we found that over 1,400 (14%) of the 10,003 training samples may have been incorrectly labelled. using noisy labels to train and evaluate an intent classifier could have disastrous consequences. first, the classifier could incorrectly classify new utterances. second, any performance measures would be based on mislabelled truth and therefore be inaccurate. finally, researchers and practitioners may make an incorrect recommendation or conclusion for the downstream task-oriented conversational system. in this paper, we investigate the potential label errors present in banking77. first, we provide background on banking77 in section 2. in section 3, we describe our methodology for determining potential label errors. we first use confident learning @xcite and identify over 900 potential label errors. next, we design a methodology based on cosine similarity and identify an additional 500 potential label errors. in section 4, we quantify the potential impacts of errors on a downstream nlp task. finally, in section 5 we conclude and outline future work. the authors also acknowledged that there are partially overlapping intent categories, and therefore, the intent detection system cannot rely only on the semantics of individual words to correctly categorize the utterance. however, they did not provide any specifics regarding the extent and impact of such overlaps. ## identifying potential label errors while implementing our own intent classifier on banking77, we noticed unexpectedly poor performance in several intent categories. we found that our classifier was confusing many of the labels. for instance, we found that up to sixteen "truth" labels were predicted as a single intent by our classifier. similarly, one predicted intent included up to twelve truth labels. ( to further understand the extend of these potential label errors, we applied and compared two automated approaches: the confident learning framework, and a cosine similarity approach. ## experiment results to illustrate the negative impact of the noisy labels on the performance of an intent classifier, we designed an experiment as follows. first, we considered two versions of the bank-ing77 dataset. the original, unmodified version, and a trimmed version whereby we removed all utterances with potential label errors identified by either the cl framework or cosine-similarity approach. we reduced the dimensionality of the embeddings using uniform manifold approximation and projection (umap) @xcite (@xmath0, @xmath1. in the supervised approach, we used lightgbm (@xmath2, @xmath3.1, @xmath4, @xmath5. using 5-fold cross validation, we measured each model's accuracy and f1-score. for comparison, we used agglomerative clustering (@xmath6, @xmath7, @xmath8. we then measured five common clustering metrics: adjusted rand index (ari); adjusted mutual information (ami), completeness, fowlkes-mallows, and homogeneity. ## conclusion and future work in this paper, we investigated potential label errors present in the popular banking77 benchmark dataset. we applied two automated techniques to identify potential label errors. first, we used the confident learning framework to find utterances based on class-conditional noise estimates. second, we developed our own cosine-similarity based technique to find utterances that are semantically similar but labeled differently. together, these approaches identified over 1,400 utterances with potential label errors. a simple experiment showed that an intent classifier's performance can be improved by removing such utterances. f1-score increased by 4.5% for the supervised classifier, and ari increased by 8% for the unsupervised classifier. given the importance of benchmark datasets in the development, evaluation, and selection of nlp techniques, it is important that the labels contain as few errors as possible. we would like to extend our work by developing an automated correction tool that can identify and fix label errors. we will also manually verify and correct errors in banking77, and it will serve as the ground truth for evaluating the performance of the automated correction tool. furthermore, we will apply the methodology on other benchmark datasets such as clinc150 and hwu64.