On Evaluation of Document Classification with RVL - CDIP (2023)

Abstract: The RVL-CDIP benchmark is widely used for measuring performance on the task of document classification. Despite its widespread use, we reveal several undesirable characteristics of the RVL-CDIP benchmark. These include (1) substantial amounts of label noise, which we estimate to be 8.1% (ranging between 1.6% to 16.9% per document category); (2) presence of many ambiguous or multi-label documents; (3) a large overlap between test and train splits, which can inflate model performance metrics; and (4) presence of sensitive personally-identifiable information like US Social Security numbers (SSNs). We argue that there is a risk in using RVL-CDIP for benchmarking document classifiers, as its limited scope, presence of errors (state-of-the-art models now achieve accuracy error rates that are within our estimated label error rate), and lack of diversity make it less than ideal for benchmarking. We further advocate for the creation of a new document classification benchmark, and provide recommendations for what characteristics such a resource should include.

## introduction within the document understanding research area, the rvl-cdip dataset @xcite has emerged as the primary benchmark for evaluating and comparing document classifiers. rvl-cdip is composed of 16 document type categories, including resume, letter, invoice, etc. its large volume of training data-320,000 samplesfacilitates benchmarking state-of-the-art deep learning and transformer-based architectures. while initially released as a computer vision benchmark in 2015, more recent state-of-the-art models now incorporate image, text, and page layout modalities. for instance, recent tri-modal models like doc-former @xcite , ernie-layout @xcite , layoutlmv3 (huang et al., ⇤ corresponding email: stefan.dataset@gmail.com 2022), and bi-vldoc @xcite now achieve classification accuracies ranging in the midto high-90s, with bi-vldoc reporting a state-ofthe-art of 97.12% on the rvl-cdip test set. this is a large improvement over earlier image-centric work, and we chart this improvement in figure 1 . as model performance on rvl-cdip improves, it becomes increasingly important to ensure that further gains are meaningful with respect to the classification task. this concern has been raised by prior work that has found that benchmark evaluation datasets often contain substantial amounts of label errors or noise (e.g., @xcite , substantial overlap between test and train data (e.g., @xcite , and data collection artifacts that cause models to overfit to spurious cues (e.g., @xcite . therefore, we cast a critical eye to the rvl-cdip benchmark to answer: is rvl-cdip still suitable for effectively measuring the performance of document classifiers? in doing so, we first observe a lack of clear label or annotation guidelines provided with the original introduction of rvl-cdip. therefore, we create verifiable label guidelines for the 16 rvl-cdip categories. with these guidelines, we are then able to conduct a review of the data, and we find that label errors account for an estimated 8.1% of data rvl-cdip's test split, a rate greater than the current state-of-the-art model accuracy error rate, indicating that contemporary high-performing models are overfitting to noise. we also observe relatively high rates of documents that have ambiguous or multiple valid labels, which is problematic given rvl-cdip is a single-label classification benchmark. additionally, we also observe a large overlap between test and train data splits, where there are (near-) duplicate documents seen in both train and test splits, as well as documents that share common templates. lastly, our review of rvl-cdip data uncovered a surprisingly large amount of sensitive personally-identifiable information, particularly in the resume category, where we found 7.7% of documents contained us social security numbers. we argue that the characteristics that we observe make rvl-cdip an unattractive benchmark for training and evaluating document classifiers. we end with recommendations for what qualities a new document classification benchmark should have. ## related work this section discusses related work in two areas: ## the rvl-cdip dataset the rvl-cdip dataset was introduced in @xcite as a benchmark for evaluating imagebased classification and retrieval tasks. 1 since then, rvl-cdip has primarily been used as a document type classification benchmark. rvl-cdip consists of 400,000 document images distributed across 16 document type categories, listed in ## label errors and ambiguities in rvl-cdip armed with better knowledge of what constitutes each of the 16 rvl-cdip categories, we analyze the contents of the rvl-cdip test set to estimate the amount of label errors and ambiguities found in this set. we manually checked for errors in the rvl-cdip test set by sampling 1,000 documents from each of the 16 categories (for a total of 16,000 documents). we used our label guidelines established in section 3.1 to help us determine the validity of each of these 16,000 samples. we tracked several types of errors and ambiguities: (1) documents found in a category that clearly are mis-labeled and instead belong in a different rvl-cdip category -we refer to this error type as mis-labeled; (2) documents that do not appear to have a single clear rvl-cdip label -we refer to this label type as unknown; the average of each category's error rates is 8.1%, which is higher than the classification accuracy error rates reported by many state-of-the-art models listed in ## overlap between test and train splits to help better understand the similarity scores, we conduct an experiment where we categorize each similarity pair into one of the following: duplicate, if the test-train pair represents the same document; template, if both documents in a pair use the same document template; and different, for all other pairs. we annotated a sample of 1,086 similarity pairs with maximum similarity scores ranging between 0.93 and 1.0. a visualization of the relationship between maximal similarity score and match type is shown in figure 6 , where we observe that the likelihood of a pair being either a duplicate or template match increases with similarity score. considering the overall median maximal similarity score is 0.958, we can estimate a lower-bound for the rate of duplicate and template match pairs by scaling the proportion of documents above the median maximal score (i.e., half, or 0.5) by the fraction of duplicate and template matches above the median (0.958). this gives us 0.5 ⇥ 0.641, and therefore we estimate that at least 32% of samples from the rvl-cdip test set have either a duplicate counterpart or a sample that shares a template layout in the training set. while there is generally no established acceptable number or percentage for test-train overlaps, prior work (e.g., søgaard et al. (2021 ## presence of sensitive information while reviewing samples from rvl-cdip, we noticed that the resume category had a non-trivial quantity of documents that contain sensitive and personally-identifiable entities. naturally, resumes typically contain a person's name and basic contact information (e.g., phone numbers or email addresses). however, we found a plethora of sensitive entities like citizenship and marital statuses, places and dates of birth, names of children and spouses, and national id numbers like us social security and canadian national id numbers. out of a sample of 1,000 documents from the resume test set, we found that 7.7% contained a us social security number. while we recognize that us social security numbers were not considered sensitive several decades ago (when many of the resume documents in rvl-cdip were created), their presence in so many documents in a publicly accessible dataset foot_4 is still striking, especially considering the coexistence of this entity type with others like person names, dates and places of birth, etc. in particular, malicious social security numbers are often connected with fraud and identity theft crimes in the usa. moreover, the sensitive entities discussed in this section are considered highly sensitive under many state and national laws. 6 additionally, we found that 43.6% of the test resumes contain birth dates, 19.9% contain places of birth, 11.4% contain marital (or spousal or parental) statuses, and 8.9% contain citizenship statuses. example documents containing sensitive pii can be seen in figure 7 . given the presence of sensitive pii in rvl-cdip, it is reasonable to wonder if sensitive pii also appears in datasets derived from rvl-cdip, like funsd @xcite . similarly, we also wonder if sensitive pii appears in datasets that were derived from the larger iit-cdip or ucsf industry documents library corpora, such as tobacco-800 @xcite , tobacco-3482 @xcite , docvqa @xcite , and ocr-idl @xcite . we will investigate this in future work. ## discussion and recommendations given our findings concerning labeling errors, test/train overlap, and presence of sensitive information in the rvl-cdip document classification benchmark, we discuss several concrete recommendations to raise awareness among researchers engaged in benchmarking classifiers using this dataset: ## conclusion rvl-cdip has been used as the de facto benchmark for evaluating state-of-the-art document classification models, but this paper provides an indepth analysis of the rvl-cdip dataset and shows that there are several undesirable characteristics of this dataset. we first provide a set of label guidelines for each rvl-cdip category, and we use this to help us quantify the presence of errors in rvl-cdip, finding that the rvl-cdip test set contains roughly 8.1% label errors. we then observe that roughly a third of the test data is highly similar to the training set. lastly we observe an unsettling amount of personally sensitive information in rvl-cdip. given these findings, we offer suggestions for a new document classification benchmark.