On the diminishing return of labeling clinical reports (2020)

Abstract: Ample evidence suggests that better machine learning models may be steadily obtained by training on increasingly larger datasets on natural language processing (NLP) problems from non-medical domains. Whether the same holds true for medical NLP has by far not been thoroughly investigated. This work shows that this is indeed not always the case. We reveal the somehow counter-intuitive observation that performant medical NLP models may be obtained with small amount of labeled data, quite the opposite to the common belief, most likely due to the domain specificity of the problem. We show quantitatively the effect of training data size on a fixed test set composed of two of the largest public chest x-ray radiology report datasets on the task of abnormality classification. The trained models not only make use of the training data efficiently, but also outperform the current state-of-the-art rule-based systems by a significant margin.

## introduction it is commonly believed that neural network classifier performance increases as more data with labels are provided, if its capacity is properly optimized and regularized @xcite . in natural language processing (nlp), pretraining on large tb-scale text corpora has become the standard practice in recent years with the emergence of bert @xcite and @xcite , before being fine-tuned on taskspecific targets. furthermore, variants of top deep leanirng (dl) architectures have dominated several language task benchmarks due to their superior computational scalability and statistical capacity @xcite . modern neural networks typically employ a large number of parameters, offering large capacity and flexibility in capturing highly non-linear linguistic phenomena. as a model family with low bias and high variance, it naturally requires much more data to avoid the pitfall of overfitting. the success of billion-parameter dl models trained on billion-word datasets have drastically transformed the landscape of nlp. does the same trend hold true in the specific domain of medicine? after all, it is reasonable to expect a significant syntactic and semantic gap between everyday conversations (twitter feeds, news articles, blog posts) and formal medical vocabulary spoken or written in the context of clinical medicine. in fact, it typically requires decades of specialized medical training to excel in this highly demanding field. such specialization is thus expected for any machine learning model that is trained to perform medical tasks, whether it is information retrieval @xcite , conversational agent @xcite , or disease extraction and classification @xcite . this work investigates the impact of corpus size on the performance of state-of-the-art dl models on multi-label medical report classification tasks. we empirically demonstrate some of the unique properties of the medical language in clinical reports and how such domain-specific features lead to a surprisingly different scaling behavior as the training data size increases. our main results on two public chest x-ray radiology report datasets consistently suggest that some of the classification tasks do not require copious amount of labeled data to achieve good performance, mostly due to the limited linguistic variation in its domain. although per category analyses reveal slight variations, this phenomenon is consistently demonstrated across four dl model families presented. performance between 6,000 and 30,000 reports remain counterintuitively comparable, demonstrating diminishing returns in labeling effort. in addition, we show that, with a relatively small amount of data, dl models outperform both our private and state-of-the-art (sota) public ruled-based systems by a large margin. non-medical nlp the early work of @xcite demonstrates a linear performance gain of an nlp disambiguation task when doubling the amount of training corpus on simple machine learning (ml) linear classifiers. recent nlp advance pushes the envelop much further by leveraging web-scale data -for instance, the common crawl project foot_0 that produces 20tb of textual data from the internet each month. to cope with such a scale, large models with billions of parameters based on the variants of bert @xcite , mt-dnn @xcite , gpt-2 @xcite , xl-net @xcite have emerged with sometimes near-human performance on selected language tasks. it is commonly recognized that better performance can be achieved by training larger models on larger datasets. non-ml medical nlp although traditional nlp performance has improved on medical tasks over time, it doesn't lend itself to the scale of significant performance improvements seen with models trained on datasets several terabytes in size @xcite . results shown in a 1999 paper (taira and soderland, 1999) on statistical nlp for medical reports, a 2006 paper @xcite ) extracting predefined problems from clinical notes, a 2013 paper mining fda drug labels @xcite , and a 2019 dataset using chexpert labeler @xcite demonstrate gradual performance improvements that still reflect the high precision, low recall phenomenon observed with non-ml medical nlp tools. ml medical nlp nonetheless, machine learning based nlp models have been explored where a reasonable amount of electronic medical records (emr) are obtained privately. chen et al. (2018) uses a convolutional neural network (cnn) trained on 2500 thoracic computed tomography (ct) reports to identify pulmonary embolism findings with high aucs. @xcite uses a corpus of 3032 musculoskeletal x-ray reports to train a recurrent neural network (rnn) to identify fracture and non-fracture cases with high precision and recall. the work from @xcite represents one of the largest studies on emr where freetext notes from doctors, nurses and other providers from 216k patients are used to predict mortality, readmission, and length of hospital stay. @xcite uses hundreds of thousands of chest x-ray reports, but focuses on the task of report generation instead of classification. @xcite evaluates the impact of combining 6 clinical and non-clinical corpora on similarity of word embeddings in the clinical domain. results showed task-dependent performance variations. a study from @xcite on ner in a low resource language showed improved performance when 3 datasets were combined. in line with the dominant trend, @xcite applied machine learning techniques to the task of confusion set disambiguation, using three orders of magnitude more training data than previously been applied to the problem. they significantly reduced the error rate simply by adding more training data. even with a billion words, the learners continued to benefit from additional training data. in contrast, @xcite confronts this claim showing that although convergence behaviour on unigram probability estimates improves when using up to one billion words, for some words, no such convergence occurs. ## experiments the primary goal of this study is to empirically examine the impact of increasing and decreasing the size of the training data to the quality of dl models produced, and to compare their performance with state-of-the-art rule-based methods. thus in all following experiments, we freeze the test set while using different sizes of training and validation set to tune the dl models. in particular, we establish incrementally bigger training data by randomly sampling a larger percentage from the entire training set. as a baseline for performance comparison, we also introduce two rule-based classifiers whose performance is independent of the training data sizes. ## conclusion the intuition from low resource languages and the general non-medical domain suggests sustained performance improvement with larger datasets. we seek to answer the lingering question about how much data is considered as sufficient in classifying clinical texts and show that the necessary corpus size varies with the complexity of the specific category but to a far less degree than what has been previously thought. with less than 6,000 labeled reports, dl models are able to yield comparable performance with using 30,000 reports on a multilabel classification problem, demonstrating the counter-intuitive effect of diminishing returns from expert labeling efforts.