CXR Data Annotation and Classification with Pre-trained Language Models (2022)

Abstract: Clinical data annotation has been one of the major obstacles for applying machine learning approaches in clinical NLP. Open-source tools such as NegBio and CheXpert are usually designed on data from specific institutions, which limit their applications to other institutions due to the differences in writing style, structure, language use as well as label definition. In this paper, we propose a new weak supervision annotation framework with two improvements compared to existing annotation frameworks: 1) we propose to select representative samples for efficient manual annotation; 2) we propose to auto-annotate the remaining samples, both leveraging on a self-trained sentence encoder. This framework also provides a function for identifying inconsistent annotation errors. The utility of our proposed weak supervision annotation framework is applicable to any given data annotation task, and it provides an efficient form of sample selection and data auto-annotation with better classification results for real applications.

## introduction previous work @xcite has showed clinical text classification can significantly benefit from supervised learning approaches when annotated data is available. however, annotation of clinical data is extremely expensive and time consuming since annotation needs human experts' domain knowledge. to circumvent this, some rule-based methods have been developed using expert knowledge, e.g., negbio @xcite and chexpert @xcite using rule labeler to automatically detect the presence of observations in radiology reports. @xcite mentioned that chexpert is computationally slow, and its output is nondifferentiable, so they proposed to train a bert based classier (chexpert++) based on the output of chexpert. @xcite proposed to combine automatic labelers with expert knowledge by first fine-tuning bert classifier on output of chexpert and then on a small set of expert annotations augmented with automated backtranslation. while annotation is data and task specific @xcite , it is further complicated by the differences in writing style, structure, language use such as the vocabularies and phrase variability among different institutes and different countries. for example, phrases "airspace changes" and "infective change(s)" are commonly seen in our local data to describe pneumonia, but rarely seen in mimic data @xcite . in addition, label definitions vary among institutes. for example, chexpert classifies the sentence "suspicious for pneumonia" as "pneumonia uncertainty" based on their rule definition, while our clinicians/radiologists would consider it as an implication of "pneumonia positive". these differences limit the application of open-source tools on different data, and further limit the applications of methods @xcite which heavily rely on opensource tools. some active learning @xcite and interactive learning methods @xcite have been commonly used for reducing the experts' annotation burden. @xcite proposed uncertainty-based and diversity-based sampling to annotate clinical ners. both sampling approaches adopt random sampling and longest sentence sampling, for comparison, to build the initial set for manual annotation. for subsequent annotation and updating, the uncertainty-based sampling relies on model's predictions while diversity-based selects samples based on pair-wise sentence similarity. pair-wise sentence similarity is calculated based on individual words, syntax or clinical concepts, with the aim to select samples with lower similarity to annotated samples in the initial set. @xcite proposed an interactive learning method, req-rec, which is very similar to the uncertainty-based sampling. the method leverages on human experts' domain knowledge to build a list of sense-specific contextual words and use them to search for related sentences to form the initial annotation set. for subsequent sampling, it is based on the model's prediction too. the more ambiguous samples will be selected for annotation. for the above approaches, it is not efficient in practice since they incur many rounds of model retraining @xcite and multiple cycles of annotation by experts. time taken for human annotation is normally affected more by the duration of the annotation cycle than by the sample sizes as experts are not readily available, especially in the clinical domain. pair-wise sentence similarity is limited by using words, syntax or extracted clinical concepts to represent sentence since they cannot capture the semantic meaning of the whole sentence. in our data, it is very common to have sentences with the same clinical concepts annotated differently due to negation or speculation. in addition, distribution and the number of samples selected for initial annotation affects the performance of the model, which will then affect the prediction quality of the remaining samples. to effectively select samples for initial annotation and avoid multiple training cycles and annotation by human experts, in this paper, we propose a new weak supervision annotation framework to overcome the retraining and multiple annotation process. within the proposed framework, we adopt deep neural networks (dnn) for sample selection and text classification, which can fill the gap of using dnns in active learning for text classification @xcite . our work has the following contributions: ## conclusion we propose a semi-supervised annotation scheme which avoids multiple model re-training and expert annotation which is applicable to cxr text data and other domain data annotation (appendix a.2). within the framework, we investigate a gap mentioned by schroder et al. (2020) by using finetuning-based models in active learning for text classification. we utilize a self-trained sentence encoder for effective sample selection through clustering, error auto-detection and sample autoannotation. based on the annotated data, we further fine-tune a pre-trained language model to obtain a robust classifier which demonstrates high performance on cxr data disease detection. this method greatly improves data annotation efficiency and relieves human annotation burden.