RRED : A Radiology Report Error Detector based on Deep Learning Framework (2022)

Abstract: Radiology report is an official record of radiologists’ interpretation of patients’ radiographs and it’s a crucial component in the overall medical diagnostic process. However, it can contain various types of errors that can lead to inadequate treatment or delay in diagnosis. To address this problem, we propose a deep learning framework to detect errors in radiology reports. Specifically, our method detects errors between findings and conclusion of chest X-ray reports based on a supervised learning framework. To compensate for the lack of data availability of radiology reports with errors, we develop an error generator to systematically create artificial errors in existing reports. In addition, we introduce a Medical Knowledge-enhancing Pre-training to further utilize the knowledge of abbreviations and key phrases frequently used in the medical domain. We believe that this is the first work to propose a deep learning framework for detecting errors in radiology reports based on a rich contextual and medical understanding. Validation on our radiologist-synthesized dataset, based on MIMIC-CXR, shows 0.80 and 0.95 of the area under precision-recall curve (AUPRC) and the area under the ROC curve (AUROC) respectively, indicating that our framework can effectively detect errors in the real-world radiology reports.

## introduction radiology report is a document containing official interpretation of patients' radiographs which is used as an important communication tool between radiologists and referring physicians @xcite . the major components of the report include basic demographic information (e.g. patient's name, identifying number), findings which explains the image findings along with pertinent clinical information, and conclusion (also called impression) which is a list of summary state-⇤ these authors contributed equally. radiology report errors can be categorized and defined in different ways, mostly based on their causes. kim and mansfield classified the errors in 12 types which include errors caused by underreading, location of the lesion, and faulty reasoning. pinto et al. claim that radiology report errors can be classified based on 4 main reasons why radiologists are sued which include observer errors, errors in interpretation, and failure to suggest proper recommendations. sangwaiya et al. has analyzed errors on location and size discrepancy of lesions in radiology reports. combining these works, we conclude that the errors that contribute most to inappropriate or delayed diagnosis are radiologists failing to identify and interpret abnormalities, and discrepancies in size or location of the lesions reported. although there have been sufficient discussions in previous studies on methods to reduce errors in radiology reports, research on algorithms to directly detect such errors has been conducted at a very basic level. lee et al. proposed a software that detects the laterality error for the side or sites between the radiology report and its examination name. minn et al. proposed an algorithm to detect gender and laterality mismatch in report and its metadata. @xcite based neural model to detect inappropriate insertions, deletions, and substitutions of words in radiology reports. as such, existing studies on error detection in radiology reports were conducted only on local parts such as gender, laterality, and a single word, and most of these were done by simple matching without considering deep contextual meanings. unfortunately, in real life, radiologists' error occur due to more complicated reasons that cannot be covered by these approaches. considering how radiologists record their interpretation and communicate with referring physicians, capturing and understanding the contextual meaning of each section in the report is an important part for practical error detectors that can be used in real life. in the field of nlp, many pre-trained language models (plms) are showing remarkable achievement in various tasks of natural language understanding since the advent of elmo @xcite and bert @xcite . recently, several studies on plm that utilize world knowledge for language understanding have appeared, showing outstanding results not only in the general domain but also in domain-specific tasks @xcite @xcite @xcite . specifically, plms specialized in the medical domain such as clinicalbert @xcite and biomegatron @xcite have shown notable performance in the medical nlp tasks. despite the remarkable achievement in the field of nlp, the main barrier to apply these technologies, is the absence of radiology reports with errors to perform plm supervised learning. two reasons are identified behind the lack of accessible data. first, identifying errors in radiology reports can only be done by well-trained radiologists which is time-consuming and requires costly manual work. second, in fact, radiology report errors do not occur as often enough for them to be collected and used to train deep learning models. it is estimated that in a daily practice, the rate of radiology report errors that are substantial to result in inappropriate or delayed diagnosis is less than 4% @xcite . also, when considering the different types of errors, classifying and collecting enough data for each type of error is an unrealistic approach. here, we introduce two novel approaches to identify errors in radiology reports based on the understanding of the nature of radiology reports while overcoming the challenge created by inadequate radiology report error data: 1) to compensate for the lack of data availability of radiology reports with errors, we introduce an artificial error generator. the error generator synthesizes errors that mimic radiologists behaviors that potentially cause errors in daily practice. it can generate different types of errors by employing appropriate and relevant medical knowledge. 2) in order to incorporate medical knowledge for detecting complex errors, we introduce a medical knowledge-enhancing pretraining task, which is inspired by ernie1.0 @xcite , to our bert based error detector. this additional pre-training task allows the detector to directly learn medical abbreviations and frequent phrases in radiology report. to validate our proposed approach, experiments are performed on mimic-cxr @xcite with part of it including intentionally generated error by a board-certified radiologists. furthermore, through additional experiments, the proposed model was able to identify errors in original mimic-cxr which was verified by human evaluation. the experiment results show that the error detector can detect errors in real-world data while it is trained on artificial errors generated by the error generator. additionally, external validation, experiments on domain adaptation, and several ablation studies well prove the generalizability of the error detector and the performance of the knowledgeenhancing pre-training task. in summary, our main contributions are as follows: ## experiments in this section, we describe the datasets, implementation details, and experiment results of the error detection task on several datasets. ## discussion through the evaluation of rred, we mainly focus on precision rather than recall. this is because this study aims to develop a practical and reliable error detector that can be used in daily practice with a low false alarm rate. we believe that minor errors are worth missing if the alarm can provide a strong guarantee of actual errors to the radiologists. despite the fact that the experimental results show notable effectiveness of our approach, there are some limitations. first, the types of errors that have been implemented and experimented with, do not represent the entire scope of radiology report errors. while interpretive and factual errors are critical in the process of diagnosis, expanding the type of errors would be beneficial to reflect real-life errors in radiology practice. second, the error generator relies on simple random swapping to generate interpretive errors. although the experimental results show how this method is effective in large dataset like mimic-cxr, it is evident that this does not fully reflect the true nature of the reallife interpretive error. if the error generator can improve its' ability to imitate the behavior of radiologists, the error detector is expected to capture complex interpretive errors more precisely. ## conclusion in this paper, we present rred, a radiology report error detector based on a rich understanding of context and medical knowledge with supervised deep learning framework. we also propose a error generator for generating synthetic report data with errors to train the detector model. through various types of evaluations, we showed that our framework can be effectively applied to real world data to detect errors that could cause inappropriate or delayed diagnosis. we also showed the significant effects of the mkp which is a proposed pretraining task to integrate medical knowledge into pre-training language model. to the best of our knowledge, this is the first work proposing plm based error detection model for radiology reports. in future works, we plan to develop rred2.0 with improved error generator and detector: 1) we will investigate more systematic approaches to generate a broader range of errors in radiology reports, in an effort to expand and improve the usability of the radiology report error detector. 2) we will expand this work to develop a vision-language error detector that can detect errors also in the findings section which is intended to record findings when reading radiographs. we expect this work to become a practical foolproof system that can reduce critical errors in radiology reports to improve the quality of radiology reporting process and further, the entire diagnosis process.